Add Exporter Plugin to Opensearch

Thanks for all :slight_smile: My Bad !

So, it seems good now :

-> Installing file:///home/osadmin/prometheus-exporter-1.2.3.zip
-> Downloading file:///home/osadmin/prometheus-exporter-1.2.3.zip
Retrieving zip from file:///home/osadmin/prometheus-exporter-1.2.3.zip
- Plugin information:
Name: prometheus-exporter
Description: Export OpenSearch metrics to Prometheus
Version: 1.2.3
OpenSearch Version: 1.2.3
Java Version: 1.8
Native Controller: false
Extended Plugins: []
 * Classname: org.opensearch.plugin.prometheus.PrometheusExporterPlugin
Folder name:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessClassInPackage.sun.misc
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed prometheus-exporter with folder name prometheus-exporter

If i want monitoring cluster, i need to deploy this plugin on all nodes ?

Thanks

Correct, please check out the docs [1], [2].

[1] https://github.com/aparo/opensearch-prometheus-exporter#configure-the-prometheus-target
[2] https://github.com/aparo/opensearch-prometheus-exporter#install

Thanks for all.
I can plug my Prometheus on Metrics API.

(l)

After usage, I find it heavy with that :frowning:

I have 72 nodes, installed on classic jvm on physical worker. So i need to install plugin on 72 nodes.

The Elasticsearch PLugin is more light and no need to install it on any nodes.
Furthermore, number of metrics is also big, 35k metrics by nodes :open_mouth:

(Also On Prometheus configuration, i need to declare 72 nodes…)

Thanks.

@GnarlyCapricorne by Elasticsearch PLugin you mean [1] or something else? Thank you.

[1] GitHub - prometheus-community/elasticsearch_exporter: Elasticsearch stats exporter for Prometheus

Yes Exactly.
The old exporter (for Elasticsearch CE) works very well because it goes up the metrics about all nodes.

… Unlike the new exporter for opensearch which requires installation on all nodes.

Also, new exporter is very resource intensive compared to the old exporter.

Just a feedback for all user.

PS : If anyone know how to modify an exporter or has already done so? Somes metrics are useless… :frowning:

Thanks

Gnarly

Hi Gnarly,

I would be really interested in learning more details.

The OpenSearch exporter works the same way as the original Elasticsearch exporter. Even for Elasticsearch exporter the recommendation was to install it on all nodes (because generally the assumption is that Prometheus is scraping metrics from all cluster nodes thus the plugin must be present on every node).

Can you please share more details about your OpenSearch cluster - Prometheus setup? How is Prometheus discovering individual OpenSearch nodes? Do you provide static list of nodes to scrape or do you rely on dynamic node discovery?

What makes you think the OpenSearch Prometheus exporter has degraded performance?

Which version of OpenSearch are you using BTW?

PS : If anyone know how to modify an exporter or has already done so? Somes metrics are useless…

Feel free to provide more details about the problem and I can instruct you.

Regards,
Lukáš

Hii,

The OpenSearch exporter works the same way as the original Elasticsearch exporter.

The elasticsearch exporter…
was installed like a java process on one physcal machine Opensearch.
I can configure the listening port.
I use es.all option to retrieve all cluster and node stats. (i not load indices, settings…).
With es.all this exporter generated 22 000 metrics AND i have all metrics for all nodes !
(If i install elasticsearch exporter on another node, we will have 22 000 * 2 metrics scraped.)
For information, we all options set, elasticsearch exporter generate 60 000 metrics by default.

Synthesis :
With just one install of this exporter, i have all metrics of cluster and nodes !
Ok it is a spof, but I just have to install it on another machine to have HA.
Volume of metrics (with full options) : 60 000 * 2 elasticsearch exporter (like 2 for HA)

The Opensearch exporter…
was installed like a plugin on Opensearch. Very easy to use.
I cannot configure the listening port (is not a problem because i can use push gateway…).
Options to limits metrics like indices, settings must be write in opensearch.yml ? Dynamic updated settings go to 404 error from GitHub - aparo/opensearch-prometheus-exporter: OpenSearch Prometheus Exporter (fork of https://github.com/vvanholl/elasticsearch-prometheus-exporter)
BUT if plugin is not deploy on all nodes, only metrics about the node where exporter was installed are scraped.
Plugin Opensearch exporter generated by default, 60 000 metrics by nodes. (I have 72 nodes…)
So 4 320 000 metrics scraped at each scrapping.

Synthesis :
For cluster informations it’s the same behavior.
For nodes stats, you must deploy plugin on all nodes to have full metrics like elasticsearch exporter. (…more consumer)

For info : Opensearch 1.2.3
My opensearch cluster is composed :
18 Linux physical machine with 300 GB RAM. (java process)
72 nodes (4 on each machine) : 3 masters, 6 coordinators (4 for dedicated writting) and 63 data nodes.
Network private. (we must use Push Gateway to export Metrics to Prometheus).

Prometheus is not direcly accessible by physical machine, we have to pass through a push gateway.
==> I don’t know how to use dynamic discovery node and i not view documentation about that. (And we don’t have acces to Prometheus with machines Opensearch)

With Opensearch or Elasticsearch Plugin; each nodes send metrics to Push Gateway.
Compared to the elasticsearch exporter, Opensearch exporter represents a much higher load because each node sends its own 60 000 metrics by default.

For me and for the moment, the easiest to use remains the elasticsearch plugin to scrape clusters and nodes metrics.

I may be wrong but looking at the load versus information reported, the elasticsearch plugin seems more profitable

I think it lacks an es.all on Opensearch Plugin like the Elasticsearch plugin does. That way, with opensearch plugin, I would just have to install the plugin on my coordinator nodes (4) and i have HA and all nodes metrics!

Hi Gnarly,

A quick question: which Elasticsearch exporter do you compare it to? It does not sound to me that you are actually comparing OpenSearch exporter to GitHub - vvanholl/elasticsearch-prometheus-exporter: Prometheus exporter plugin for Elasticsearch but to some other implementation (perhaps to a “side-car” exporter).

Let me have a look at your input and I will get back to you.

Lukáš

1 Like

Hi Gnarly,

in the meantime I am going to add new config for nodes filter option so that you can get all metrics from a single node as well.

Lukáš

Hi @GnarlyCapricorne,

Prometheus exporter for OpenSearch 1.3.0 has been just released (see Prometheus exporter plugin 1.3.0.0 released!).
It brings some new features, specifically, it allows to configure “nodes filter” settings. I think you might be interested in this feature. Please see release notes for more details.

Regards,
Lukáš

1 Like

based on the description i’m pretty sure this exporter here was meant:

which is not an ES plugin but a standalone component talking to ES APIs:

Hi,
Is there any documentation to build this project https://github.com/aparo/opensearch-prometheus-exporter from source code ?

Hi Manoj,

We have a documentation on how to build the prometheus exporter plugin from the source in this repo:
https://github.com/aiven/prometheus-exporter-plugin-for-opensearch#build-from-source

I believe similar approach should be possible with the Alberto’s repo as well.
BTW, are you trying to build the plugin for OpenSearch 1.x version or more recent one?

Regards,
Lukáš

Hi Lukas,
I’m trying to build the plugin for OpenSearch 1.2.4
And does building this project requires JDK 14 ?
And can I use latest gradle (7.4.2) in my local to build this ?

This is what I tried.
Used JDK 14 and Gradle 7.4.2. Got this below error while executing “gradle clean”

FAILURE: Build failed with an exception.

* Where:
Build file '/local/opensearch-prometheus-exporter/build.gradle' line: 35

* What went wrong:
A problem occurred evaluating root project 'prometheus-exporter'.
> Failed to apply plugin class 'org.opensearch.gradle.info.GlobalBuildInfoPlugin'.
   > Could not create plugin of type 'GlobalBuildInfoPlugin'.
      > Could not generate a decorated class for type GlobalBuildInfoPlugin.
         > org/gradle/jvm/toolchain/JavaInstallation

Used JDK 14 and Gradle 6.8. Executing “gradle clean init wrapper” successfully created gradlew file in local.
Got this below error while executing “./gradlew build”

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dependencyLicenses'.
> Licences dir /local/opensearch-prometheus-exporter/licenses does not exist, but there are dependencies: simpleclient-0.8.0.jar
  simpleclient_common-0.8.0.jar

Manually created licenses directory and tried again.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dependencyLicenses'.
> Missing SHA for simpleclient-0.8.0.jar. Run "gradle updateSHAs" to create them

Then, tried “gradle updateSHAs && ./gradlew build”. Now I’m getting this error.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dependencyLicenses'.
> Missing LICENSE for simpleclient-0.8.0.jar, expected in simpleclient-LICENSE.txt

Hi @lukas-vlcek @aparo ,

Any update on this ?
Building OpenSearch Exporter from it’s source code.

Thanks and Regards,
Manoj

Hi Majon,

please notice that my availability during the next week will be limited. I will probably get back to this thread on next Tuesday.

Thanks for understanding,
Lukáš

For me it wont work with Opensearch v2.0.1

sh-4.2$ ./bin/opensearch-plugin install -b https://github.com/aparo/opensearch-prometheus-exporter/releases/download/1.2.3/prometheus-exporter-1.2.3.zip
-> Installing https://github.com/aparo/opensearch-prometheus-exporter/releases/download/1.2.3/prometheus-exporter-1.2.3.zip
-> Downloading https://github.com/aparo/opensearch-prometheus-exporter/releases/download/1.2.3/prometheus-exporter-1.2.3.zip
-> Failed installing https://github.com/aparo/opensearch-prometheus-exporter/releases/download/1.2.3/prometheus-exporter-1.2.3.zip
-> Rolling back https://github.com/aparo/opensearch-prometheus-exporter/releases/download/1.2.3/prometheus-exporter-1.2.3.zip
-> Rolled back https://github.com/aparo/opensearch-prometheus-exporter/releases/download/1.2.3/prometheus-exporter-1.2.3.zip
Exception in thread "main" java.lang.IllegalArgumentException: Plugin [prometheus-exporter] was built for OpenSearch version 1.2.3 but version 2.0.1 is running

oke so Aiven’s plugin does work, Aparo’s does not since it’s not maintained.

@lukas-vlcek Hello to others I will write in Czech because it will be much easier for me to understand.
Ahoj, mám rozjetý Opensearch ve verzi 1.3.3, mám dva clustery, jeden tří nodový testovací, kde mám všechny nody se všemi rolemi a pak mám produkční kde mám 3 master, 3 ingest,data,hot nody a 3 data,warm nody. Mám ty stroje monitorované Node exporterem, ale kvůli heapu a lepšímu monitorování aplikace bych rád měl ještě tento prometheus-exporter.
Opensearch mi běží v docker-composu.
Stáhnul jsem si odsud zip

Jaké mám možnosti instalace prosím? Je možnost že bych ho připojil do docker-compose yamlu aby startoval automaticky se zapnutím dockeru a nebo to musím instalovat tak, že si ten zip natáhnu dovnitř do kontejneru a tam budu spouštět (asi ručně?) pomocí
./bin/opensearch-plugin install -b prometheus-exporter-1.3.3.zip
A máte svůj dashboard nebo v grafaně pak mohu použít např. tento?

Translated by google:
Hello, I have Opensearch in version 1.3.3 running, I have two clusters, one three-node test, where I have all nodes with all roles, and then I have a production one where I have 3 master, 3 ingest, data, hot nodes and 3 data, warm nodes. I have those machines monitored by Node exporter, but for heap and better application monitoring, I would like to have this prometheus-exporter.
Opensearch is running for me in docker-compos.
I downloaded the zip from here

What are my installation options please? Is it possible to connect it to docker-compose yaml so that it starts automatically when docker is turned on, or I have to install it by pulling the zip inside the container and starting it there (perhaps manually?) using
./bin/opensearch-plugin install -b prometheus-exporter-1.3.3.zip
And do you have your dashboard or can I use, for example, this one in grafana?