Opensearch JVM monitor

OpenSearch
1

Hello, which tool are you using to monitor heap in docker?
I have 9 nodes Opensearch cluster and I am monitoring the servers by Node Exporter but I need a tool to collect heap from running Opensearch like JVM.
Ideally, it would be metrics collected by Prometheus and I would visualize them in Grafana.

2

Hello @vnovotny98, please check GitHub - aiven/prometheus-exporter-plugin-for-opensearch: Prometheus exporter plugin for OpenSearch

1 Like
3

I will try. Thanks :heart:

4

@reta
I installed plugin on every node:

[opensearch@node03-prod ~]$ ./bin/opensearch-plugin install file:///usr/share/opensearch/config/prometheus-exporter-1.3.3.0.zip
-> Installing file:///usr/share/opensearch/config/prometheus-exporter-1.3.3.0.zip
-> Downloading file:///usr/share/opensearch/config/prometheus-exporter-1.3.3.0.zip
[=================================================] 100%??
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessClassInPackage.sun.misc
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed prometheus-exporter with folder name prometheus-exporter

But I cant see no metrics.
I tried inside or outside docker but still got this error
curl -k https://address:9200/_prometheus/metrics

{"error":"Incorrect HTTP method for uri [/_prometheus/metrics] and method [GET], allowed: [POST]","status":405}
With POST
{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}

Can you help with that?

5

hey @vnovotny98 , sure, just to make sure: the plugins installation requires restart of all affected nodes, have you done this? If yes, could you please run curl -k https://localhost:9200/_prometheus/metrics on any of the nodes with plugin installed? Thank you.

6

Hello @reta , I have opensearch in docker-compose and when I restart node, i need to jump in docker with

docker exec -it ā€˜opensearch-nodeā€™ /bin/bash

and then install plugin again, because when I install him on node and then restart node, then he is not listed and installed

bin/opensearch-plugin list

So I didnā€™t restart nodes. I will need help with persistent this plugin maybe. I have zip file that take to docker with set volume in docker-compose.yml
My docker-compose.yml
looks like:

services:
opensearch-node1:
image: opensearchproject/opensearch:1.3.3
container_name: apesmaster-node1-elastic
environment:
- cluster.name=apesmaster-cluster
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- ā€œOPENSEARCH_JAVA_OPTS=-Xms3144m -Xmx3144mā€ # minimum and maximum Java heap size, recommend setting both to 50% of system RAM - limit

volumes:
  - /mnt/elkstore/usr/share/opensearch/data/:/usr/share/opensearch/data
  - /root/prometheus-exporter-1.3.3.0.zip:/usr/share/opensearch/config/prometheus-exporter-1.3.3.0.zip

After restart I have prometheus-exporter in docker but not installed plugin.

7

Hello @vnovotny98 ,

I am afraid you need to build your own Docker image with plugin installed, sample Dockerfile below:

FROM opensearchproject/opensearch:1.3.3
ADD  --chown=opensearch:opensearch https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/1.3.3.0/prometheus-exporter-1.3.3.0.zip /usr/share/opensearch/prometheus-exporter-1.3.3.0.zip 
RUN bin/opensearch-plugin install -b -s file:///usr/share/opensearch/prometheus-exporter-1.3.3.0.zip

And build from it in docker-compose specification (use build instead of image):

...
  opensearch-node1:
    build: .
    ...

That should do it:

> curl -k https://localhost:9200/_prometheus/metrics -u admin:admin  

# HELP opensearch_index_segments_number Current number of segments  
....
8

Thanks @reta
you helped me a lot!

Now I am scraping Opensearch with Prometheus!

The last thing I need is to find some JSON that uses the right metrics, so I can visualize in Grafana, I tried

but nothing uses these metrics, can you help me or should I start a new topic for this?

9

You are welcome, @vnovotny98 , sadly I donā€™t know anything about the Grafanaā€™s dashboard part, will try to research, may be @lukas-vlcek you could chime in?

10

Just in case anyone else lands here, I recommend Sematext Cloud. Iā€™m biased because I work at Sematext, but it has a lot of stuff that might be useful:

  • OS and JVM-specific metrics (e.g. for monitoring heap and GC)
  • it can parse GC logs so you can see more details of what happens, and there are lots of tips to tell you what to do e.g. if you have lots of allocation failures
  • it can get OpenSearch-specific metrics (from search latency per shard to cache hit ratio to script compilation and everything in between) and can parse OpenSearch logs as well to give you e.g. most expensive queries or errors during startup

For the moment OpenSearch-specific metrics and logs live under the Elasticsearch umbrella, but weā€™re in the process of forking our application types :wink:

1 Like
11

thereā€™s a pending issue to ship some pre-built dashboards directly with the plugin: Provide Grafana dashboard and Prometheus Rules & Alerts Ā· Issue #12 Ā· aiven/prometheus-exporter-plugin-for-opensearch Ā· GitHub

while that doesnā€™t resolve your immediate problem you could mention that JVM metrics would be of interest to you.

also, you can just take an existing dashboard and then extend it with the metrics which you want. as mentioned by @reta you can see all available metrics under the listed URL.
if you take an Elasticsearch based dashboard youā€™ll just have to replace the es_ prefix with opensearch_ and then it should all continue to work.