# Opensearch JVM monitor **URL:** \ **Category:** OpenSearch\ **Created:** [August 24, 2022, 9:25am UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721 "2022-08-24T09:25:22Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![vnovotny98](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vnovotny98](https://forum.opensearch.org/u/vnovotny98)\ **Post date:** [August 24, 2022, 9:25am UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/1 "2022-08-24T09:25:22Z")
Hello, which tool are you using to monitor heap in docker? I have 9 nodes Opensearch cluster and I am monitoring the servers by Node Exporter but I need a tool to collect heap from running Opensearch like JVM. Ideally, it would be metrics collected by Prometheus and I would visualize them in Grafana. ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [August 24, 2022, 12:32pm UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/2 "2022-08-24T12:32:52Z")
Hello @vnovotny98, please check [GitHub - aiven/prometheus-exporter-plugin-for-opensearch: Prometheus exporter plugin for OpenSearch](https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/) ---
**Author:** ![vnovotny98](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vnovotny98](https://forum.opensearch.org/u/vnovotny98)\ **Post date:** [August 24, 2022, 1:04pm UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/3 "2022-08-24T13:04:15Z")
I will try. Thanks ❤ ---
**Author:** ![vnovotny98](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vnovotny98](https://forum.opensearch.org/u/vnovotny98)\ **Post date:** [August 29, 2022, 7:01am UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/4 "2022-08-29T07:01:07Z")
@reta I installed plugin on every node: ```auto [opensearch@node03-prod ~]$ ./bin/opensearch-plugin install file:///usr/share/opensearch/config/prometheus-exporter-1.3.3.0.zip -> Installing file:///usr/share/opensearch/config/prometheus-exporter-1.3.3.0.zip -> Downloading file:///usr/share/opensearch/config/prometheus-exporter-1.3.3.0.zip [=================================================] 100%?? @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission accessClassInPackage.sun.misc * java.lang.RuntimePermission accessDeclaredMembers * java.lang.reflect.ReflectPermission suppressAccessChecks See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y -> Installed prometheus-exporter with folder name prometheus-exporter ``` But I cant see no metrics. I tried inside or outside docker but still got this error curl -k [https://address:9200/\_prometheus/metrics](https://address:9200/_prometheus/metrics) `{"error":"Incorrect HTTP method for uri [/_prometheus/metrics] and method [GET], allowed: [POST]","status":405}` With POST `{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}` Can you help with that? ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [August 29, 2022, 12:12pm UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/5 "2022-08-29T12:12:38Z")
hey @vnovotny98 , sure, just to make sure: the plugins installation requires restart of all affected nodes, have you done this? If yes, could you please run `curl -k https://localhost:9200/_prometheus/metrics` on any of the nodes with plugin installed? Thank you. ---
**Author:** ![vnovotny98](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vnovotny98](https://forum.opensearch.org/u/vnovotny98)\ **Post date:** [August 29, 2022, 1:34pm UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/6 "2022-08-29T13:34:35Z")
Hello @reta , I have opensearch in docker-compose and when I restart node, i need to jump in docker with > docker exec -it ‘opensearch-node’ /bin/bash and then install plugin again, because when I install him on node and then restart node, then he is not listed and installed > bin/opensearch-plugin list So I didn’t restart nodes. I will need help with persistent this plugin maybe. I have zip file that take to docker with set volume in docker-compose.yml My docker-compose.yml looks like: > services: > opensearch-node1: > image: opensearchproject/opensearch:1.3.3 > container\_name: apesmaster-node1-elastic > environment: > - cluster.name=apesmaster-cluster > - bootstrap.memory\_lock=true # along with the memlock settings below, disables swapping > - “OPENSEARCH\_JAVA\_OPTS=-Xms3144m -Xmx3144m” # minimum and maximum Java heap size, recommend setting both to 50% of system RAM - limit > > ``` > volumes: > - /mnt/elkstore/usr/share/opensearch/data/:/usr/share/opensearch/data > - /root/prometheus-exporter-1.3.3.0.zip:/usr/share/opensearch/config/prometheus-exporter-1.3.3.0.zip > > ``` After restart I have prometheus-exporter in docker but not installed plugin. ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [August 29, 2022, 7:12pm UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/7 "2022-08-29T19:12:35Z")
Hello @vnovotny98 , I am afraid you need to build your own Docker image with plugin installed, sample `Dockerfile` below: ```auto FROM opensearchproject/opensearch:1.3.3 ADD --chown=opensearch:opensearch https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/1.3.3.0/prometheus-exporter-1.3.3.0.zip /usr/share/opensearch/prometheus-exporter-1.3.3.0.zip RUN bin/opensearch-plugin install -b -s file:///usr/share/opensearch/prometheus-exporter-1.3.3.0.zip ``` And build from it in `docker-compose` specification (use `build` instead of `image`): ```auto ... opensearch-node1: build: . ... ``` That should do it: ```auto > curl -k https://localhost:9200/_prometheus/metrics -u admin:admin # HELP opensearch_index_segments_number Current number of segments .... ``` ---
**Author:** ![vnovotny98](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vnovotny98](https://forum.opensearch.org/u/vnovotny98)\ **Post date:** [September 1, 2022, 7:37am UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/8 "2022-09-01T07:37:54Z")
Thanks @reta you helped me a lot! Now I am scraping Opensearch with Prometheus! The last thing I need is to find some JSON that uses the right metrics, so I can visualize in Grafana, I tried > **[OpenSearch Prometheus](https://grafana.com/grafana/dashboards/15178-opensearch-prometheus/)** > > OpenSearch cluster monitoring with OpenSearch Prometheus Exporter and Node Exporter > **[Elasticsearch Exporter Quickstart and Dashboard](https://grafana.com/grafana/dashboards/14191-elasticsearch-overview/)** > > A quickstart to setup the Prometheus Elasticsearch Exporter with preconfigured dashboards, alerting rules, and recording rules. but nothing uses these metrics, can you help me or should I start a new topic for this? ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [September 1, 2022, 6:44pm UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/9 "2022-09-01T18:44:29Z")
You are welcome, @vnovotny98 , sadly I don’t know anything about the Grafana’s dashboard part, will try to research, may be @lukas-vlcek you could chime in? ---
**Author:** ![radu.gheorghe](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/radu.gheorghe/32/3516_2.png) [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)\ **Post date:** [September 23, 2022, 4:01pm UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/10 "2022-09-23T16:01:33Z")
Just in case anyone else lands here, I recommend [Sematext Cloud](https://sematext.com/cloud/). I’m biased because I work at Sematext, but it has a lot of stuff that might be useful: - OS and JVM-specific metrics (e.g. for monitoring heap and GC) - it can parse GC logs so you can see more details of what happens, and there are lots of tips to tell you what to do e.g. if you have lots of allocation failures - it can get OpenSearch-specific metrics (from search latency per shard to cache hit ratio to script compilation and everything in between) and can parse OpenSearch logs as well to give you e.g. most expensive queries or errors during startup For the moment OpenSearch-specific metrics and logs live under the Elasticsearch umbrella, but we’re in the process of forking our application types 😉 ---
**Author:** ![ralph](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/ralph/32/1194_2.png) [@ralph](https://forum.opensearch.org/u/ralph)\ **Post date:** [September 26, 2022, 6:49am UTC](https://forum.opensearch.org/t/opensearch-jvm-monitor/10721/11 "2022-09-26T06:49:39Z")
there’s a pending issue to ship some pre-built dashboards directly with the plugin: [Provide Grafana dashboard and Prometheus Rules & Alerts · Issue #12 · aiven/prometheus-exporter-plugin-for-opensearch · GitHub](https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/issues/12) while that doesn’t resolve your immediate problem you could mention that JVM metrics would be of interest to you. also, you can just take an existing dashboard and then extend it with the metrics which you want. as mentioned by @reta you can see all available metrics under the listed URL. if you take an Elasticsearch based dashboard you’ll just have to replace the `es_` prefix with `opensearch_` and then it should all continue to work.