Followed the discussion to install plugin. Plugin installed without issue. But it’s showing anything on endpoint.
https://testmachine:9200/_prometheus/metrics
It’s giving
{“error”:“no handler found for uri [/_prometheus/metrics] and method [GET]”}
Also where we can find the configuration files of plugin.
I faced this issue while using the pristine es exporter
Is this sepearate project fork because of some compatibility issues between opensearch and ES exporter or for any other reason?
Forgot to restart opensearch service. It’s showing up metrics now. But still couldn’t locate how to customize the prefix of metrics or filteration capability. do we need to prepare the plugin with these settings?
Hi @vicky
Let me walk you through the configuration process a real quick:
Let’s start with a fresh Opensearch distribution. In this case it will be the Linux x64 minimal distribution:
Before we start the Opensearch node, verify there is no plugin installed on it:
% ./bin/opensearch-plugin list
# Empty response ...
Install the Prometheus plugin (documented here):
% ./bin/opensearch-plugin install -b https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.1.0.0/prometheus-exporter-2.1.0.0.zip
-> Installing https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.1.0.0/prometheus-exporter-2.1.0.0.zip
-> Downloading https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.1.0.0/prometheus-exporter-2.1.0.0.zip
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessClassInPackage.sun.misc
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed prometheus-exporter with folder name prometheus-exporter
Verify we have the plugin installed:
% ./bin/opensearch-plugin list
prometheus-exporter
Configure the metric prefix:
echo prometheus.metric_name.prefix: \"test_\" >> config/opensearch.yml
# Verify it has been recorded:
% cat config/opensearch.yml | grep prometheus.metric_name.prefix
prometheus.metric_name.prefix: "test_"
Now start the node and get some Prometheus metrics:
% ./bin/opensearch
# Switch to different terminal session
% curl -s -X GET localhost:9200/_prometheus/metrics | grep indices_get_current_number
# HELP test_indices_get_current_number Current rate of get commands
# TYPE test_indices_get_current_number gauge
test_indices_get_current_number{cluster="opensearch",node="Lukass-MBP",nodeid="A90E4OkkQLiRrBD4iVXueg",} 0.0
As we can see the metric name starts with the “test_” prefix.
Without the custom settings the metric name would have had the default prefix (“opensearch_”) like this:
% curl -s -X GET localhost:9200/_prometheus/metrics | grep indices_get_current_number
# HELP opensearch_indices_get_current_number Current rate of get commands
# TYPE opensearch_indices_get_current_number gauge
opensearch_indices_get_current_number{cluster="opensearch",node="Lukass-MBP",nodeid="A90E4OkkQLiRrBD4iVXueg",} 0.0
Regards,
Lukáš
Many thanks.
It looks like that prometheus-exporter vs your plugin metrics are not exactly similar.
With elasticsearch_exporter-1.5.0-1.el7.x86_64
[root@elasticsearch-data-1 ~]# curl -k -u testuser:testuser http://xx.xx.xx.xx:9108/metrics | grep 'elasticsearch_cluster' | grep -v '#' | sort
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 92614 0 92614 0 0 3131k 0 --:--:-- --:--:-- --:--:-- 3230k
elasticsearch_cluster_health_active_primary_shards{cluster="testcluster"} 35
elasticsearch_cluster_health_active_shards{cluster="testcluster"} 60
elasticsearch_cluster_health_delayed_unassigned_shards{cluster="testcluster"} 0
elasticsearch_cluster_health_initializing_shards{cluster="testcluster"} 0
elasticsearch_cluster_health_json_parse_failures 0
elasticsearch_cluster_health_number_of_data_nodes{cluster="testcluster"} 2
elasticsearch_cluster_health_number_of_in_flight_fetch{cluster="testcluster"} 0
elasticsearch_cluster_health_number_of_nodes{cluster="testcluster"} 3
elasticsearch_cluster_health_number_of_pending_tasks{cluster="testcluster"} 0
elasticsearch_cluster_health_relocating_shards{cluster="testcluster"} 0
elasticsearch_cluster_health_status{cluster="testcluster",color="green"} 1
elasticsearch_cluster_health_status{cluster="testcluster",color="red"} 0
elasticsearch_cluster_health_status{cluster="testcluster",color="yellow"} 0
elasticsearch_cluster_health_task_max_waiting_in_queue_millis{cluster="testcluster"} 0
elasticsearch_cluster_health_total_scrapes 19
elasticsearch_cluster_health_unassigned_shards{cluster="testcluster"} 0
elasticsearch_cluster_health_up 1
elasticsearch_clusterinfo_last_retrieval_success_ts{url="https://localhost:9200"} 1.66031658e+09
elasticsearch_clusterinfo_up{url="https://localhost:9200"} 1
elasticsearch_clusterinfo_version_info{build_date="2022-06-30T21:31:20.473786012Z",build_hash="388c80ad94529b1d9aad0a735c4740dce2932a32",cluster="testcluster",cluster_uuid="XAAqfYArRPeeoMgov4-xBA",lucene_version="9.2.0",version="7.10.2"} 0
With plugin based exporter.
[root@elasticsearch-data-0 ~]# curl -k -u testuser:testuser https://xx.xx.xx.xx:9200/_prometheus/metrics | grep 'elasticsearch_cluster' | grep -v '#' | sort
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 177k 100 177k 0 0 961k 0 --:--:-- --:--:-- --:--:-- 961k
elasticsearch_cluster_datanodes_number{cluster="testcluster",} 2.0
elasticsearch_cluster_inflight_fetch_number{cluster="testcluster",} 0.0
elasticsearch_cluster_is_timedout_bool{cluster="testcluster",} 0.0
elasticsearch_cluster_nodes_number{cluster="testcluster",} 3.0
elasticsearch_cluster_pending_tasks_number{cluster="testcluster",} 0.0
elasticsearch_cluster_routing_allocation_disk_threshold_enabled{cluster="testcluster",} 0.0
elasticsearch_cluster_routing_allocation_disk_watermark_flood_stage_pct{cluster="testcluster",} 95.0
elasticsearch_cluster_routing_allocation_disk_watermark_high_pct{cluster="testcluster",} 90.0
elasticsearch_cluster_routing_allocation_disk_watermark_low_pct{cluster="testcluster",} 85.0
elasticsearch_cluster_shards_active_percent{cluster="testcluster",} 100.0
elasticsearch_cluster_shards_number{cluster="testcluster",type="active",} 60.0
elasticsearch_cluster_shards_number{cluster="testcluster",type="active_primary",} 35.0
elasticsearch_cluster_shards_number{cluster="testcluster",type="initializing",} 0.0
elasticsearch_cluster_shards_number{cluster="testcluster",type="relocating",} 0.0
elasticsearch_cluster_shards_number{cluster="testcluster",type="unassigned",} 0.0
elasticsearch_cluster_status{cluster="testcluster",} 0.0
elasticsearch_cluster_task_max_waiting_time_seconds{cluster="testcluster",} 0.0
I am using the following config.
prometheus.metric_name.prefix: "elasticsearch_"
prometheus.indices: false
prometheus.cluster.settings: true
prometheus.nodes.filter: "_all"
Yes, they are not the same.
As explained here it was forked from specific commit hash and has been developed independently since then.
In particular the metric prefix name configuration option was introduced in 1.3.0.0 release.
Regards,
Lukáš