How can I find the index rate info

OpenDistro
1

Hi, it’s easy to check the index rate of each index in Elasticsearch/Kibana under Stack Monitoring. A screenshot is attached below.

Screenshot from ES
Screenshot from ES3840×1846 272 KB

But I am having trouble finding the same info in opensearch-dashboard. Does opensearch provides it? If it requires user to create a query to find the index rate, could opensearch share the query, or even better, a visualization or dashboard.

Thank you

2

Currently, it is not available, Stack Monitoring is a unique feature from Elasticsearch. But It would be great if Opensearch have the same :kissing:

3

We have a much more advanced dashboard we use which is based on Prometheus and Grafana to monitor our entire pipeline. I can probably share a copy, but it does require that you use a proper monitoring system (Prometheus) and you can generate alerts from alertmanager if you’d like.

Here is another example with directions on the exporter: Monitoring Elasticsearch with Prometheus: Configuration, dashboards and alerts. (promcat.io)

Another example too: Monitoring ElasticSearch Cluster using Prometheus and Grafana - bidhankhatri.com.np

4

Thank you @jkowall
I have the elasticsearch-exporter deployed(image: Quay). It looks like it only collects ingest rate data on a node level, but not an index level. And all index related graphs are broken like the screenshot below

Screenshot from 2022-02-07 09-41-01
Screenshot from 2022-02-07 09-41-012588×1646 113 KB

It seems metrics like elasticsearch_indices_docs_primary doesn’t even exist in my prometheus database

5

Confirmed that I have flags like --es.all --es.indices --es.shards enabled in my exporter like below

[root@host-1 ~]# kubectl exec -it -n test-os elastic-exporter-5cf47f4545-2z8tr -- sh
/ $ 
/ $ ps |more
PID   USER     TIME  COMMAND
    1 10000     0:00 /bin/elasticsearch_exporter --es.uri=https://USER:PASSWORD@test-os-master.test-os.svc:9200 --es.ssl-skip-verify --es.all --es.indices --es.shards
   14 10000     0:00 sh
   22 10000     0:00 ps
   23 10000     0:00 more
/ $

However, I can’t find the metric

[root@host-1 ~]# kubectl get svc -n test-os elastic-exporter
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
elastic-exporter   ClusterIP   10.43.125.103   <none>        9114/TCP   11d
[root@host-1 ~]# 

[root@host2 ~]# curl -s 10.43.125.103:9114/metrics | wc -l
4169
[root@host2 ~]# curl -s 10.43.125.103:9114/metrics |grep elasticsearch_indices_docs_primary
[root@host2 ~]# curl -s 10.43.125.103:9114/metrics |grep elasticsearch_indices_store_size_bytes_primary
[root@host2 ~]# curl -s 10.43.125.103:9114/metrics |grep elasticsearch_indices_store_size_bytes_total
[root@host2 ~]# curl -s 10.43.125.103:9114/metrics |grep elasticsearch_indices_segment_doc_values_memory_bytes_primary
[root@host2 ~]#
6

OK. I think it’s a permission issue. Checking what permissions are needed

level=warn ts=2022-02-07T19:01:30.190751746Z caller=indices.go:1096 msg="failed to fetch and decode index stats" err="HTTP Request failed with code 403"
level=warn ts=2022-02-07T19:01:40.190992731Z caller=indices.go:1096 msg="failed to fetch and decode index stats" err="HTTP Request failed with code 403"
1 Like
7

By adding indices_monitor permissions to my exporter account, index level data does show up. And I can get the index rate like rate(elasticsearch_indices_docs_primary{exported_cluster="$cluster"}[1m])
Thank you for all the help

1 Like