Describe the issue:
When i try the OpenSearch Dashboard alerting for Prometheus Metrics or Metrics Graph there are not working.
I have a Prometheus, OpenSearch and OpenSearch Dashboard.
So i open the OpenSearch Dashboard and i Go to the Dev tools the path of Dev tool is “OpenSearch Dashboard > Management > Dev Tools” and i open the Dev Tools.
And i run this below code for automatic make the Prometheus Data source.
When i run this code i go to the Data Sources the path of Data Sources is “OpenSearch Dashboard > Management > Dashboard Management > Data Sources” there are add new Data source name is “my_prometheus”. this is the Image of Data source:
And than i see the Indexes the path of Indexes is “OpenSearch Dashboard > Management > Index Management > Indexes” and i see one indexes “.ql-Datasource” add one Document like below image :
And than i see the Metrics of “my_prometheus” the path of Metrics “OpenSearch Dashboard > Observability > Metrics” and select the Drop down Metrics Source “Prometheus” and there are lots of Prometheus metrics are open in Available Metrics like below image :
And than i see the Alerting and create a Monitor in Alerting the path of Create Monitor is “OpenSearch Dashboard > OpenSearch Plugins > Alerting > Monitors > Create Monitor” in the Form “Select Data” i select indexes “.ql-datasources” and i try the query for show all fields of “my_prometheus” but there are not any response see the below image :
@malyadri The main reason of empty response is the user. In your example you’re using admin user. To see the content of the .ql-datasources index you must use superadmin certificate defined in the opensearhc.yml under admin_dn.
I would take as rule of thumb that any indices starting with “.” must be read by superadmin.
Regarding your alerting, I don’t understand why would you set alerts against a config index. This index holds only data source configuration. No other data will be ingested there.
Make sure your alerting rule uses a valid PromQL query, proper evaluation interval, and references the correct Prometheus data source used in your dashboard.