How to create Alerts from Prometheus metrics in OpenSearch Dashboard?

@pablo please help me on this issue

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
version: 2.32.0
appVersion: “2.19.1”

Describe the issue:
When i try the OpenSearch Dashboard alerting for Prometheus Metrics or Metrics Graph there are not working.

I have a Prometheus, OpenSearch and OpenSearch Dashboard.

So i open the OpenSearch Dashboard and i Go to the Dev tools the path of Dev tool is “OpenSearch Dashboard > Management > Dev Tools” and i open the Dev Tools.

And i run this below code for automatic make the Prometheus Data source.

POST _plugins/_query/_datasources 
{
    "name" : "my_prometheus",
    "connector": "prometheus",
    "properties" : {
        "prometheus.uri" : "http://prometheus-server.default.svc.cluster.local:80"
    },
    "allowedRoles" : ["all_access"]
}

When i run this code i go to the Data Sources the path of Data Sources is “OpenSearch Dashboard > Management > Dashboard Management > Data Sources” there are add new Data source name is “my_prometheus”. this is the Image of Data source:

And than i see the Indexes the path of Indexes is “OpenSearch Dashboard > Management > Index Management > Indexes” and i see one indexes “.ql-Datasource” add one Document like below image :

And than i see the Metrics of “my_prometheus” the path of Metrics “OpenSearch Dashboard > Observability > Metrics” and select the Drop down Metrics Source “Prometheus” and there are lots of Prometheus metrics are open in Available Metrics like below image :

And than i see the Alerting and create a Monitor in Alerting the path of Create Monitor is “OpenSearch Dashboard > OpenSearch Plugins > Alerting > Monitors > Create Monitor” in the Form “Select Data” i select indexes “.ql-datasources” and i try the query for show all fields of “my_prometheus” but there are not any response see the below image :

Configuration:
I want create Prometheus alerts in dashboard which we like create alerts for logs

Relevant Logs or Screenshots:

@pablo

please help me on this

@malyadri
Hi, is there any plan to test Grafana which is compatible with your prometheus exporter?

@malyadri The main reason of empty response is the user. In your example you’re using admin user. To see the content of the .ql-datasources index you must use superadmin certificate defined in the opensearhc.yml under admin_dn.

See my example.

[opensearch@opensearch-cluster-master-0 ~]$ curl --insecure --cert ./config/kirk.pem --key ./config/kirk-key.pem https://localhost:9200/.ql-datasources/_search?pretty
{
  "took" : 24,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".ql-datasources",
        "_id" : "my_prometheus",
        "_score" : 1.0,
        "_source" : {
          "name" : "my_prometheus",
          "description" : "",
          "connector" : "PROMETHEUS",
          "allowedRoles" : [
            "all_access"
          ],
          "properties" : {
            "prometheus.uri" : "http://prometheus-server.default.svc.cluster.local:80"
          },
          "resultIndex" : "query_execution_result_my_prometheus",
          "status" : "ACTIVE"
        }
      }
    ]
  }

I would take as rule of thumb that any indices starting with “.” must be read by superadmin.

Regarding your alerting, I don’t understand why would you set alerts against a config index. This index holds only data source configuration. No other data will be ingested there.

Hi, I am trying to create alerts using Prometheus metrices.

I can be able to see the metrices in dashboard in observability but while creating alerts I couldn’t get the metrices data

I want to create alert :
example query: source = my_prometheus.query_range(‘avg (kube_pod_container_resource_limits)’, 1744549131, 1744635531, ‘1h’)

how can I create alert based on Prometheus metrices






@malyadri I think your confusion comes from the fact that you expect that Prometheus data to be stored in the OpenSearch index.

Prometheus data are not stored in the OpenSearch cluster but actively pulled from Prometheus using data source configuration.

The OpenSearch Alerts can be created against existing indices and OpenSearch cluster metrics.

In this scenario you must use Prometheus Alerting to set alerts.

1 Like

Make sure your alerting rule uses a valid PromQL query, proper evaluation interval, and references the correct Prometheus data source used in your dashboard.