How to create Alerts from Prometheus metrics in OpenSearch Dashboard?

OpenSearch Dashboards
, ,
1

@pablo please help me on this issue

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
version: 2.32.0
appVersion: “2.19.1”

Describe the issue:
When i try the OpenSearch Dashboard alerting for Prometheus Metrics or Metrics Graph there are not working.

I have a Prometheus, OpenSearch and OpenSearch Dashboard.

So i open the OpenSearch Dashboard and i Go to the Dev tools the path of Dev tool is “OpenSearch Dashboard > Management > Dev Tools” and i open the Dev Tools.

And i run this below code for automatic make the Prometheus Data source.

POST _plugins/_query/_datasources 
{
    "name" : "my_prometheus",
    "connector": "prometheus",
    "properties" : {
        "prometheus.uri" : "http://prometheus-server.default.svc.cluster.local:80"
    },
    "allowedRoles" : ["all_access"]
}

When i run this code i go to the Data Sources the path of Data Sources is “OpenSearch Dashboard > Management > Dashboard Management > Data Sources” there are add new Data source name is “my_prometheus”. this is the Image of Data source:

dataso
dataso1912×514 42.3 KB

And than i see the Indexes the path of Indexes is “OpenSearch Dashboard > Management > Index Management > Indexes” and i see one indexes “.ql-Datasource” add one Document like below image :

OpenInde
OpenInde1901×985 143 KB

And than i see the Metrics of “my_prometheus” the path of Metrics “OpenSearch Dashboard > Observability > Metrics” and select the Drop down Metrics Source “Prometheus” and there are lots of Prometheus metrics are open in Available Metrics like below image :

Metrics
Metrics1908×930 105 KB

And than i see the Alerting and create a Monitor in Alerting the path of Create Monitor is “OpenSearch Dashboard > OpenSearch Plugins > Alerting > Monitors > Create Monitor” in the Form “Select Data” i select indexes “.ql-datasources” and i try the query for show all fields of “my_prometheus” but there are not any response see the below image :

alert2
alert21908×991 65.2 KB

Configuration:
I want create Prometheus alerts in dashboard which we like create alerts for logs

Relevant Logs or Screenshots:

2

@pablo

please help me on this

3

@malyadri
Hi, is there any plan to test Grafana which is compatible with your prometheus exporter?

4

@malyadri The main reason of empty response is the user. In your example you’re using admin user. To see the content of the .ql-datasources index you must use superadmin certificate defined in the opensearhc.yml under admin_dn.

See my example.

[opensearch@opensearch-cluster-master-0 ~]$ curl --insecure --cert ./config/kirk.pem --key ./config/kirk-key.pem https://localhost:9200/.ql-datasources/_search?pretty
{
  "took" : 24,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".ql-datasources",
        "_id" : "my_prometheus",
        "_score" : 1.0,
        "_source" : {
          "name" : "my_prometheus",
          "description" : "",
          "connector" : "PROMETHEUS",
          "allowedRoles" : [
            "all_access"
          ],
          "properties" : {
            "prometheus.uri" : "http://prometheus-server.default.svc.cluster.local:80"
          },
          "resultIndex" : "query_execution_result_my_prometheus",
          "status" : "ACTIVE"
        }
      }
    ]
  }

I would take as rule of thumb that any indices starting with “.” must be read by superadmin.

Regarding your alerting, I don’t understand why would you set alerts against a config index. This index holds only data source configuration. No other data will be ingested there.

5

Hi, I am trying to create alerts using Prometheus metrices.

I can be able to see the metrices in dashboard in observability but while creating alerts I couldn’t get the metrices data

I want to create alert :
example query: source = my_prometheus.query_range(‘avg (kube_pod_container_resource_limits)’, 1744549131, 1744635531, ‘1h’)

how can I create alert based on Prometheus metrices

image
image939×306 21.9 KB

image
image977×326 8.7 KB

image
image1916×826 188 KB

image
image1907×822 104 KB

image
image1916×826 107 KB

image
image1265×450 34.7 KB

6

@malyadri I think your confusion comes from the fact that you expect that Prometheus data to be stored in the OpenSearch index.

Prometheus data are not stored in the OpenSearch cluster but actively pulled from Prometheus using data source configuration.

The OpenSearch Alerts can be created against existing indices and OpenSearch cluster metrics.

In this scenario you must use Prometheus Alerting to set alerts.

image
image1178×414 21.8 KB

1 Like
7

Make sure your alerting rule uses a valid PromQL query, proper evaluation interval, and references the correct Prometheus data source used in your dashboard.