Metric with prometheus on Opensearch

OpenSearch
1

Versions

Opensearch 3.2
Opensearch-dashboards 3.0

Hello I’m trying to integrate prometheus metrics into Opensearch, but I’m having some issue on it.
I’ve configured the datasource as follow:

[
{
“name”: “my-prometheus”,
“description”: “”,
“connector”: “PROMETHEUS”,
“allowedRoles”: ,
“properties”: {
“prometheus.auth.type”: “basicauth”,
“prometheus.uri”: “http://192.168.122.189:9090/”
},
“resultIndex”: “query_execution_result_my-prometheus”,
“status”: “ACTIVE”
}
]

I get the metrics on Observability > Metrics correctly, but I cannot procede to create a working dashboards.
The prometheus job is job=“kube-state-metrics”.
I would to use VisBuilder, or something similar, because the only working is PPL and it have few options on it.

for example the following PPL get error, for missing Calcite

SOURCE = my-prometheus.kube_pod_container_resource_requests
| where env = “benchmark” and resource = “memory”
| stats sum(@value) as requested
| eval allocatable = [
SOURCE = my-prometheus.kube_node_status_allocatable
| where env = “benchmark” and resource = “memory”
| stats sum(@value) as total_allocatable
| fields total_allocatable
]
| eval used_percent = (requested * 100.0) / allocatable
| fields used_percent

so I cannot join two query on same visualization.

Could you please route me to correct way to configure the kube-state-metrics from prometheus to opensearch?

Thanks

2

@pido The following error was produced due to a missing configuration in your opensearch.yml

image
image1430×812 43.8 KB

Did you follow the OpenSearch documentation and set plugins.calcite.enable to true?
The default value is false.

3

Hello,

Yes I’ve configured the calcite plugin on cluster using this command:

PUT _cluster/settings
{
“persistent”: {
“plugins.calcite.enabled”: true
}
}
but I get this error on PPL

{ “error”: { “reason”: “There was internal problem at backend”, “details”: “Table ‘my-prometheus.kube_pod_container_resource_requests’ not found”, “type”: “CalciteException” }, “status”: 500 }

using this query

source = my-prometheus.kube_pod_container_resource_requests
| where env = “apay-benchmark” and resource = “memory”
| stats sum(@value) as memory_requested_bytes by span(@timestamp, 1m)
| eval memory_requested_GiB = memory_requested_bytes / (1024 * 1024 * 1024)
| join [
source = my-prometheus.kube_node_status_allocatable
| where resource = “memory”
| stats sum(@value) as memory_allocatable_bytes by span(@timestamp, 1m)
| eval memory_allocatable_GiB = memory_allocatable_bytes / (1024 * 1024 * 1024)
]
| eval memory_requested_pct = (memory_requested_GiB / memory_allocatable_GiB) * 100
| fields span(@timestamp,1m), memory_requested_pct