Prometheus not able to scrape metrics on pod

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OS: 2.10.0

Describe the issue:
Hi All,
I have deployed opensearch using helm chart. now I am trying to scrape the opensearch jvm metrics from the pod. and installed the prometheus-exporter plugin with below config.

plugins:
enabled: true
installList:
- https://artifacts.opensearch.org/releases/plugins/repository-gcs/2.10.0/repository-gcs-2.10.0.zip
- https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.10.0.0/prometheus-exporter-2.10.0.0.zip

when i curl the api i am getting the response
curl -u $USER:$PASSWORD https://opensearch.myorg.net/_prometheus/metrics

then when i check at Prometheus end i dont get any metrics, later found that Prometheus is not able to scrape because of the tls enabled.
then tried the below steps, but still i am not getting the metrics.

below is my configs

Configuration:

role

    # prometheus role to scrape metrics.
    prometheus:
      cluster_permissions:
        - "cluster:monitor/prometheus/metrics"
        - "cluster:monitor/health"
        - "cluster:monitor/state"
        - "cluster:monitor/nodes/info"
        - "cluster:monitor/nodes/stats"
      index_permissions:
        - index_patterns:
          - "*"
          allowed_actions:
          - "indices:monitor/stats"

role mapping

    prometheus:
      reserved: false
      users:
      - "*"

ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: opensearch-metrics
namespace: logquery
spec:
endpoints:
- path: /_prometheus/metrics
port: http
scheme: https
tlsConfig:
insecureSkipVerify: true
# Used for authentication
ca:
secret:
name: tls-for-opensearch
key: tls.crt
cert:
secret:
name: tls-for-opensearch
key: tls.crt
keySecret:
name: tls-for-opensearch
key: tls.key
selector:
matchLabels:
app.kubernetes.io/name: opensearch

Pod annotation

podAnnotations:
prometheus.io/scrape: “true”
prometheus.io/path: “/_prometheus/metrics”
prometheus.io/port: “9200”

Can anyone help here.
Thanks in advance.

2

Hi,

thanks for the question. I will need to write a proper tutorial on this. Give me some time (as this includes not only OpenSearch configuration but also Prometheus and K8s configurations).

I opened a new ticket for it: [Tutorial] Write complete tutorial on how to setup OpenSearch with the plugin in K8s and Prometheus craping it · Issue #240 · Aiven-Open/prometheus-exporter-plugin-for-opensearch · GitHub

Regards,
Lukáš

1 Like