OpenSearch Observability Forum

Plugins Observability
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:Hi Team,

We are monitoring the OpenSearch Kubernetes Operator using Prometheus.

The operator exposes Go runtime metrics and controller-runtime metrics (reconcile, workqueue, leader election, etc.), but we couldn’t find any HTTP server metrics such as:

  • http_requests_total
  • http_request_duration_seconds
  • http_request_duration_seconds_bucket
  • http_response_status_total

These metrics are required to build standard HTTP availability and latency SLOs.

Configuration:OpenSearch Kubernetes Operator: v3.0.3 Prometheus: kube-prometheus-stack OpenSearch Image: docker.io/opensearchproject/opensearch-operator:3.0.0-alpha Kubernetes: AKS OS: Ubuntu 24.04 and image: docker.io/opensearchproject/opensearch:3.4.0

Relevant Logs or Screenshots:

2

Hi @Amitjain0142

Looking at the code it appears the operator does not currently expose HTTP server metrics (http_requests_total, http_request_duration_seconds, http_response_status_total). These are not instrumented in the current codebase.

What is available today that can partially serve as SLO signals:

  • controller_runtime_reconcile_errors_total / controller_runtime_reconcile_total → reconcile error rate, a rough proxy for operator availability
  • opensearch_operator_cluster_health → cluster health state (0=green, 1=yellow, 2=red, -1=unknown)
  • opensearch_operator_cluster_reconcile_error{reconciler="..."} → per-reconciler error counts

These reflect operator and cluster functional health but do not cover HTTP-level latency or status code distributions.

For the full HTTP metrics you need, the operator would require an instrumented HTTP transport using promhttp.InstrumentRoundTripperCounter / InstrumentRoundTripperDuration. The injection point already exists in the code (WithOSClientTransport), it just hasn’t been wired up yet.

I’d recommend opening a GitHub issue on the opensearch-k8s-operator repository requesting HTTP metrics instrumentation. A PR adding this would be a welcome contribution.

Hope this helps