Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Tested on OpenSearch 2.11.1 & 2.12
Describe the issue:
I am trying to setup prometheus datasource, but not seeing any metrics on both of my clusters running 2.11.1 & 2.12. I am seeing below errors on the opensearch logs but I have already tried with 2 separate prometheus servers with same results. Any help is appreciated.
Configuration:
POST _plugins/_query/_datasources
{
“name” : “test”,
“connector”: “prometheus”,
“properties” : {
“prometheus.uri” : “http://x.x.x.x:9090/”
}
}
Relevant Logs or Screenshots:
[2024-03-05T11:39:00,015][ERROR][o.o.s.p.r.RestPPLQueryAction] [node-1.domain] Error happened during query handling
org.opensearch.sql.prometheus.exceptions.PrometheusClientException: Request to Prometheus is Unsuccessful with code : 405
at org.opensearch.sql.prometheus.client.PrometheusClientImpl.readResponse(PrometheusClientImpl.java:131) ~[prometheus-2.12.0.0.jar:?]
at org.opensearch.sql.prometheus.client.PrometheusClientImpl.getAllMetrics(PrometheusClientImpl.java:81) ~[prometheus-2.12.0.0.jar:?]
at org.opensearch.sql.prometheus.request.system.PrometheusListMetricsRequest.lambda$search$1(PrometheusListMetricsRequest.java:43) ~[prometheus-2.12.0.0.jar:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
at org.opensearch.sql.prometheus.request.system.PrometheusListMetricsRequest.search(PrometheusListMetricsRequest.java:39) ~[prometheus-2.12.0.0.jar:?]
at org.opensearch.sql.prometheus.storage.system.PrometheusSystemTableScan.open(PrometheusSystemTableScan.java:28) ~[prometheus-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.protector.ResourceMonitorPlan.open(ResourceMonitorPlan.java:49) ~[opensearch-2.12.0.0.jar:?]
at java.base/java.util.Collections$SingletonList.forEach(Collections.java:5186) ~[?:?]
at org.opensearch.sql.planner.physical.PhysicalPlan.open(PhysicalPlan.java:29) ~[core-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$0(OpenSearchExecutionEngine.java:49) [opensearch-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.client.OpenSearchNodeClient.schedule(OpenSearchNodeClient.java:177) [opensearch-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.execute(OpenSearchExecutionEngine.java:43) [opensearch-2.12.0.0.jar:?]
at org.opensearch.sql.executor.QueryService.lambda$executePlan$1(QueryService.java:66) [core-2.12.0.0.jar:?]
at java.base/java.util.Optional.ifPresentOrElse(Optional.java:198) [?:?]
at org.opensearch.sql.executor.QueryService.executePlan(QueryService.java:63) [core-2.12.0.0.jar:?]
at org.opensearch.sql.executor.QueryService.execute(QueryService.java:41) [core-2.12.0.0.jar:?]
at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:65) [core-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:31) [opensearch-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:45) [opensearch-2.12.0.0.jar:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.12.0.jar:2.12.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-03-05T11:39:00,153][ERROR][o.o.s.p.r.RestPPLQueryAction] [node-1.domain] Error happened during query handling
org.opensearch.sql.prometheus.exceptions.PrometheusClientException: Prometheus returned unexpected body, please verify your prometheus server setup.
at org.opensearch.sql.prometheus.client.PrometheusClientImpl.readResponse(PrometheusClientImpl.java:120) ~[prometheus-2.12.0.0.jar:?]
at org.opensearch.sql.prometheus.client.PrometheusClientImpl.getAllMetrics(PrometheusClientImpl.java:81) ~[prometheus-2.12.0.0.jar:?]
at org.opensearch.sql.prometheus.request.system.PrometheusListMetricsRequest.lambda$search$1(PrometheusListMetricsRequest.java:43) ~[prometheus-2.12.0.0.jar:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
at org.opensearch.sql.prometheus.request.system.PrometheusListMetricsRequest.search(PrometheusListMetricsRequest.java:39) ~[prometheus-2.12.0.0.jar:?]
at org.opensearch.sql.prometheus.storage.system.PrometheusSystemTableScan.open(PrometheusSystemTableScan.java:28) ~[prometheus-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.protector.ResourceMonitorPlan.open(ResourceMonitorPlan.java:49) ~[opensearch-2.12.0.0.jar:?]
at java.base/java.util.Collections$SingletonList.forEach(Collections.java:5186) ~[?:?]
at org.opensearch.sql.planner.physical.PhysicalPlan.open(PhysicalPlan.java:29) ~[core-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$0(OpenSearchExecutionEngine.java:49) [opensearch-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.client.OpenSearchNodeClient.schedule(OpenSearchNodeClient.java:177) [opensearch-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.execute(OpenSearchExecutionEngine.java:43) [opensearch-2.12.0.0.jar:?]
at org.opensearch.sql.executor.QueryService.lambda$executePlan$1(QueryService.java:66) [core-2.12.0.0.jar:?]
at java.base/java.util.Optional.ifPresentOrElse(Optional.java:198) [?:?]
at org.opensearch.sql.executor.QueryService.executePlan(QueryService.java:63) [core-2.12.0.0.jar:?]
at org.opensearch.sql.executor.QueryService.execute(QueryService.java:41) [core-2.12.0.0.jar:?]
at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:65) [core-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:31) [opensearch-2.12.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:45) [opensearch-2.12.0.0.jar:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.12.0.jar:2.12.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]