Versions :
OpenSearch Server: 2.11.0
Opensearch Dashboard: 2.11.0
Server OS: Docker Desktop Kubernetes on Windows 11
Browser: Firefox 120.0b7
Describe the issue:
When creating data source for prometheus Metrics analytics - OpenSearch documentation and going to Observability-> Logs-> Explorer I get the error status saying “Data Source can’t be visualized” . The PPL error is visble in logs (see below)
Any idea what I am doing wrong? When changing opensearch-dashboards image to version “2.10.0” , but keeping opensearch server at 2.11.0, I can visualize the same query.
PPL query: source = my_prometheus.container_cpu_usage_seconds_total | stats avg(@value) by span(@timestamp,1h)
Configuration:
prometheus datasource:
POST _plugins/_query/_datasources
{
"name" : "my_prometheus",
"connector": "prometheus",
"properties" : {
"prometheus.uri" : "http://prometheus-server.metrics:9090"
}
}
opesearch-dashboards pod:
apiVersion: apps/v1
kind: Deployment
metadata:
name: &PODNAME monitoring-dashboards
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: *PODNAME
template:
metadata:
labels:
app.kubernetes.io/component: *PODNAME
spec:
containers:
- name: *PODNAME
image: opensearchproject/opensearch-dashboards:latest
env:
- name: OPENSEARCH_HOSTS
value: '["http://monitoring-opensearch-server:9200"]'
- name: DISABLE_SECURITY_DASHBOARDS_PLUGIN
value: "true"
- name: SERVER_BASEPATH
value: /monitoring
- name: SERVER_REWRITEBASEPATH
value: "true"
ports:
- name: web
containerPort: 5601
open-dashboards.yml:
opensearchDashboards.branding:
applicationTitle: "System Monitoring Dashboards"
useExpandedHeader: false
opensearch.hosts: http / /monitoring-opensearch-server:9200 # rewriten in blog
opensearch.ssl.verificationMode: none
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
server.host: '0.0.0.0'
server pod:
apiVersion: apps/v1
kind: Deployment
metadata:
name: &PODNAME monitoring-opensearch-server
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: *PODNAME
template:
metadata:
labels:
app.kubernetes.io/component: *PODNAME
spec:
volumes:
- name: *PODNAME
persistentVolumeClaim:
claimName: *PODNAME
containers:
- name: *PODNAME
image: opensearchproject/opensearch:latest
volumeMounts:
- mountPath: /usr/share/opensearch/data
name: *PODNAME
env:
- name: discovery.type
value: single-node
- name: OPENSEARCH_JAVA_OPTS
value: -Xms512m -Xmx512m
- name: DISABLE_INSTALL_DEMO_CONFIG
value: "true"
- name: DISABLE_SECURITY_PLUGIN
value: "true"
ports:
- name: api
containerPort: 9200
- name: performance
containerPort: 9600
Relevant Logs or Screenshots:
Following output is in dashboards logs:
' "type": "IllegalStateException"\n' +
' },\n' +
' "status": 503\n' +
'}',
toString: [Function (anonymous)],
toJSON: [Function (anonymous)]
}
{"type":"response","@timestamp":"2023-11-08T16:55:15Z","tags":[],"pid":1,"method":"post","statusCode":503,"req":{"url":"/api/ppl/search","method":"post","headers":{"host":"localhost","x-request-id":"b7a248df7076687df2493e24c8ac049a","x-real-ip":"192.168.65.3","x-forwarded-for":"192.168.65.3","x-forwarded-host":"localhost","x-forwarded-port":"80","x-forwarded-proto":"http","x-forwarded-scheme":"http","x-scheme":"http","content-length":"291","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0","accept":"*/*","accept-language":"sk,en-US;q=0.8,en;q=0.5,cs;q=0.3","accept-encoding":"gzip, deflate, br","referer":"http://localhost/monitoring/app/observability-logs","content-type":"application/json","osd-version":"2.11.0","osd-xsrf":"osd-fetch","origin":"http://localhost","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","x-forwarded-email":"milan.unger@siemens-healthineers.com","x-forwarded-groups":"admin, monitoring, datascience, sparql","x-forwarded-preferred-username":"Milan Unger","pragma":"no-cache","cache-control":"no-cache"},"remoteAddress":"10.1.0.221","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0","referer":"http://localhost/monitoring/app/observability-logs"},"res":{"statusCode":503,"responseTime":89,"contentLength":9},"message":"POST /api/ppl/search 503 89ms - 9.0B"}
PPL query fetch err: StatusCodeError: Bad Request
at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
at IncomingMessage.emit (node:events:525:35)
at IncomingMessage.emit (node:domain:489:12)
at endReadableNT (node:internal/streams/readable:1359:12)
at processTicksAndRejections (node:internal/process/task_queues:82:21) {
status: 400,
displayName: 'BadRequest',
path: '/_plugins/_ppl',
query: { format: 'viz' },
body: '{\n' +
' "error": {\n' +
' "reason": "Invalid Query",\n' +
' "details": "invalid to get timestampValue from value of type UNDEFINED",\n' +
' "type": "ExpressionEvaluationException"\n' +
' },\n' +
' "status": 400\n' +
'}',
statusCode: 400,
response: '{\n' +
' "error": {\n' +
' "reason": "Invalid Query",\n' +
' "details": "invalid to get timestampValue from value of type UNDEFINED",\n' +
' "type": "ExpressionEvaluationException"\n' +
' },\n' +
' "status": 400\n' +
'}',
toString: [Function (anonymous)],
toJSON: [Function (anonymous)]
}
Error in WebUI:
