# Prometheus metrics visualization display "Data source can't be visualized"

**URL:** <https://forum.opensearch.org/t/prometheus-metrics-visualization-display-data-source-cant-be-visualized/16599>\
**Category:** OpenSearch Dashboards\
**Tags:** troubleshoot\
**Created:** [November 8, 2023, 5:10pm UTC](https://forum.opensearch.org/t/prometheus-metrics-visualization-display-data-source-cant-be-visualized/16599 "2023-11-08T17:10:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Milan](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/milan/32/6391_2.png) [@Milan](https://forum.opensearch.org/u/Milan)\
**Post date:** [November 8, 2023, 5:10pm UTC](https://forum.opensearch.org/t/prometheus-metrics-visualization-display-data-source-cant-be-visualized/16599/1 "2023-11-08T17:10:59Z")

</div>

**Versions** :  
OpenSearch Server: 2.11.0  
Opensearch Dashboard: 2.11.0  
Server OS: Docker Desktop Kubernetes on Windows 11  
Browser: Firefox 120.0b7

**Describe the issue** :  
When creating data source for prometheus [Metrics analytics - OpenSearch documentation](http://opensearch.org/docs/latest/observing-your-data/prometheusmetrics/) and going to _Observability-\> Logs-\> Explorer_ I get the error status saying “Data Source can’t be visualized” . The PPL error is visble in logs (see below)

Any idea what I am doing wrong? When changing opensearch-dashboards image to version “2.10.0” , but keeping opensearch server at 2.11.0, I can visualize the same query.

PPL query: `source = my_prometheus.container_cpu_usage_seconds_total | stats avg(@value) by span(@timestamp,1h)`

**Configuration** :  
prometheus datasource:

```plaintext
POST _plugins/_query/_datasources 
{
    "name" : "my_prometheus",
    "connector": "prometheus",
    "properties" : {
        "prometheus.uri" : "http://prometheus-server.metrics:9090"
    }
}

```

opesearch-dashboards pod:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: &PODNAME monitoring-dashboards
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: *PODNAME
  template:
    metadata:
      labels:
        app.kubernetes.io/component: *PODNAME
    spec:
      containers:
      - name: *PODNAME
        image: opensearchproject/opensearch-dashboards:latest
        env:
          - name: OPENSEARCH_HOSTS
            value: '["http://monitoring-opensearch-server:9200"]'
          - name: DISABLE_SECURITY_DASHBOARDS_PLUGIN
            value: "true"
          - name: SERVER_BASEPATH
            value: /monitoring
          - name: SERVER_REWRITEBASEPATH
            value: "true"

        ports: 
          - name: web
            containerPort: 5601

```

open-dashboards.yml:

```yaml
opensearchDashboards.branding:
  applicationTitle: "System Monitoring Dashboards"
  useExpandedHeader: false
opensearch.hosts: http / /monitoring-opensearch-server:9200 # rewriten in blog
opensearch.ssl.verificationMode: none
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
server.host: '0.0.0.0'

```

server pod:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: &PODNAME monitoring-opensearch-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: *PODNAME
  template:
    metadata:
      labels:
        app.kubernetes.io/component: *PODNAME
    spec:
      volumes:
      - name: *PODNAME
        persistentVolumeClaim:
          claimName: *PODNAME
      containers:
      - name: *PODNAME
        image: opensearchproject/opensearch:latest
        volumeMounts:
          - mountPath: /usr/share/opensearch/data
            name: *PODNAME
        env:
          - name: discovery.type
            value: single-node
          - name: OPENSEARCH_JAVA_OPTS
            value: -Xms512m -Xmx512m
          - name: DISABLE_INSTALL_DEMO_CONFIG
            value: "true"
          - name: DISABLE_SECURITY_PLUGIN
            value: "true"
        ports: 
          - name: api
            containerPort: 9200
          - name: performance
            containerPort: 9600

```

**Relevant Logs or Screenshots** :  
Following output is in dashboards logs:

```auto
    ' "type": "IllegalStateException"\n' +
    ' },\n' +
    ' "status": 503\n' +
    '}',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}
{"type":"response","@timestamp":"2023-11-08T16:55:15Z","tags":[],"pid":1,"method":"post","statusCode":503,"req":{"url":"/api/ppl/search","method":"post","headers":{"host":"localhost","x-request-id":"b7a248df7076687df2493e24c8ac049a","x-real-ip":"192.168.65.3","x-forwarded-for":"192.168.65.3","x-forwarded-host":"localhost","x-forwarded-port":"80","x-forwarded-proto":"http","x-forwarded-scheme":"http","x-scheme":"http","content-length":"291","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0","accept":"*/*","accept-language":"sk,en-US;q=0.8,en;q=0.5,cs;q=0.3","accept-encoding":"gzip, deflate, br","referer":"http://localhost/monitoring/app/observability-logs","content-type":"application/json","osd-version":"2.11.0","osd-xsrf":"osd-fetch","origin":"http://localhost","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","x-forwarded-email":"milan.unger@siemens-healthineers.com","x-forwarded-groups":"admin, monitoring, datascience, sparql","x-forwarded-preferred-username":"Milan Unger","pragma":"no-cache","cache-control":"no-cache"},"remoteAddress":"10.1.0.221","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0","referer":"http://localhost/monitoring/app/observability-logs"},"res":{"statusCode":503,"responseTime":89,"contentLength":9},"message":"POST /api/ppl/search 503 89ms - 9.0B"}
PPL query fetch err: StatusCodeError: Bad Request
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (node:events:525:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  status: 400,
  displayName: 'BadRequest',
  path: '/_plugins/_ppl',
  query: { format: 'viz' },
  body: '{\n' +
    ' "error": {\n' +
    ' "reason": "Invalid Query",\n' +
    ' "details": "invalid to get timestampValue from value of type UNDEFINED",\n' +
    ' "type": "ExpressionEvaluationException"\n' +
    ' },\n' +
    ' "status": 400\n' +
    '}',
  statusCode: 400,
  response: '{\n' +
    ' "error": {\n' +
    ' "reason": "Invalid Query",\n' +
    ' "details": "invalid to get timestampValue from value of type UNDEFINED",\n' +
    ' "type": "ExpressionEvaluationException"\n' +
    ' },\n' +
    ' "status": 400\n' +
    '}',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}

```

Error in WebUI:

 ![os_ppl_2.11.0](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/6/61de1a3b4075c21d7f5e90b5f79ad2d84c87918f.png)

---

<div class="post-metadata">

**Author:** ![Milan](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/milan/32/6391_2.png) [@Milan](https://forum.opensearch.org/u/Milan)\
**Post date:** [November 8, 2023, 5:13pm UTC](https://forum.opensearch.org/t/prometheus-metrics-visualization-display-data-source-cant-be-visualized/16599/2 "2023-11-08T17:13:39Z")

</div>

The same query rendered after changing the image of daschboards to 2.10.0 tag, keeping opensearch-server at 2.11.0

 ![os_ppl_2.10.0](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/9/92d5bab5d6d907e4f23fc744e96da3c704bf9913.png)

---

<div class="post-metadata">

**Author:** ![Gsmitt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gsmitt/32/1718_2.png) [@Gsmitt](https://forum.opensearch.org/u/Gsmitt)\
**Post date:** [November 11, 2023, 1:54am UTC](https://forum.opensearch.org/t/prometheus-metrics-visualization-display-data-source-cant-be-visualized/16599/3 "2023-11-11T01:54:11Z")

</div>

Hey @Milan

I ran into the same issue. Ubuntu-22.0.4, OS/OSD-2.11.0