Otel Metrics is Empty but Index has documents

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:
I’m running an Otel collector that’s scraping hostmetrics, which I’m feeding to Data Prepper running in Docker and finally into an OpenSearch standalone instance, with OpenSearch Dashboards

I am able to see the metrics being emitted in stdout, and when I check under indexes, I see the relevant index is getting populated. But when I go to Metrics and select Open Telemetry as Metrics source, the Otel Index drop down is empty. Am I missing something?

Configuration:
metrics-pipelins:
workers: 1
source:
otel_metrics_source:
ssl: false
processor:
- otel_metrics:
sink:
- stdout:
- opensearch:
hosts: [“https://observability.home.arpa:9200”]
insecure: true
username: admin
password: *******
index_type: custom
index: otel-metrics

Relevant Logs or Screenshots:

image
image714×942 32.5 KB

{“unit”:“{thread}”,“exemplars”:,“kind”:“GAUGE”,“name”:“system.cpu.load_average.5m”,“flags”:0,“description”:“Average CPU Load over 5 minutes.”,“startTime”:“2024-05-31T16:17:28Z”,“time”:“2024-07-02T02:03:34.483894408Z”,“serviceName”:“home-server”,“value”:0.18,“schemaUrl”:“https://opentelemetry.io/schemas/1.9.0",“instrumentationScope.name”:“otelcol/hostmetricsreceiver/load”,“resource.attributes.host@name”:“rob-laptop”,“resource.attributes.os@type”:“linux”,“resource.attributes.host@id”:“0f15d86e718342a5961a131b9cc63553”,“resource.attributes.service@name”:“home-server”,“instrumentationScope.version”:"0.101.0”}

2

It happen to me as well. Do we know how to solve it?

3

I have the same issue. The index has documents that are visible in the Discover view after creating an index pattern and I can create dashboards of course as well but the Metrics section is empty. A sample metric JSON below.

image
image741×466 23.8 KB

{
  "_index": "metrics2-otel-v1-2024.10.17",
  "_id": "IilTm5IBPMXNRv73aCs8",
  "_version": 1,
  "_score": null,
  "_source": {
    "kind": "SUM",
    "flags": 0,
    "description": "Measure of memory used after the most recent garbage collection event on this pool",
    "serviceName": "mobiletestapp-otel-/home/dmoss",
    "schemaUrl": "https://opentelemetry.io/schemas/1.20.0",
    "isMonotonic": false,
    "unit": "By",
    "aggregationTemporality": "AGGREGATION_TEMPORALITY_CUMULATIVE",
    "exemplars": [],
    "name": "process.runtime.jvm.memory.usage_after_last_gc",
    "startTime": "2024-10-17T15:56:36.552983958Z",
    "attributes": {
      "resource.attributes.process@command_args":     --- REMOVED  ---
      "resource.attributes.telemetry@sdk@language": "java",
      "resource.attributes.host@name": "FR-MossaKowskiD",
      "resource.attributes.process@pid": 1720,
      "resource.attributes.host@arch": "amd64",
      "resource.attributes.process@runtime@description": "Private Build OpenJDK 64-Bit Server VM 17.0.7+7-Ubuntu-0ubuntu122.04.2",
      "resource.attributes.process@executable@path": "/usr/lib/jvm/java-17-openjdk-amd64/bin/java",
      "resource.attributes.telemetry@sdk@version": "1.27.0",
      "resource.attributes.service@name": "mobiletestapp-otel-/home/dmoss",
      "instrumentationScope.version": "1.27.0-alpha",
      "instrumentationScope.name": "io.opentelemetry.runtime-telemetry-java8",
      "metric.attributes.type": "heap",
      "resource.attributes.process@runtime@version": "17.0.7+7-Ubuntu-0ubuntu122.04.2",
      "resource.attributes.telemetry@sdk@name": "opentelemetry",
      "resource.attributes.process@runtime@name": "OpenJDK Runtime Environment",
      "metric.attributes.pool": "G1 Survivor Space",
      "resource.attributes.os@type": "linux",
      "resource.attributes.telemetry@auto@version": "1.27.0",
      "resource.attributes.os@description": "Linux 5.15.153.1-microsoft-standard-WSL2"
    },
    "time": "2024-10-17T16:32:07.292803869Z",
    "value": 40894464
  },
  "fields": {
    "startTime": [
      "2024-10-17T15:56:36.552Z"
    ],
    "time": [
      "2024-10-17T16:32:07.292Z"
    ]
  },
  "sort": [
    1729182727292
  ]
}

This data is being pushed by DataPrepper which is configured as follows:

metrics-pipeline:
  source:
    otel_metrics_source:
      ssl: false
      authentication:
        unauthenticated:
  processor:
    - otel_metrics:
        calculate_histogram_buckets: true
        calculate_exponential_histogram_buckets: true
        exponential_histogram_max_allowed_scale: 10
        flatten_attributes: false
  sink:
    - opensearch:
        hosts: ["http://opensearch-node1:9200"]
        #index: metrics2-otel-v1-%{yyyy.MM.dd}
        index: metrics_index
        insecure: true
        index_type: custom
        #template_file: "templates/ss4o_metrics.json"
        #bulk_size: 4
4

There is a GitHub issue related to this that may help you setting this up. There is a workaround but overall this needs to be fixed properly.

How to Opentelemetry metrics are get in OpenSearch dashboard? · Issue #2172 · opensearch-project/dashboards-observability · GitHub