Dataprepper in rename metrics field (resource.attributes.k8s@node@name) this not working

Data Prepper
1

hello

i am use open-search and dataprepper latest version and opentelemetry

there are Kubernetes metrics get in opentelemetry and export to dataprepper
dataprepper send to opensearch there are metrics in get data like

{
  "_index": "ss4o_metrics-otel-2024.08.30",
  "_id": "l_dlo5EBJVoZess-u0w4",
  "_version": 1,
  "_score": null,
  "_source": {
    "unit": "",
    "exemplars": [],
    "kind": "GAUGE",
    "flags": 0,
    "description": "Whether a container has passed its readiness probe (0 for no, 1 for yes)",
    "startTime": "2024-08-30T12:58:35.776070206Z",
    "attributes": {
      "instrumentationScope.name": "otelcol/k8sclusterreceiver",
      "resource.attributes.container@image@tag": "2.8.0",
      "resource.attributes.k8s@pod@uid": "ffcef16d-8451-4f5d-9089-fdf5807f57f6",
      "resource.attributes.k8s@node@name": "minikube",
      "resource.attributes.k8s@namespace@name": "default",
      "resource.attributes.k8s@deployment@name": "mydataprepper-data-prepper",
      "resource.attributes.k8s@container@name": "data-prepper",
      "metric.attributes.attribute1": 123,
      "resource.attributes.k8s@pod@start_time": "2024-08-30T12:58:05Z",
      "resource.attributes.k8s@pod@name": "mydataprepper-data-prepper-5fcbb978d7-4md57",
      "resource.attributes.container@image@name": "opensearchproject/data-prepper",
      "instrumentationScope.version": "0.106.1"
    },
    "time": "2024-08-30T13:07:15.959422084Z",
    "serviceName": null,
    "value": 1,
    "schemaUrl": "https://opentelemetry.io/schemas/1.18.0",
    "customname1": "k8s.container.ready"
  },
  "fields": {
    "startTime": [
      "2024-08-30T12:58:35.776Z"
    ],
    "time": [
      "2024-08-30T13:07:15.959Z"
    ]
  },
  "sort": [
    1725023235959
  ]
}

there are upper show name to change customname1 this is working this change in dataprepper config but resource.attributes.k8s@node@name in rename and resource.attributes.k8s_node_name this not working

i want to remove attributes in resource.attributes.k8s@node@name and another also in remove ‘@’ add ‘_’ or change filed name
there are upper show name to change customname1 this is working

dataprepper config below

    otel-metrics-pipeline:
      workers: 8
      delay: 3000
      source:
        otel_metrics_source:
          health_check_service: true
          ssl: false     
      processor:
        - otel_metrics:
            calculate_histogram_buckets: true
            calculate_exponential_histogram_buckets: true
            exponential_histogram_max_allowed_scale: 10
            flatten_attributes: false
        - rename_keys:
            entries:          
            - from_key: "name"
              to_key: "customname1"
              overwrite_if_to_key_exists: true
            - from_key: "attributes.resource.attributes.k8s@node@name"
              to_key: "resource.attributes.k8s_node_name"
              overwrite_if_to_key_exists: true           
              
      sink:
        - opensearch:
            hosts: ["https://opensearch-cluster-master.myopensearch.svc.cluster.local:9200"]
            username: "admin"
            password: "TadhakDev01"
            insecure: true
            index_type: custom
            index: ss4o_metrics-otel-%{yyyy.MM.dd}
            bulk_size: 4
2

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.