Metrics implementation using data-prepper and opensearch

Data Prepper
, ,
1

Hi

I am using data-prepper with opensearch sink for trace-analytics pipeline. I’m wondering if it’s possible to configure metrics in same pipeline, if yes what’s the procedure and architecture?
opensearch cluster: v2.3
data-prepper: latest docker image
otel-collector: v0.70.0

Also one more question: like, in other project, i’ve implemented metrics → prometheus → grafana pipeline, i want to migrate to opensearch & opensearch dashboards, what will be the difference in doing so?

2

Hi @divnoorsingh,

It is possible to use the same pipeline configuration for both trace analytics and metrics. You can add to your trace analytics pipeline configuration with something like the following

metricpipeline:
  source:
    otel_metrics_source:
      ssl: false
      port: 21892
  processor:
    - otel_metrics_raw_processor:
  sink:
    - opensearch:
        hosts: [ "https://localhost:9200" ]
        index: metrics_index
        username: "admin"
        password: "admin"
tracepipeline:
  source:
    otel_trace_source:
      port: 21890
      ssl: false
  ... remaining trace analytics processors and sink like you have configured already

Note that the otel_metrics_source and the otel_trace_source must be running on different ports, and you will just need to point your otel exporters to the correct endpoint and port.

3

Thanks for the reply. I think, i’m eventually on the right track for implementing the same. But one thing that i’m not sure of is there something like auto-instrumentation for emitting metrics on application side, similar to tracing, coz i don’t want to modify the code of different microservices at this point.