Using otel metrics source to data prepper

Data Prepper
1

Hello everyone,

I am trying to use data-prepper with otel metrics as source but so far I am not able to successfully do it. I have tried both secure(ssl)/insecure. My data prepper pipeline config in insecure mode is

simple-sample-pipeline:
  workers: 2
  delay: "5000"
  source:
    otel_metrics_source:
      ssl: false
      authentication:
        unauthenticated:
  processor:
    - string_converter:
        upper_case: true
  sink:
    - stdout:

And the otel config has exporters section same as mentioned in data-prepper/trace_setup.md at main · opendistro-for-elasticsearch/data-prepper · GitHub

exporters:
  otlp/data-prepper:
    endpoint: 127.0.0.1:21891
    insecure: true

But I get the error that “insecure” is not valid configuration

Error: failed to get config: cannot unmarshal the configuration: error reading exporters configuration for "otlp/data-prepper": 1 error(s) decoding:

* '' has invalid keys: insecure

2022/09/20 18:11:52 application run finished with error: failed to get config: cannot unmarshal the configuration: error reading exporters configuration for "otlp/data-prepper": 1 error(s) decoding:

* '' has invalid keys: insecure

If I remove “insecure: true”, I am seeing this error, which clearly means it is trying secure(ssl) connection. How can get “insecure” connection established between otel and dataprepper?


Err: connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

Secondly, what exact configs should be used on both otel and dataprepper side, if I want to use secure connection? How can I get/generate test certificates/keys to use for this purpose?

Thanks!
Krishna

2

I found that the correct config for exporter on OTEL/ADOT is

exporters:
  otlp/data-prepper:
    endpoint: 127.0.0.1:21891
    tls:
      insecure: true

With that change, I see that OTEL/ADOT and data-prepper are connected in insecure mode.
But when I start aws-otel-collector (public.ecr.aws/aws-observability/aws-otel-collector:latest), otel-sample-app (public.ecr.aws/aws-otel-test/aws-otel-java-spark:latest), and traffic-generator (ellerbrock/alpine-bash-curl-ssl:latest), I am seeing the following stack trace on the data-prepper side

 java.lang.ClassCastException: class io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest cannot be cast to class com.amazon.dataprepper.model.event.Event (io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest and com.amazon.dataprepper.model.event.Event are in unnamed module of loader 'app')
        at com.amazon.dataprepper.plugins.prepper.StringPrepper.execute(StringPrepper.java:69) ~[data-prepper.jar:2.0.0-SNAPSHOT]
        at com.amazon.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:62) ~[data-prepper.jar:2.0.0-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) ~[?:?]

Any idea what’s wrong

3

Hello kkondaka, I looked into the issue you are experiencing. The string_converter processor is incompatable with the otel_metrics_source source. If you remove the string_converter processor everything should work as expected.

I’ve opened a github issue (1791) to improve the otel_metrics_source documentation.

If you have a good use case for using otel_metrics_source and string_converter opening an issue on github is a great way to let the community know.

Sorry, normally I would post some helpful links but the forum is having an issue allowing me to reference github.

4

Thanks! That makes sense. Will modify to not use the string converter.