Configuring data prepper source to receive events over HTTP/HTTPS

Data Prepper
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch v2.12.0
Data prepper v.2.6.2

Describe the issue:
I’m trying to figure out how to configure data prepper on a kubernetes cluster to receive OTEL events over HTTP/HTTPS.

For this use case, I have a data prepper instance running inside a gke cluster which is supposed to sink events to an opensearch cluster running inside this same cluster. This source is openTelemetry running on a different GCP project.

After reading some documentation (https://opensearch.org/docs/latest/data-prepper/#source), I’m assuming that I can somehow expose dataprepper to receive these events from openTelemetry.

I tried following this tutorial (Demystify OpenTelemetry, with OpenSearch as a single plane of glass. | by Erik Åström | Medium) but after deploying data-prepper with a service.yaml similar to the one appended in the doc, I noticed data-prepper wasn’t listening on every port I specified. It was only listening on 4900 and 21892 which I think were for API and Logs
respectively.

I’m not sure where to send trace and metric events.

Configuration:
I was hoping someone would share with me an example configuration for this use case or a similar one.

ty,

2

When you use the otel logs source, it by default listens on 21892 and by default has no path, so / is used. It’s a gRPC service.

otel_logs_source - OpenSearch Documentation is the more complete link to follow to configure the otel logs source.