OpenTelemetry integration for all signals (log,metrics and traces)

Data Prepper
#1

Hello,

We are using the OpenSearch stack and we would adopt the OpenTelemetry standard in order to handle all Observability concerns: metrics, traces and logs.

At the moment, we have successfully configured the OpenTelemetry collector in order to export Traces to Dataprepper.

About Logs, I know that actually, I could use Fluentbit + Dataprepper, but I would stay as simple as possible and avoid using other collectors (like Fluentbit)

What about metrics and (direct) logs?
Will DataPrepper handle also these signals? If yes, what is the Roadmap?
Is it right that, in the future, OpenSearch will natively support OpenTelemetry standards without using DataPrepper?

Thank you

#2

Hello @ziomill,

Thank you for raising this question. First, a community member has submitted a contribution to support metrics ingestion. It is available in this PR: Support OpenTelemetry Metrics by kmssap · Pull Request #1154 · opensearch-project/data-prepper · GitHub. We expect to release it in Data Prepper 1.4.

Regarding logs, we currently do only support ingest over HTTP using the Fluentbit format. We do expect to support OTel logs in the future, but have no concrete plans right not. The OTel Logs model is still experimental unlike trace and metrics which are more stable. While we do not have current plans to supports logs, we are happy to take community contributions similar to the metrics PR.

#3

Hello @dlv
For now, I will take a look at the metrics contribution.
Thank you!