Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Describe the issue :
Today, OpenSearch supports OpenTelemetry ecosystem through two primary ingestion methods:
OpenTelemetry Collector’s OpenSearch Exporter: Enables direct ingestion of logs and traces.
Request for Feedback
Your feedback will help us understand the path ahead for ingesting telemetry data into OpenSearch:
Current Ingestion Strategy : What is your current approach for sending telemetry data into OpenSearch?Challenges Faced : What challenges or limitations do you encounter with either of the approaches (the exporter vs. native ingestion)?Preferred Method : Based on your current architecture, do you have a preferred ingestion method?
RFC :#18130:
opened 12:54AM - 29 Apr 25 UTC
Meta
untriaged
## Background
OpenSearch is a distributed search and analytics engine supportin… g various use cases, including observability, search, vector store, and security analytics. It integrates seamlessly with OpenSearch Dashboards, a user interface that enables users to discover and visualize OpenSearch data. This integration facilitates the creation of dashboards, alert monitors, anomaly detection, and post-mortem notebooks for root cause analysis.
Today, OpenSearch supports OpenTelemetry ecosystem through two primary ingestion methods:
1. **OpenTelemetry Collector's OpenSearch Exporter**: Enables direct ingestion of logs and traces.
2. **Native OTLP receivers in OpenSearch ingestion**: Incorporates OTLP endpoints to ingest logs, traces, and metrics, supporting production-scale ingestion.
## OpenSearch Exporter for telemetry ingestion
The OpenSearch Exporter was introduced in the [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/opensearchexporter) toward the end of 2023, enabling the ingestion of OTLP-formatted logs and traces into OpenSearch. Since its introduction, the exporter has primarily relied on community contributions for maintenance and feature development.
Currently, the OpenSearch Exporter has been marked as unmaintained. We plan to resume our support for the OpenSearch Exporter. We are seeking input from both OpenTelemetry and OpenSearch communities to evaluate the future direction of ingesting telemetry data into OpenSearch and to actively contribute in this ecosystem.
### **Example Configuration Using OpenSearch Exporter in OpenTelemetry Collector:**
```
exporters:
opensearch:
logs_index: otel
http:
endpoint: "http://opensearch:9200"
service:
pipelines:
logs:
receivers: [...]
processors: [...]
exporters: [opensearch]
```
The OpenSearch exporter is currently utilized in the [OpenTelemetry Demo](https://github.com/open-telemetry/opentelemetry-demo) Application to analyze logs. Here’s the reference to the [OTel collector config](https://github.com/open-telemetry/opentelemetry-demo/blob/1227f208e9d2c666be664738a6f46e584648ae60/src/otel-collector/otelcol-config.yml#L93-L98)
## Native OTLP **receivers** in OpenSearch (Recommended)
OpenSearch provides native support for ingesting telemetry data including logs, metrics, and traces that adhere to the OpenTelemetry Protocol (OTLP). The ingestion platform includes built-in receivers capable of handling both HTTP and gRPC traffic, facilitating seamless integration with existing OTLP exporters in OTel collector.
### Example Snippet using OpenSearch ingestion with OTel OTLP exporter
```
exporters:
otlp/logs:
endpoint: "opensearch-ingestion:21892"
service:
pipelines:
logs:
receivers: [...]
processors: [...]
exporters: [otlp/logs]
```
The OpenSearch ingestion OTLP endpoints are currently used in the [OpenSearch’s fork](https://github.com/opensearch-project/opentelemetry-demo) of the OpenTelemetry Demo Application to analyze logs and traces. Here’s the reference to the [OTel collector config](https://github.com/opensearch-project/opentelemetry-demo/blob/ffe04b7c82639811ad2064aeb8957f9645a58dfa/src/otel-collector/otelcol-config.yml#L96-L106) and the [OpenSearch ingestion config](https://github.com/opensearch-project/opentelemetry-demo/blob/main/src/dataprepper/pipelines.template.yaml).
## Request for Feedback
Your feedback will help us understand the path ahead for ingesting telemetry data into OpenSearch:
1. **Current Ingestion Strategy**: What is your current approach for sending telemetry data into OpenSearch?
2. **Challenges Faced**: What challenges or limitations do you encounter with either of the approaches (the exporter vs. native ingestion)?
3. **Preferred Method**: Based on your current architecture, do you have a preferred ingestion method?
## References
* OpenSearch ingestion OTLP recievers:
* Logs: https://docs.opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-logs-source/
* Metrics: https://docs.opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-metrics-source/
* Traces: https://docs.opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-trace-source/
* OpenTelemetry Collector exporters:
* OTLP gRPC exporter: https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter
* OTLP HTTP exporter: https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter
* OpenSearch exporter: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/opensearchexporter
* Diagram reference from OpenTelemtery Docs : https://opentelemetry.io/docs/
* OpenTelemetry specification for vendor support: https://opentelemetry.io/docs/specs/otel/vendors/#supports-opentelemetry
### Issues
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/35781
It is also open under the OpenTelemetry project GitHub repo under issue #39707:
opened 12:58AM - 29 Apr 25 UTC
discussion needed
exporter/opensearch
## Background
OpenSearch is a distributed search and analytics engine supportin… g various use cases, including observability, search, vector store, and security analytics. It integrates seamlessly with OpenSearch Dashboards, a user interface that enables users to discover and visualize OpenSearch data. This integration facilitates the creation of dashboards, alert monitors, anomaly detection, and post-mortem notebooks for root cause analysis.
Today, OpenSearch supports OpenTelemetry ecosystem through two primary ingestion methods:
1. **OpenTelemetry Collector's OpenSearch Exporter**: Enables direct ingestion of logs and traces.
2. **Native OTLP receivers in OpenSearch ingestion**: Incorporates OTLP endpoints to ingest logs, traces, and metrics, supporting production-scale ingestion.
## OpenSearch Exporter for telemetry ingestion
The OpenSearch Exporter was introduced in the [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/opensearchexporter) toward the end of 2023, enabling the ingestion of OTLP-formatted logs and traces into OpenSearch. Since its introduction, the exporter has primarily relied on community contributions for maintenance and feature development.
Currently, the OpenSearch Exporter has been marked as unmaintained. We plan to resume our support for the OpenSearch Exporter. We are seeking input from both OpenTelemetry and OpenSearch communities to evaluate the future direction of ingesting telemetry data into OpenSearch and to actively contribute in this ecosystem.
### **Example Configuration Using OpenSearch Exporter in OpenTelemetry Collector:**
```
exporters:
opensearch:
logs_index: otel
http:
endpoint: "http://opensearch:9200"
service:
pipelines:
logs:
receivers: [...]
processors: [...]
exporters: [opensearch]
```
The OpenSearch exporter is currently utilized in the [OpenTelemetry Demo](https://github.com/open-telemetry/opentelemetry-demo) Application to analyze logs. Here’s the reference to the [OTel collector config](https://github.com/open-telemetry/opentelemetry-demo/blob/1227f208e9d2c666be664738a6f46e584648ae60/src/otel-collector/otelcol-config.yml#L93-L98)
## Native OTLP **receivers** in OpenSearch (Recommended)
OpenSearch provides native support for ingesting telemetry data including logs, metrics, and traces that adhere to the OpenTelemetry Protocol (OTLP). The ingestion platform includes built-in receivers capable of handling both HTTP and gRPC traffic, facilitating seamless integration with existing OTLP exporters in OTel collector.
### Example Snippet using OpenSearch ingestion with OTel OTLP exporter
```
exporters:
otlp/logs:
endpoint: "opensearch-ingestion:21892"
service:
pipelines:
logs:
receivers: [...]
processors: [...]
exporters: [otlp/logs]
```
The OpenSearch ingestion OTLP endpoints are currently used in the [OpenSearch’s fork](https://github.com/opensearch-project/opentelemetry-demo) of the OpenTelemetry Demo Application to analyze logs and traces. Here’s the reference to the [OTel collector config](https://github.com/opensearch-project/opentelemetry-demo/blob/ffe04b7c82639811ad2064aeb8957f9645a58dfa/src/otel-collector/otelcol-config.yml#L96-L106) and the [OpenSearch ingestion config](https://github.com/opensearch-project/opentelemetry-demo/blob/main/src/dataprepper/pipelines.template.yaml).
## Request for Feedback
Your feedback will help us understand the path ahead for ingesting telemetry data into OpenSearch:
1. **Current Ingestion Strategy**: What is your current approach for sending telemetry data into OpenSearch?
2. **Challenges Faced**: What challenges or limitations do you encounter with either of the approaches (the exporter vs. native ingestion)?
3. **Preferred Method**: Based on your current architecture, do you have a preferred ingestion method?
## References
* OpenSearch ingestion OTLP recievers:
* Logs: https://docs.opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-logs-source/
* Metrics: https://docs.opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-metrics-source/
* Traces: https://docs.opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-trace-source/
* OpenTelemetry Collector exporters:
* OTLP gRPC exporter: https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter
* OTLP HTTP exporter: https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter
* OpenSearch exporter: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/opensearchexporter
* Diagram reference from OpenTelemtery Docs : https://opentelemetry.io/docs/
* OpenTelemetry specification for vendor support: https://opentelemetry.io/docs/specs/otel/vendors/#supports-opentelemetry
### Issues
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/35781
1 Like