Opentelemetry collector not able to export data into data prepper (Message:Exporting failed.)

Hello All,

Issue : aws -collector trying to push data into Data prepper, failing with the message as export failed.

**Environment ** : Open telemetry agent , open telemetry collector and Data prepper are are running on same Ec2 instance (data prepper also is running in Ec2 machine not on docker ).

Observation : Data is pushing from telemetry client to collection from the log , data prepper is connecting to elastic search and creating the index without data , hence connection between open telemetry collection and elastic search has established .

Attaching the opentelemetry collector config file :
receivers:
otlp:
protocols:
exporters:
otlp/data-prepper:
endpoint: localhost:21890
insecure: true
service:
pipelines:
traces:
receivers: [jaeger, otlp, zipkin]
exporters: [otlp/data-prepper]

Attaching the data prepper pipeline .yaml file
entry-pipeline:
buffer:
bounded_blocking:
batch_size: 256
buffer_size: 1024
delay: “100”
sink:
-
pipeline:
name: raw-pipeline
-
pipeline:
name: service-map-pipeline
source:
otel_trace_source:
ssl: false
workers: 4
raw-pipeline:
buffer:
bounded_blocking:
batch_size: 256
buffer_size: 1024
delay: “3000”
prepper:
-
otel_trace_raw_prepper: ~
sink:
-
elasticsearch:
hosts:
- “https://XXXXXamazonaws.com
password: “XXXX”
trace_analytics_raw: true
username: “XXXX”
source:
pipeline:
name: entry-pipeline
workers: 4
service-map-pipeline:
buffer:
bounded_blocking:
batch_size: 256
buffer_size: 1024
delay: “100”
prepper:
-
service_map_stateful: ~
sink:
-
elasticsearch:
hosts:
- “https://XXXXX.amazonaws.com
password: “XXXXX”
trace_analytics_service_map: true
username: “XXXXX”
source:
pipeline:
name: entry-pipeline
workers: 4

Telemetry collector Log :

Summary

{2021-07-15 09:20:25.950525164 +0000 UTC m=+0.056752961, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/receiver/otlpreceiver/otlp.go:83, Message:Starting GRPC server on endpoint 0.0.0.0:55680, Stack:}
{2021-07-15 09:20:25.950561626 +0000 UTC m=+0.056789434, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/receiver/otlpreceiver/otlp.go:101, Message:Starting HTTP server on endpoint 0.0.0.0:55681, Stack:}
{2021-07-15 09:20:25.950675979 +0000 UTC m=+0.056903784, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/service/internal/builder/receivers_builder.go:75, Message:Receiver started., Stack:}
{2021-07-15 09:20:25.950702399 +0000 UTC m=+0.056930206, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/service/application.go:197, Message:Everything is ready. Begin running and processing data., Stack:}
{2021-07-15 09:20:50.957987182 +0000 UTC m=+25.064214953, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/exporter/exporterhelper/queued_retry.go:314, Message:Exporting failed. Will retry the request after interval., Stack:}
{2021-07-15 09:20:56.482180469 +0000 UTC m=+30.588408252, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/exporter/exporterhelper/queued_retry.go:314, Message:Exporting failed. Will retry the request after interval., Stack:}


Data-prepper log :

Summary

2021-07-15T05:34:17,082 [main] INFO com.amazon.dataprepper.pipeline.server.DataPrepperServer - Creating Data Prepper server without TLS
2021-07-15T05:34:20,324 [main] INFO com.amazon.dataprepper.pipeline.Pipeline - Pipeline [raw-pipeline] - Submitting request to initiate the pipeline processing
2021-07-15T05:34:20,325 [main] INFO com.amazon.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server running at :4900
2021-07-15T05:34:20,435 [service-map-pipeline-prepper-worker-3-thread-1] INFO com.amazon.dataprepper.pipeline.ProcessWorker - service-map-pipeline Worker: No records received from buffer
2021-07-15T05:34:20,439 [entry-pipeline-prepper-worker-1-thread-1] INFO com.amazon.dataprepper.pipeline.ProcessWorker - entry-pipeline Worker: No records received from buffer
2021-07-15T05:34:23,332 [raw-pipeline-prepper-worker-5-thread-1] INFO com.amazon.dataprepper.pipeline.ProcessWorker - raw-pipeline Worker: No records received from buffer

Can somebody help to resolve the issue

Hi tiju, thanks for submitting this issue and attaching the logs. The Data Prepper logs look fine, so I’m wondering why the OTel collector can’t reach Data Prepper.

Two questions to get started:

  1. I see Data Prepper is not running in a docker container, but is the OTel collector in a docker container?
  2. Can you make a localhost request to Data Prepper manually and get results? Try curl localhost:4900/list to see if that succeeds or not. The response should be a small JSON payload of the pipeline names defined in the YAML file.

Thank you wrijeff for the response
Answers for your points :
Point 1: No Otlel collector is not running in docker .
point 2 : yes getting the response, as show below
{“pipelines”:[{“name”:“entry-pipeline”},{“name”:“service-map-pipeline”},{“name”:“raw-pipeline”}]}

Hey @tiju

Were you able to resolve this issue? Facing similar in my case:((