Hello All,
Issue : aws -collector trying to push data into Data prepper, failing with the message as export failed.
**Environment ** : Open telemetry agent , open telemetry collector and Data prepper are are running on same Ec2 instance (data prepper also is running in Ec2 machine not on docker ).
Observation : Data is pushing from telemetry client to collection from the log , data prepper is connecting to elastic search and creating the index without data , hence connection between open telemetry collection and elastic search has established .
Attaching the opentelemetry collector config file :
receivers:
otlp:
protocols:
exporters:
otlp/data-prepper:
endpoint: localhost:21890
insecure: true
service:
pipelines:
traces:
receivers: [jaeger, otlp, zipkin]
exporters: [otlp/data-prepper]
Attaching the data prepper pipeline .yaml file
entry-pipeline:
buffer:
bounded_blocking:
batch_size: 256
buffer_size: 1024
delay: “100”
sink:
-
pipeline:
name: raw-pipeline
-
pipeline:
name: service-map-pipeline
source:
otel_trace_source:
ssl: false
workers: 4
raw-pipeline:
buffer:
bounded_blocking:
batch_size: 256
buffer_size: 1024
delay: “3000”
prepper:
-
otel_trace_raw_prepper: ~
sink:
-
elasticsearch:
hosts:
- “https://XXXXXamazonaws.com”
password: “XXXX”
trace_analytics_raw: true
username: “XXXX”
source:
pipeline:
name: entry-pipeline
workers: 4
service-map-pipeline:
buffer:
bounded_blocking:
batch_size: 256
buffer_size: 1024
delay: “100”
prepper:
-
service_map_stateful: ~
sink:
-
elasticsearch:
hosts:
- “https://XXXXX.amazonaws.com”
password: “XXXXX”
trace_analytics_service_map: true
username: “XXXXX”
source:
pipeline:
name: entry-pipeline
workers: 4
Telemetry collector Log :
Summary
{2021-07-15 09:20:25.950525164 +0000 UTC m=+0.056752961, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/receiver/otlpreceiver/otlp.go:83, Message:Starting GRPC server on endpoint 0.0.0.0:55680, Stack:}
{2021-07-15 09:20:25.950561626 +0000 UTC m=+0.056789434, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/receiver/otlpreceiver/otlp.go:101, Message:Starting HTTP server on endpoint 0.0.0.0:55681, Stack:}
{2021-07-15 09:20:25.950675979 +0000 UTC m=+0.056903784, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/service/internal/builder/receivers_builder.go:75, Message:Receiver started., Stack:}
{2021-07-15 09:20:25.950702399 +0000 UTC m=+0.056930206, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/service/application.go:197, Message:Everything is ready. Begin running and processing data., Stack:}
{2021-07-15 09:20:50.957987182 +0000 UTC m=+25.064214953, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/exporter/exporterhelper/queued_retry.go:314, Message:Exporting failed. Will retry the request after interval., Stack:}
{2021-07-15 09:20:56.482180469 +0000 UTC m=+30.588408252, Level:info, Caller:go.opentelemetry.io/collector@v0.27.0/exporter/exporterhelper/queued_retry.go:314, Message:Exporting failed. Will retry the request after interval., Stack:}
Data-prepper log :
Summary
2021-07-15T05:34:17,082 [main] INFO com.amazon.dataprepper.pipeline.server.DataPrepperServer - Creating Data Prepper server without TLS
2021-07-15T05:34:20,324 [main] INFO com.amazon.dataprepper.pipeline.Pipeline - Pipeline [raw-pipeline] - Submitting request to initiate the pipeline processing
2021-07-15T05:34:20,325 [main] INFO com.amazon.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server running at :4900
2021-07-15T05:34:20,435 [service-map-pipeline-prepper-worker-3-thread-1] INFO com.amazon.dataprepper.pipeline.ProcessWorker - service-map-pipeline Worker: No records received from buffer
2021-07-15T05:34:20,439 [entry-pipeline-prepper-worker-1-thread-1] INFO com.amazon.dataprepper.pipeline.ProcessWorker - entry-pipeline Worker: No records received from buffer
2021-07-15T05:34:23,332 [raw-pipeline-prepper-worker-5-thread-1] INFO com.amazon.dataprepper.pipeline.ProcessWorker - raw-pipeline Worker: No records received from buffer
Can somebody help to resolve the issue