Data Prepper & OpenTelemetry

All,
I figured I would share my pain and suffering :laughing: I must admit this was a good challenge for someone who has not worked with Trace Analysis/Data Prepper/Fluent bit/OpenTelemetry before. Took me few to understand that its not one size fits all here. What order the applications to be connected to what other application. I was treating it like Nxlog and LogStash. There are some familiarities but overall, what I really need to understand was what Protocol goes to what port along with what environment variables does what and how to configure these in the real world. Im not using Docker and did see the statement abount using dockers to make life easy but unfortunately we have no need to use docker for production setup.

For example:

  • otel_trace_source 21890
  • Data Prepper HTTP 2021
  • OTLP 4317
  • OTLPHTTP 4318

After researching I found out I need Fluent bit to send data to OpenTelemetry then send it to Data Prepper which sends that data to OpenSearch.
I wish there were a few less steps in this, mainly because this was where the major confusion was. Still donโ€™t know what Iโ€™m doing but I was able to make it work. The following was my configuration made and any advice or suggestions is more than welcome.

Configuration_Files
############# FLUENT-BIT ###############
[SERVICE]
  Flush           5
  Daemon          off
  Log_Level       debug
[INPUT]
  name                  tail
  refresh_interval      5
  path                  /var/log/apache2/*.log
  read_from_head        true
  tag apache2
[INPUT]
  Name                 node_exporter_metrics
  Tag                  node_metrics
  Scrape_interval      2
[INPUT]
  Name                 event_type
  Type                 traces
[OUTPUT]
  Name                 opentelemetry
  Match                *
  Host                 localhost
  Port                 4318
  Metrics_uri          /v1/metrics
  Logs_uri             /v1/logs
  Traces_uri           /v1/traces
  Log_response_payload True
  Tls                  off
  Tls.verify           Off
  # add user-defined labels
  add_label            app fluent-bit
  add_label            color blue

  
  

################### OTELCOL #############
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlphttp:
    endpoint: localhost:4318
    tls:
      insecure: true
  otlp/data-prepper:
    endpoint: localhost:21890
    tls:
      insecure: true
  otlp:
    endpoint: localhost:4317
    tls:
      insecure: true
extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/data-prepper]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/data-prepper]



##############  DATA-PREPPER ###########################
entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - otel_trace_raw:
  sink:
    - opensearch:
        hosts: [ "https://domain.com:9200" ]
        cert:  /opt/opensearch-data-prepper/config/root-ca.pem
        username: "admin"
        password: "changeit"
        index_type: trace-analytics-raw
        max_retries: 20
        bulk_size: 4
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: [ "https://domain.com:9200" ]
        cert: /opt/opensearch-data-prepper/config/root-ca.pem
        username: "admin"
        password: "changeit"
        max_retries: 20
        bulk_size: 4
        index_type: trace-analytics-service-map

Results:



Resources Used:

Hope this might help someone in the future :+1: