All,
I figured I would share my pain and suffering 
I must admit this was a good challenge for someone who has not worked with Trace Analysis/Data Prepper/Fluent bit/OpenTelemetry before. Took me few to understand that its not one size fits all here. What order the applications to be connected to what other application. I was treating it like Nxlog and LogStash. There are some familiarities but overall, what I really need to understand was what Protocol goes to what port along with what environment variables does what and how to configure these in the real world. Im not using Docker and did see the statement abount using dockers to make life easy but unfortunately we have no need to use docker for production setup.
For example:
- otel_trace_source 21890
- Data Prepper HTTP 2021
- OTLP 4317
- OTLPHTTP 4318
After researching I found out I need Fluent bit to send data to OpenTelemetry then send it to Data Prepper which sends that data to OpenSearch.
I wish there were a few less steps in this, mainly because this was where the major confusion was. Still don’t know what I’m doing but I was able to make it work. The following was my configuration made and any advice or suggestions is more than welcome.
Configuration_Files
############# FLUENT-BIT ###############
[SERVICE]
Flush 5
Daemon off
Log_Level debug
[INPUT]
name tail
refresh_interval 5
path /var/log/apache2/*.log
read_from_head true
tag apache2
[INPUT]
Name node_exporter_metrics
Tag node_metrics
Scrape_interval 2
[INPUT]
Name event_type
Type traces
[OUTPUT]
Name opentelemetry
Match *
Host localhost
Port 4318
Metrics_uri /v1/metrics
Logs_uri /v1/logs
Traces_uri /v1/traces
Log_response_payload True
Tls off
Tls.verify Off
# add user-defined labels
add_label app fluent-bit
add_label color blue
################### OTELCOL #############
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
exporters:
otlphttp:
endpoint: localhost:4318
tls:
insecure: true
otlp/data-prepper:
endpoint: localhost:21890
tls:
insecure: true
otlp:
endpoint: localhost:4317
tls:
insecure: true
extensions:
health_check:
pprof:
zpages:
service:
extensions: [health_check, pprof, zpages]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/data-prepper]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp/data-prepper]
############## DATA-PREPPER ###########################
entry-pipeline:
delay: "100"
source:
otel_trace_source:
ssl: false
sink:
- pipeline:
name: "raw-pipeline"
- pipeline:
name: "service-map-pipeline"
raw-pipeline:
source:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
sink:
- opensearch:
hosts: [ "https://domain.com:9200" ]
cert: /opt/opensearch-data-prepper/config/root-ca.pem
username: "admin"
password: "changeit"
index_type: trace-analytics-raw
max_retries: 20
bulk_size: 4
service-map-pipeline:
delay: "100"
source:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
sink:
- opensearch:
hosts: [ "https://domain.com:9200" ]
cert: /opt/opensearch-data-prepper/config/root-ca.pem
username: "admin"
password: "changeit"
max_retries: 20
bulk_size: 4
index_type: trace-analytics-service-map
Results:
Resources Used:
Hope this might help someone in the future 