Explanation of the first method :
I cloned this GitHub repository : opensearch-project/data-prepper.
Then added the needed services to the already existing docker-compose.yml that has opensearch nodes :
data-prepper:
restart: unless-stopped
container_name: data-prepper
image: opensearchproject/data-prepper:latest
volumes:
- ./data-prepper/examples/trace_analytics_no_ssl.yml:/usr/share/data-prepper/pipelines.yaml
- ./data-prepper/examples/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml
- ./data-prepper/examples/demo/root-ca.pem:/usr/share/data-prepper/root-ca.pem
ports:
- “21890:21890”
networks:
- our-net
depends_on:
- “TisaOS-node1”
otel-collector:
container_name: otel-collector
image: otel/opentelemetry-collector:0.54.0
command: [ “–config=/etc/otel-collector-config.yml” ]
working_dir: “/project”
volumes:
- ${PWD}/:/project
- ./otel-collector-config.yml:/etc/otel-collector-config.yml
- /data-prepper/examples/demo/demo-data-prepper.crt:/etc/demo-data-prepper.crt
ports:
- “4318:4318”
depends_on:
- data-prepper
networks:
- our-net
jaeger-agent:
container_name: jaeger-agent
image: jaegertracing/jaeger-agent:latest
command: [ “–reporter.grpc.host-port=otel-collector:14250” ]
ports:
- “5775:5775/udp”
- “6831:6831/udp”
- “6832:6832/udp”
- “5778:5778/tcp”
networks:
- our-net
jaeger-hot-rod:
image: jaegertracing/example-hotrod:latest
command: [ “all” ]
environment:
- JAEGER_AGENT_HOST=jaeger-agent
- JAEGER_AGENT_PORT=6831
ports:
- “8080-8083:8080-8083”
depends_on:
- jaeger-agent
networks:
- our-net
The configuration of the pipeline : trace_analytics_no_ssl.yml
entry-pipeline:
delay: “100”
source:
otel_trace_source:
ssl: false
sink:
- pipeline:
name: “raw-pipeline”
- pipeline:
name: “service-map-pipeline”
raw-pipeline:
source:
pipeline:
name: “entry-pipeline”
prepper:
- otel_trace_raw_prepper:
sink:
- opensearch:
hosts: [ “http://IP:9200” ]
cert: “/usr/share/data-prepper/root-ca.pem”
username: “admin”
password: “admin”
trace_analytics_raw: true
service-map-pipeline:
delay: “100”
source:
pipeline:
name: “entry-pipeline”
prepper:
- service_map_stateful:
sink:
- opensearch:
hosts: [“http://IP:9200”]
cert: “/usr/share/data-prepper/root-ca.pem”
username: “admin”
password: “admin”
trace_analytics_service_map: true
data-prepper-config.yaml :
ssl: false
Open Telemetry configuration :
receivers:
jaeger:
protocols:
grpc:
exporters:
otlp/2:
endpoint: data-prepper:21890
tls:
insecure: true
insecure_skip_verify: true
logging:
service:
pipelines:
traces:
receivers: [jaeger]
exporters: [logging, otlp/2]
When starting docker-compose, I had errors :
data prepper : no valid pipeline is available for execution
opentelemetry : connection refused