This is from Send trace data to opensearch - #2 by issrae by @ issrae
Hello,
I’m trying to send the trace data of a wordpress website to opensearch.
I started by sending trace data using wordpress plugin decalog to Jaeger (all in one deployment using docker) and got to visualize them in Jaeger UI.
Now that I’m trying to send them from there to opensearch, I cloned the repository mentioned in the documentation opensearch-project/data-prepper , and added the otel collector and data prepper services to my already existing opensearch service in docker-compose. But I’m having some difficulties to send it from here to opentelemetry to data prepper and opensearch.
And so I did another deployment that is much minimized :
I’ve seen examples for elasticsearch using Jaeger with Opentelemetry as a backend, so I changed them to fit my case as the follwing :
docker run --rm -it -v ${PWD}:/config
-e SPAN_STORAGE_TYPE=elasticsearch opensearchproject/opensearch
jaegertracing/jaeger-opentelemetry-collector
–config-file=/config/config.yaml
–es.server-urls=http://IP:9200
–es.num-shards=3
With config.yaml :
exporters:
otlp/data-prepper:
endpoint: http://IP:9200
insecure: true
processors:
attributes:
actions:
-key: user
action: delete
service:
pipelines:
traces:
processors: [attributes]
But this does not seem to work either, I get the following error :
./opensearch-docker-entrypoint.sh: line 140: /usr/share/opensearch/jaegertracing/jaeger-opentelemetry-collector: No such file or directory
Thank you very much in advance.
Explanation of the first method :
I cloned this GitHub repository : opensearch-project/data-prepper.
Then added the needed services to the already existing docker-compose.yml that has opensearch nodes :
data-prepper:
restart: unless-stopped
container_name: data-prepper
image: opensearchproject/data-prepper:latest
volumes:
- ./data-prepper/examples/trace_analytics_no_ssl.yml:/usr/share/data-prepper/pipelines.yaml
- ./data-prepper/examples/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml
- ./data-prepper/examples/demo/root-ca.pem:/usr/share/data-prepper/root-ca.pem
ports: - “21890:21890”
networks: - our-net
depends_on: - “TisaOS-node1”
otel-collector:
container_name: otel-collector
image: otel/opentelemetry-collector:0.54.0
command: [ “–config=/etc/otel-collector-config.yml” ]
working_dir: “/project”
volumes: - ${PWD}/:/project
- ./otel-collector-config.yml:/etc/otel-collector-config.yml
- /data-prepper/examples/demo/demo-data-prepper.crt:/etc/demo-data-prepper.crt
ports: - “4318:4318”
depends_on: - data-prepper
networks: - our-net
jaeger-agent:
container_name: jaeger-agent
image: jaegertracing/jaeger-agent:latest
command: [ “–reporter.grpc.host-port=otel-collector:14250” ]
ports: - “5775:5775/udp”
- “6831:6831/udp”
- “6832:6832/udp”
- “5778:5778/tcp”
networks: - our-net
jaeger-hot-rod:
image: jaegertracing/example-hotrod:latest
command: [ “all” ]
environment: - JAEGER_AGENT_HOST=jaeger-agent
- JAEGER_AGENT_PORT=6831
ports: - “8080-8083:8080-8083”
depends_on: - jaeger-agent
networks: - our-net
The configuration of the pipeline : trace_analytics_no_ssl.yml
entry-pipeline:
delay: “100”
source:
otel_trace_source:
ssl: false
sink:
- pipeline:
name: “raw-pipeline” - pipeline:
name: “service-map-pipeline”
raw-pipeline:
source:
pipeline:
name: “entry-pipeline”
prepper: - otel_trace_raw_prepper:
sink: - opensearch:
hosts: [ “http://IP:9200” ]
cert: “/usr/share/data-prepper/root-ca.pem”
username: “admin”
password: “admin”
trace_analytics_raw: true
service-map-pipeline:
delay: “100”
source:
pipeline:
name: “entry-pipeline”
prepper: - service_map_stateful:
sink: - opensearch:
hosts: [“http://IP:9200”]
cert: “/usr/share/data-prepper/root-ca.pem”
username: “admin”
password: “admin”
trace_analytics_service_map: true
data-prepper-config.yaml :
ssl: false
Open Telemetry configuration :
receivers:
jaeger:
protocols:
grpc:
exporters:
otlp/2:
endpoint: data-prepper:21890
tls:
insecure: true
insecure_skip_verify: true
logging:
service:
pipelines:
traces:
receivers: [jaeger]
exporters: [logging, otlp/2]
When starting docker-compose, I had errors :
data prepper : no valid pipeline is available for execution
opentelemetry : connection refused