Send trace data to opensearch

Plugins Observability
1

Hello,
I’m trying to send the trace data of a wordpress website to opensearch.

I started by sending trace data using wordpress plugin decalog to Jaeger (all in one deployment using docker) and got to visualize them in Jaeger UI.

Now that I’m trying to send them from there to opensearch, I cloned the repository mentioned in the documentation opensearch-project/data-prepper , and added the otel collector and data prepper services to my already existing opensearch service in docker-compose. But I’m having some difficulties to send it from here to opentelemetry to data prepper and opensearch.

And so I did another deployment that is much minimized :

I’ve seen examples for elasticsearch using Jaeger with Opentelemetry as a backend, so I changed them to fit my case as the follwing :

docker run --rm -it -v ${PWD}:/config \
-e SPAN_STORAGE_TYPE=elasticsearch opensearchproject/opensearch\
jaegertracing/jaeger-opentelemetry-collector \
–config-file=/config/config.yaml \
–es.server-urls=http://IP:9200 \
–es.num-shards=3

With config.yaml :

exporters:
otlp/data-prepper:
endpoint: http://IP:9200
insecure: true
processors:
attributes:
actions:
-key: user
action: delete
service:
pipelines:
traces:
processors: [attributes]

But this does not seem to work either, I get the following error :
./opensearch-docker-entrypoint.sh: line 140: /usr/share/opensearch/jaegertracing/jaeger-opentelemetry-collector: No such file or directory

Thank you very much in advance.

2

Explanation of the first method :

I cloned this GitHub repository : opensearch-project/data-prepper.
Then added the needed services to the already existing docker-compose.yml that has opensearch nodes :

data-prepper:
restart: unless-stopped
container_name: data-prepper
image: opensearchproject/data-prepper:latest
volumes:
- ./data-prepper/examples/trace_analytics_no_ssl.yml:/usr/share/data-prepper/pipelines.yaml
- ./data-prepper/examples/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml
- ./data-prepper/examples/demo/root-ca.pem:/usr/share/data-prepper/root-ca.pem
ports:
- “21890:21890”
networks:
- our-net
depends_on:
- “TisaOS-node1”
otel-collector:
container_name: otel-collector
image: otel/opentelemetry-collector:0.54.0
command: [ “–config=/etc/otel-collector-config.yml” ]
working_dir: “/project”
volumes:
- ${PWD}/:/project
- ./otel-collector-config.yml:/etc/otel-collector-config.yml
- /data-prepper/examples/demo/demo-data-prepper.crt:/etc/demo-data-prepper.crt
ports:
- “4318:4318”
depends_on:
- data-prepper
networks:
- our-net
jaeger-agent:
container_name: jaeger-agent
image: jaegertracing/jaeger-agent:latest
command: [ “–reporter.grpc.host-port=otel-collector:14250” ]
ports:
- “5775:5775/udp”
- “6831:6831/udp”
- “6832:6832/udp”
- “5778:5778/tcp”
networks:
- our-net
jaeger-hot-rod:
image: jaegertracing/example-hotrod:latest
command: [ “all” ]
environment:
- JAEGER_AGENT_HOST=jaeger-agent
- JAEGER_AGENT_PORT=6831
ports:
- “8080-8083:8080-8083”
depends_on:
- jaeger-agent
networks:
- our-net

The configuration of the pipeline : trace_analytics_no_ssl.yml

entry-pipeline:
delay: “100”
source:
otel_trace_source:
ssl: false
sink:
- pipeline:
name: “raw-pipeline”
- pipeline:
name: “service-map-pipeline”
raw-pipeline:
source:
pipeline:
name: “entry-pipeline”
prepper:
- otel_trace_raw_prepper:
sink:
- opensearch:
hosts: [ “http://IP:9200” ]
cert: “/usr/share/data-prepper/root-ca.pem”
username: “admin”
password: “admin”
trace_analytics_raw: true
service-map-pipeline:
delay: “100”
source:
pipeline:
name: “entry-pipeline”
prepper:
- service_map_stateful:
sink:
- opensearch:
hosts: [“http://IP:9200”]
cert: “/usr/share/data-prepper/root-ca.pem”
username: “admin”
password: “admin”
trace_analytics_service_map: true

data-prepper-config.yaml :

ssl: false

Open Telemetry configuration :

receivers:
jaeger:
protocols:
grpc:

exporters:
otlp/2:
endpoint: data-prepper:21890
tls:
insecure: true
insecure_skip_verify: true
logging:

service:
pipelines:
traces:
receivers: [jaeger]
exporters: [logging, otlp/2]

When starting docker-compose, I had errors :
data prepper : no valid pipeline is available for execution
opentelemetry : connection refused

3

Hi @issrae , since this issue is for data-prepper, i copied your content to a new thread [Data Prepper] Send trace data to opensearch

1 Like
4

Hi @issrae, we are tracking this issue here and I posted a reply in the post. Please take a look.

5

Hello, I checked the reply and gave feedback as well. Please take a look. Thank you.