I’m sending trace data of wordpress website using wordpress plugin decalog to jaeger agent, and from there to opentelemetry and then data prepper, to the sink opensearch. But I’m getting no endpoint (“grpc_log”: true) error for jaeger and otel collector and failed to parse config file for the data prepper. I couldn’t solve those errors until now, so some help would be very much appreciated.
Docker-compose :
services:
data-prepper:
restart: unless-stopped
container_name: data-prepper
image: opensearchproject/data-prepper:latest
volumes:
- ./data-prepper/examples/trace_analytics_no_ssl.yml:/usr/share/data-prepper/pipelines.yaml
- ./data-prepper/examples/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml
- ./data-prepper/examples/demo/root-ca.pem:/usr/share/data-prepper/root-ca.pem
ports:
- “21890:21890”
networks:
- ournet
depends_on:
- “TisaOS-node1”
otel-collector:
container_name: otel-collector
image: otel/opentelemetry-collector:0.54.0
command: [ “–config=/etc/otel-collector-config.yml” ]
working_dir: “/project”
volumes:
- ${PWD}/:/project
- ./otel-collector-config.yml:/etc/otel-collector-config.yml
- ./data-prepper/examples/demo/demo-data-prepper.crt:/etc/demo-data-prepper.crt
ports:
- “4317:4317”
depends_on:
- data-prepper
networks:
- ournet
jaeger-agent:
container_name: jaeger-agent
image: jaegertracing/jaeger-agent:latest
command: [ “–reporter.grpc.host-port=otel-collector:14250” ]
ports:
- “5775:5775/udp”
- “6831:6831/udp”
- “6832:6832/udp”
- “5778:5778/tcp”
networks:
- ournet
The configuration of the pipeline : trace_analytics_no_ssl.yml
entry-pipeline:
delay: “100”
source:
otel_trace_source:
ssl: false
sink:
- pipeline:
name: “raw-pipeline”
- pipeline:
name: “service-map-pipeline”
raw-pipeline:
source:
pipeline:
name: “entry-pipeline”
prepper:
- otel_trace_raw_prepper:
sink:
- opensearch:
hosts: [ “http://IP:9200” ]
cert: “/usr/share/data-prepper/root-ca.pem”
username: “admin”
password: “admin”
trace_analytics_raw: true
service-map-pipeline:
delay: “100”
source:
pipeline:
name: “entry-pipeline”
prepper:
- service_map_stateful:
sink:
- opensearch:
hosts: [“http://IP:9200”]
cert: “/usr/share/data-prepper/root-ca.pem”
username: “admin”
password: “admin”
trace_analytics_service_map: true
data-prepper-config.yaml
ssl: false
Open Telemetry configuration
receivers:
jaeger:
protocols:
grpc:
exporters:
otlp/data-prepper:
endpoint: http://data-prepper:21890
tls:
insecure: true
insecure_skip_verify: true
logging:
service:
pipelines:
traces:
receivers: [jaeger]
exporters: [otlp/data-prepper]
I’m getting the following errors :
For jaeger agent :
jaeger-agent | {“level”:“info”,“ts”:1659021878.5910108,“caller”:“healthcheck/handler.go:129”,“msg”:“Health Check state change”,“status”:“ready”}
jaeger-agent | {“level”:“info”,“ts”:1659021878.5910616,“caller”:“grpc/builder.go:109”,“msg”:“Checking connection to collector”}
jaeger-agent | {“level”:“info”,“ts”:1659021878.591072,“caller”:“grpc/builder.go:120”,“msg”:“Agent collector connection state change”,“dialTarget”:“otel-collector:14250”,“status”:“CONNECTING”}
jaeger-agent | {“level”:“info”,“ts”:1659021878.591132,“caller”:“app/agent.go:69”,“msg”:“Starting jaeger-agent HTTP server”,“http-port”:5778}
jaeger-agent | {“level”:“info”,“ts”:1659021878.5911915,“caller”:“grpclog/component.go:71”,“msg”:“[balancer]base.baseBalancer: handle SubConn state change: 0xc000351c10, CONNECTING”,“system”:“grpc”,“grpc_log”:true}
jaeger-agent | {“level”:“warn”,“ts”:1659021878.6262655,“caller”:“channelz/funcs.go:342”,“msg”:“[core][Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {\n "Addr": "otel-collector:14250",\n "ServerName": "otel-collector:14250",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup otel-collector on 127.0.0.11:53: server misbehaving"”,“system”:“grpc”,“grpc_log”:true}
For opentelemetry :
otel-collector grpc: addrConn.createTransport failed to connect to {
otel-collector | “Addr”: “data-prepper:21890”,
otel-collector | “ServerName”: “data-prepper:21890”,
otel-collector | “Attributes”: null,
otel-collector | “BalancerAttributes”: null,
otel-collector | “Type”: 0,
otel-collector | “Metadata”: null
otel-collector Err: connection error: desc = “transport: Error while dialing dial tcp IP:21890: connect: connection refused” {“grpc_log”: true}
For data prepper :
data-prepper | 2022-07-28T10:25:24,101 [main] WARN org.springframework.context.support.AbstractApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘dataPrepperServer’ defined in URL [jar:file:/usr/share/data-prepper/data-prepper.jar!/com/amazon/dataprepper/pipeline/server/DataPrepperServer.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘httpServer’ defined in class path resource [com/amazon/dataprepper/pipeline/server/config/DataPrepperServerConfiguration.class]: Unsatisfied dependency expressed through method ‘httpServer’ parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘listPipelinesHandler’ defined in class path resource [com/amazon/dataprepper/pipeline/server/config/DataPrepperServerConfiguration.class]: Unsatisfied dependency expressed through method ‘listPipelinesHandler’ parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘dataPrepper’ defined in URL [jar:file:/usr/share/data-prepper/data-prepper.jar!/com/amazon/dataprepper/DataPrepper.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.amazon.dataprepper.DataPrepper]: Constructor threw exception; nested exception is com.amazon.dataprepper.parser.ParseException: Failed to parse the configuration file /usr/share/data-prepper/pipelines.yaml