Hi Team…
Can anyone help me out ?
For POC i have setup (Opensearch, Kafka,Opentelemetry,Data prepper). in one server.
My All services are running fine but when i am doing curl command to opensearch to get the index doc Its not showing the reason is Data prepper is not able to get the logs through opentelemetry.
Data prepper files:
otel-log-pipelines.yaml
otel-logs-pipeline:
source:
otel_logs_source:
port: 21892
ssl: false
buffer:
kafka:
bootstrap_servers: [“localhost:9092”]
encryption:
type: none
topics:
- name: opensearch-events
group_id: otel-data-prepper
create_topic: true
sink:
- opensearch:
hosts: [ “https://localhost:9200” ]
insecure: true
username: admin
password: #####
index: otel_logs
And Otel files:
cat otel-config.yaml
receivers:
filelog:
include: [ /home/jainrahul01/opensearch-data-prepper/file.log]
operators:
- type: json_parser
parse_from: body
exporters:
otlp/data-prepper:
endpoint: #########
tls:
insecure: true
insecure_skip_verify: true
logging:
service:
pipelines:
logs:
receivers: [filelog]
exporters: [logging,otlp/data-prepper]
Logs for data prepper:
://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/otel-logs-source#ssl
2024-05-24T15:04:57,530 [otel-logs-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.otellogs.OTelLogsSource - Started otel_logs_source…
2024-05-24T15:04:57,530 [otel-logs-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [otel-logs-pipeline] - Submitting request to initiate the pipeline processing
Logs for otel:
2024-05-24T15:06:18.576Z warn localhostgate/featuregate.go:63 The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default. {“feature gate ID”: “component.UseLocalHostAsDefaultHost”}
2024-05-24T15:06:18.776Z info fileconsumer/file.go:228 Started watching file {“kind”: “receiver”, “name”: “filelog”, “data_type”: “logs”, “component”: “fileconsumer”, “path”: “/home/jainrahul01/opensearch-data-prepper/file.log”}
curl -k -u admin -XGET https://localhost:9200/otel_logs/_search?pretty=true
Enter host password for user ‘admin’:
{
“took” : 1,
“timed_out” : false,
“_shards” : {
“total” : 1,
“successful” : 1,
“skipped” : 0,
“failed” : 0
},
“hits” : {
“total” : {
“value” : 0,
“relation” : “eq”
},
“max_score” : null,
“hits” :
}
}