thank you very much, @cmanning09 ! I’m getting a better understanding, I’ve changed my pipeline to otel_trace_raw and source reacord_type: “event”, now the pipeline is able to process the trace information successfully but still I’m getting null values for some of the requests, in the UI Trace dashboard it’s empty, what could be the problem?
My flow: raw-pipeline → trace-group-pipeline → opensearch sink to send data…
Updated Pipeline script:
otel-trace-pipeline:
# workers is the number of threads processing data in each pipeline.
# We recommend same value for all pipelines.
# default value is 1, set a value based on the machine you are running Data Prepper
#workers: 2
# delay in milliseconds is how often the worker threads should process data.
# Recommend not to change this config as we want the otel-trace-pipeline to process as quick as possible
# default value is 3_000 ms
delay: "100"
source:
otel_trace_source:
ssl: false # Change this to enable encryption in transit
record_type: event
authentication:
unauthenticated:
buffer:
bounded_blocking:
# buffer_size is the number of ExportTraceRequest from otel-collector the data prepper should hold in memeory.
# We recommend to keep the same buffer_size for all pipelines.
# Make sure you configure sufficient heap
# default value is 512
buffer_size: 10240
# This is the maximum number of request each worker thread will process within the delay.
# Default is 8.
# Make sure buffer_size >= workers * batch_size
batch_size: 160
sink:
- pipeline:
name: "raw-pipeline"
- pipeline:
name: "service-map-pipeline"
raw-pipeline:
source:
pipeline:
name: "otel-trace-pipeline"
buffer:
bounded_blocking:
# Configure the same value as in otel-trace-pipeline
# Make sure you configure sufficient heap
# default value is 512
buffer_size: 10240
# The raw processor does bulk request to your OpenSearch sink, so configure the batch_size higher.
# If you use the recommended otel-collector setup each ExportTraceRequest could contain max 50 spans. https://github.com/opensearch-project/data-prepper/tree/v0.7.x/deployment/aws
# With 64 as batch size each worker thread could process upto 3200 spans (64 * 50)
batch_size: 160
processor:
- otel_trace_raw:
sink:
- pipeline:
name: "trace-group-pipeline"
trace-group-pipeline:
#workers: 2
source:
pipeline:
name: "raw-pipeline"
processor:
- otel_trace_group:
hosts: ["http://10.81.211.25:9200"]
username: "admin"
password: "admin"
sink:
- opensearch:
hosts: ["http://10.81.211.25:9200"]
index_type: trace-analytics-raw
insecure: true
username: "admin"
password: "admin"
service-map-pipeline:
#workers: 2
delay: "100"
source:
pipeline:
name: "otel-trace-pipeline"
processor:
- service_map_stateful:
# The window duration is the maximum length of time the data prepper stores the most recent trace data to evaluvate service-map relationships.
# The default is 3 minutes, this means we can detect relationships between services from spans reported in last 3 minutes.
# Set higher value if your applications have higher latency.
#window_duration: 180
buffer:
bounded_blocking:
# buffer_size is the number of ExportTraceRequest from otel-collector the data prepper should hold in memeory.
# We recommend to keep the same buffer_size for all pipelines.
# Make sure you configure sufficient heap
# default value is 512
buffer_size: 10240
# This is the maximum number of request each worker thread will process within the delay.
# Default is 8.
# Make sure buffer_size >= workers * batch_size
batch_size: 160
sink:
- opensearch:
hosts: [ "http://10.81.211.25:9200" ]
#trace_analytics_service_map: true
index_type: trace-analytics-service-map
insecure: true
# Change to your credentials
username: "admin"
password: "admin"
# Add a certificate file if you are accessing an OpenSearch cluster with a self-signed certificate
#cert: /path/to/cert
# If you are connecting to an Amazon OpenSearch Service domain without
# Fine-Grained Access Control, enable these settings. Comment out the
# username and password above.
#aws_sigv4: true
#aws_region: us-east-1
Null TraceGroup sample data:
[
{
"_index": "otel-v1-apm-span-000001",
"_id": "4b7e5ca59b40be05",
"_score": 7.4577246,
"_source": {
"traceId": "06ae51d662ff69c132eae6416b3d50ff",
"droppedLinksCount": 0,
"kind": "SPAN_KIND_CLIENT",
"droppedEventsCount": 0,
"traceGroupFields": {
"endTime": null,
"durationInNanos": null,
"statusCode": null
},
"traceGroup": null,
"serviceName": "dev-agent-api",
"parentSpanId": "0fb5563b62c38f8c",
"spanId": "4b7e5ca59b40be05",
"traceState": "",
"name": "dcep",
"startTime": "2022-09-02T09:01:03.756723300Z",
"links": [],
"endTime": "2022-09-02T09:01:03.758052500Z",
"droppedAttributesCount": 0,
"durationInNanos": 1329200,
"events": [],
"span.attributes.db@statement_type": "Text",
"instrumentationLibrary.version": "1.0.0.0",
"resource.attributes.service@instance@id": "d15bf4b8-3d89-41d3-a4de-288a0133bddc",
"span.attributes.db@name": "dcep",
"resource.attributes.service@name": "dev-agent-api",
"status.code": 0,
"span.attributes.db@system": "postgresql",
"instrumentationLibrary.name": "OpenTelemetry.EntityFrameworkCore"
}
},
{
"_index": "otel-v1-apm-span-000001",
"_id": "0fb5563b62c38f8c",
"_score": 7.4577246,
"_source": {
"traceId": "06ae51d662ff69c132eae6416b3d50ff",
"droppedLinksCount": 0,
"kind": "SPAN_KIND_SERVER",
"droppedEventsCount": 0,
**"traceGroupFields": {**
** "endTime": null,**
** "durationInNanos": null,**
** "statusCode": null**
** },**
** "traceGroup": null,**
"serviceName": "dev-agent-api",
"parentSpanId": "df2b3492e7bba1b0",
"spanId": "0fb5563b62c38f8c",
"traceState": "",
"name": "api/Config/skills",
"startTime": "2022-09-02T09:01:03.752806800Z",
"links": [],
"endTime": "2022-09-02T09:01:03.759687900Z",
"droppedAttributesCount": 0,
"durationInNanos": 6881100,
"events": [],
"span.attributes.http@url": "http://croma.dcepdev.punelab.local/agent/api/Config/skills",
"instrumentationLibrary.version": "1.0.0.0",
"resource.attributes.service@instance@id": "d15bf4b8-3d89-41d3-a4de-288a0133bddc",
"resource.attributes.service@name": "dev-agent-api",
"status.code": 0,
"instrumentationLibrary.name": "OpenTelemetry.Instrumentation.AspNetCore",
"span.attributes.http@method": "GET",
"span.attributes.http@user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36",
"span.attributes.http@route": "api/Config/skills",
"span.attributes.http@host": "croma.dcepdev.punelab.local",
"span.attributes.http@target": "/agent/api/Config/skills",
"span.attributes.http@scheme": "http",
"span.attributes.http@flavor": "1.1",
"span.attributes.http@status_code": 200
}
}
]
Here is some documentation for that and what you can expect.