Problems integrating AWS open telemetry collector for lambda/nodejs with OpenSearch data prepper

Data Prepper
1

I’m testing the AWS open telemetry collector for lambda/nodejs (aws-otel-nodejs-amd64-ver-1-0-1) using the following lambda layer: arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-0-1:2.
I have opensearch/data-prepper deployed on an ec2 machine and use the aws opensearch managed service.

However, I don’t manage to get the traces into opensearch. I’ve tried 2 different configurations of the open telemetry collector.

Both give a (different) error message. What am I missing?

data-prepper config file:

entry-pipeline:
  workers: 2
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
      health_check_service: true
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  workers: 2
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
    - otel_trace_raw_prepper:
  sink:
    - opensearch:
        hosts: [ "https://vpc-invxxxxxxxxxxxx6eiec3woa.us-east-1.es.amazonaws.com" ]
        aws_sigv4: false
        aws_region: "us-east-1"
        username: "os-admin"
        password: "xxxxxx"
        trace_analytics_raw: true
service-map-pipeline:
  workers: 2      
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: [ "https://vpc-invxxxxxxxxxxxx6eiec3woa.us-east-1.es.amazonaws.com" ]
        aws_sigv4: false
        aws_region: "us-east-1"
        username: "os-admin"
        password: "xxxxxxx"
        trace_analytics_service_map: true

First experiment (using otlp/http):

Otel config file:

receivers:
  otlp:
    protocols:
      grpc:
      http:
 
exporters:
  otlphttp/data-prepper:
    endpoint: "http://172.xx.xx.xx:4900"
    tls:
      insecure: true
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/data-prepper]
  telemetry:
    logs:
      level: "debug"

Then I get a 404 error on the exporter side (found in cloudwatch)

2022-03-22T12:43:52.605Z        debug        otlphttpexporter/otlp.go:127        Preparing to make HTTP request        {
    "kind": "exporter",
    "name": "otlphttp/data-prepper",
    "url": "http://172.xx.xx.xx:4900/v1/traces"
}
 
2022-03-22T12:43:52.687Z        info        exporterhelper/queued_retry.go:215        Exporting failed. Will retry the request after interval.        {
    "kind": "exporter",
    "name": "otlphttp/data-prepper",
    "error": "error exporting items, request to http://172.xx.xx.xx:4900/v1/traces responded with HTTP Status Code 404",
    "interval": "5.52330144s"
}

Second experiment (using grpc):

Otel config file:

receivers:
  otlp:
    protocols:
      grpc:
      http:
 
exporters:
  otlp/data-prepper:
    endpoint: "172.xx.xx.xx:4900"
    tls:
      insecure: true
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/data-prepper]
  telemetry:
    logs:
      level: "debug"

Then I get the following error on the exporter side (cloudwatch):

2022-03-22T12:54:56.595Z        info        exporterhelper/queued_retry.go:215        Exporting failed. Will retry the request after interval.        {
    "kind": "exporter",
    "name": "otlp/data-prepper",
    "error": "rpc error: code = Unavailable desc = connection closed before server preface received",
    "interval": "5.822800266s"
}
2

Hi @jkvbe,

Looking at your Data Prepper config file and otel collector config I recommend double checking the ports. By default, Data Prepper uses port 4900 for sharing Data Prepper metrics. The otel_trace_source uses port 21890 as the default.

A good example can be found in the Data Prepper smoke tests.

1 Like
3

Hi @smbayer,

Thanks, I was misguided by the data-prepper log message stating it was listening on port 4900.

Now the connection works. Up to the next problem :slight_smile: I only see the traces in OpenSearch, not the services and service map.

1 Like
4

Hi @jkvbe,

If you only have one service pubishing data to your otel collector, no documents are sent to the otel-v1-apm-service-map index. There is an open issue on GitHub to track when otel_trace_raw_prepper will be updated to work as expected with a single service.

If you have multiple services publishing data please try the following curl
to gather some data on what is being published to your opensearch cluster. Double check the results does not contain any sensitive information and reply with the results

curl -s -k -u '<opensearch-username>:<opensearch-password>' 'https://<opensearch-url>/_cat/indices'
curl -s -k -u '<opensearch-username>:<opensearch-password>' 'https://<opensearch-url>/otel-v1-apm-span-000001/_search'
curl -s -k -u '<opensearch-username>:<opensearch-password>' 'https://<opensearch-url>/otel-v1-apm-service-map/_search'
5

Thanks. I’m afraid I’m running against the bug (it’s just a simple demo application with 1 service).