Data Prepper & OpenTelemetry

Data Prepper
1

All,

OpenSearch/ OpenSearch-Dashboard 2.8.0
Ubuntu 22.0.4
Data Prepper 2.3.1 from here
OpenTelemetry v0.80.0 install from here

I was able to install and run both Data Prepper and Otel on ubuntu. My issue seams like nothing is happening meaning no data is sent to Opensearch for Trace analytics, which Im trying to learn how this works.

Data Prepper

data-prepper-config.yaml 
root@ansible:/opt/opensearch-data-prepper/config# cat data-prepper-config.yaml
ssl: false
authentication:
     http_basic:
        username: "admin"
        password: "changeit"
root@ansible:/opt/opensearch-data-prepper/config#
pipelines.yaml 
root@ansible:/opt/opensearch-data-prepper/config# cat  pipelines.yaml
entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - otel_metrics_raw_processor:
        calculate_histogram_buckets: true
        calculate_exponential_histogram_buckets: true
        exponential_histogram_max_allowed_scale: 10
        flatten_attributes: false
  sink:
    - opensearch:
        hosts: [ "https://domain.com:9200" ]
        cert:  /opt/opensearch-data-prepper/config/root-ca.pem
        username: "admin"
        password: "changeit"
        index_type: trace-analytics-raw
        max_retries: 20
        bulk_size: 4
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: [ "https://domain-.com:9200" ]
        cert: /opt/opensearch-data-prepper/config/root-ca.pem
        username: "admin"
        password: "changeit"
        max_retries: 20
        bulk_size: 4
        index_type: trace-analytics-service-map
Startup_Log_file 
Data Prepper now supports reading pipeline and data-prepper configuration files
from Data Prepper home directory automatically.
You can continue to specify paths to configuration files as command line arguments,
but that support will be dropped in a future release.

JAVA_HOME is set to /opt/opensearch-data-prepper/openjdk/jdk-17.0.4.1+1
2023-07-03T22:08:26,719 [main] INFO  org.opensearch.dataprepper.DataPrepperArgumentConfiguration - Command line args: config/pipelines.yaml,config/data-prepper-config.yaml
2023-07-03T22:08:28,605 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building pipeline [entry-pipeline] from provided configuration
2023-07-03T22:08:28,606 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [otel_trace_source] as source component for the pipeline [entry-pipeline]
2023-07-03T22:08:28,745 [main] WARN  org.opensearch.dataprepper.plugins.source.oteltrace.OTelTraceSource - Creating otel-trace-source without authentication. This is not secure.
2023-07-03T22:08:28,745 [main] WARN  org.opensearch.dataprepper.plugins.source.oteltrace.OTelTraceSource - In order to set up Http Basic authentication for the otel-trace-source, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/otel-trace-source#authentication-configurations
2023-07-03T22:08:28,747 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building buffer for the pipeline [entry-pipeline]
2023-07-03T22:08:28,754 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building processors for the pipeline [entry-pipeline]
2023-07-03T22:08:28,755 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building sinks for the pipeline [entry-pipeline]
2023-07-03T22:08:28,756 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [pipeline] as sink component
2023-07-03T22:08:28,757 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [pipeline] as sink component
2023-07-03T22:08:28,758 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Constructing MultiBufferDecorator with [0] secondary buffers for pipeline [entry-pipeline]
2023-07-03T22:08:28,768 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building pipeline [service-map-pipeline] from provided configuration
2023-07-03T22:08:28,768 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [pipeline] as source component for the pipeline [service-map-pipeline]
2023-07-03T22:08:28,768 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building buffer for the pipeline [service-map-pipeline]
2023-07-03T22:08:28,770 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building processors for the pipeline [service-map-pipeline]
2023-07-03T22:08:28,770 [main] WARN  org.opensearch.dataprepper.plugin.DefaultPluginFactory - Plugin name 'service_map_stateful' is deprecated and will be removed in the next major release. Consider using the updated plugin name 'service_map'.
2023-07-03T22:08:30,204 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building sinks for the pipeline [service-map-pipeline]
2023-07-03T22:08:30,204 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [opensearch] as sink component
2023-07-03T22:08:30,255 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Constructing MultiBufferDecorator with [1] secondary buffers for pipeline [service-map-pipeline]
2023-07-03T22:08:30,256 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building pipeline [raw-pipeline] from provided configuration
2023-07-03T22:08:30,256 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [pipeline] as source component for the pipeline [raw-pipeline]
2023-07-03T22:08:30,256 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building buffer for the pipeline [raw-pipeline]
2023-07-03T22:08:30,258 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building processors for the pipeline [raw-pipeline]
2023-07-03T22:08:30,258 [main] WARN  org.opensearch.dataprepper.plugin.DefaultPluginFactory - Plugin name 'otel_metrics_raw_processor' is deprecated and will be removed in the next major release. Consider using the updated plugin name 'otel_metrics'.
2023-07-03T22:08:30,268 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building sinks for the pipeline [raw-pipeline]
2023-07-03T22:08:30,269 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [opensearch] as sink component
2023-07-03T22:08:30,272 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Constructing MultiBufferDecorator with [1] secondary buffers for pipeline [raw-pipeline]
2023-07-03T22:08:30,632 [main] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [service-map-pipeline] - sink is not ready for execution, retrying
2023-07-03T22:08:30,632 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2023-07-03T22:08:30,673 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2023-07-03T22:08:31,361 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the cert provided in the config.
2023-07-03T22:08:32,948 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager - Found version 0 for existing index template otel-v1-apm-service-map-index-template
2023-07-03T22:08:32,948 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager - Index template otel-v1-apm-service-map-index-template should not be updated, current version 0 >= existing version 0
2023-07-03T22:08:32,974 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initialized OpenSearch sink
2023-07-03T22:08:32,974 [main] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [raw-pipeline] - sink is not ready for execution, retrying
2023-07-03T22:08:32,975 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2023-07-03T22:08:32,975 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2023-07-03T22:08:32,976 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the cert provided in the config.
2023-07-03T22:08:33,134 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager - Found version 1 for existing index template otel-v1-apm-span-index-template
2023-07-03T22:08:33,135 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager - Index template otel-v1-apm-span-index-template should not be updated, current version 1 >= existing version 1
2023-07-03T22:08:33,164 [main] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initialized OpenSearch sink
2023-07-03T22:08:33,166 [main] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [entry-pipeline] - Initiating pipeline execution
2023-07-03T22:08:36,994 [main] WARN  org.opensearch.dataprepper.plugins.source.oteltrace.OTelTraceSource - Creating otel_trace_source without SSL/TLS. This is not secure.
2023-07-03T22:08:36,995 [main] WARN  org.opensearch.dataprepper.plugins.source.oteltrace.OTelTraceSource - In order to set up TLS for the otel_trace_source, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/otel-trace-source#ssl
2023-07-03T22:08:37,379 [main] INFO  org.opensearch.dataprepper.plugins.source.oteltrace.OTelTraceSource - Started otel_trace_source on port 21890...
2023-07-03T22:08:37,380 [main] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [entry-pipeline] - Submitting request to initiate the pipeline processing
2023-07-03T22:08:37,382 [main] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [service-map-pipeline] - Initiating pipeline execution
2023-07-03T22:08:37,383 [main] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [service-map-pipeline] - Submitting request to initiate the pipeline processing
2023-07-03T22:08:37,383 [main] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [raw-pipeline] - Initiating pipeline execution
2023-07-03T22:08:37,384 [main] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [raw-pipeline] - Submitting request to initiate the pipeline processing
2023-07-03T22:08:37,403 [main] WARN  org.opensearch.dataprepper.pipeline.server.HttpServerProvider - Creating Data Prepper server without TLS. This is not secure.
2023-07-03T22:08:37,403 [main] WARN  org.opensearch.dataprepper.pipeline.server.HttpServerProvider - In order to set up TLS for the Data Prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/configuration.md#server-configuration
2023-07-03T22:08:37,430 [main] INFO  org.opensearch.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server running at :4900

OpenTelemetry

config.yaml 
root@ansible:/etc/otelcol# cat  config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:55681

exporters:
  otlp/data-prepper:
    endpoint: http://192.168.1.100:21890
    tls:
      insecure: true
      insecure_skip_verify: true
  logging:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging,otlp/data-prepper]
otelcol.conf 
root@ansible:/etc/otelcol# cat  otelcol.conf
# Systemd environment file for the otelcol service

# Command-line options for the otelcol service.
# Run `/usr/bin/otelcol --help` to see all available options.
OTELCOL_OPTIONS="--config=/etc/otelcol/config.yaml"
Startup_Log_file 
Jul 03 21:54:10 domain.com systemd[1]: Started OpenTelemetry Collector.
-- Subject: A start job for unit otelcol.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit otelcol.service has finished successfully.
--
-- The job identifier is 170511.
Jul 03 21:54:11 ansible.enseva-labs.net otelcol[427003]: 2023-07-03T21:54:11.931-0500        info        service/telemetry.go:81        Setting up own telemetry...
Jul 03 21:54:12 ansible.enseva-labs.net otelcol[427003]: 2023-07-03T21:54:11.936-0500        info        service/telemetry.go:104        Serving Prometheus metrics {"address": ":8888", "level": "Basic"}
Jul 03 21:54:12 ansible.enseva-labs.net otelcol[427003]: 2023-07-03T21:54:11.936-0500        info        exporter@v0.80.0/exporter.go:275        Development component. May change in the future. {"kind": "exporter", "data_type": "traces", "name": "logging"}
Jul 03 21:54:12 ansible.enseva-labs.net otelcol[427003]: 2023-07-03T21:54:11.940-0500        info        service/service.go:131        Starting otelcol...        {"Version": "0.80.0", "NumCPU": 4}
Jul 03 21:54:12 ansible.enseva-labs.net otelcol[427003]: 2023-07-03T21:54:11.940-0500        info        extensions/extensions.go:30        Starting extensions...
Jul 03 21:54:12 ansible.enseva-labs.net otelcol[427003]: 2023-07-03T21:54:11.944-0500        warn        internal@v0.80.0/warning.go:40        Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks{"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
Jul 03 21:54:12 ansible.enseva-labs.net otelcol[427003]: 2023-07-03T21:54:11.944-0500        info        otlpreceiver@v0.80.0/otlp.go:83        Starting GRPC server{"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:55681"}
Jul 03 21:54:12 ansible.enseva-labs.net otelcol[427003]: 2023-07-03T21:54:11.944-0500        info        service/service.go:148        Everything is ready. Begin running and processing data.
~

Im not sure what ports to use nor how to configure these to services correctly to send data to Opensearch for observability

At the monment Im using the following

Filebeat/Metricbeat/Winlogbeat/Rsyslog/Nxlog/Packetbeat/Auditbeat with out issues.

knowing how those services work, I tried to apply it to Data Prepper with out success.

Dashboard is empty

image
image1496×860 79 KB

Any help or advice would be apperciated.
Thanks in advance

2

Think I’m missing a piece “Fluentbit”

3

Ok I see now, I found this example on GIT.

From that example I understand how this comes together.

Installed & configured fluent-bit --> data --> Data prepper Had to adjust Fluent-bit “Path” to get my apache logs. Created a index pattern. Still working on Metrics for Trace analytics. I have worked with prometheus and Grafana unsure how Data prepper can be configured.

4

Update: Observability

Fluent-bit is not bad using a simple configuration for Memory/CPU & Apache logs.

fluent-bit-conf 
[INPUT]
  name                  tail
  refresh_interval      5
  path                  /var/log/apache2/*.log
  read_from_head        true

[INPUT]
    Name    cpu
    Tag     cpu

[INPUT]
    Name    mem
    Tag     memory
[OUTPUT]
  Name http
  Match *
  Host localhost
  Port 2021
  URI /log/ingest
  Format json

As for Trace analytics nothing to observe. Still blank page and still working on it.

Observability using prometheus for Metric’s , completed with no issues. Documentation found here work very good.

Not sure how any of these service is supposed to work with Data Prepper, or if I should go around it.

5

I think i get it now :thinking:

Trace Analytics relies on adding instrumentation to the application and generating trace data.

OpenTelemetry Collector receives data from the application and formats it into OpenTelemetry data.

Data Prepper processes the OpenTelemetry data, transforms it for use in OpenSearch, and indexes it on an OpenSearch cluster.

The part I was missing was adding instrumentation to the application before OpenTelemetry.

6

All,
I figured I would share my pain and suffering :laughing: I must admit this was a good challenge for someone who has not worked with Trace Analysis/Data Prepper/Fluent bit/OpenTelemetry before. Took me few to understand that its not one size fits all here. What order the applications to be connected to what other application. I was treating it like Nxlog and LogStash. There are some familiarities but overall, what I really need to understand was what Protocol goes to what port along with what environment variables does what and how to configure these in the real world. Im not using Docker and did see the statement abount using dockers to make life easy but unfortunately we have no need to use docker for production setup.

For example:

  • otel_trace_source 21890
  • Data Prepper HTTP 2021
  • OTLP 4317
  • OTLPHTTP 4318

After researching I found out I need Fluent bit to send data to OpenTelemetry then send it to Data Prepper which sends that data to OpenSearch.
I wish there were a few less steps in this, mainly because this was where the major confusion was. Still don’t know what I’m doing but I was able to make it work. The following was my configuration made and any advice or suggestions is more than welcome.

Configuration_Files 
############# FLUENT-BIT ###############
[SERVICE]
  Flush           5
  Daemon          off
  Log_Level       debug
[INPUT]
  name                  tail
  refresh_interval      5
  path                  /var/log/apache2/*.log
  read_from_head        true
  tag apache2
[INPUT]
  Name                 node_exporter_metrics
  Tag                  node_metrics
  Scrape_interval      2
[INPUT]
  Name                 event_type
  Type                 traces
[OUTPUT]
  Name                 opentelemetry
  Match                *
  Host                 localhost
  Port                 4318
  Metrics_uri          /v1/metrics
  Logs_uri             /v1/logs
  Traces_uri           /v1/traces
  Log_response_payload True
  Tls                  off
  Tls.verify           Off
  # add user-defined labels
  add_label            app fluent-bit
  add_label            color blue

  
  

################### OTELCOL #############
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlphttp:
    endpoint: localhost:4318
    tls:
      insecure: true
  otlp/data-prepper:
    endpoint: localhost:21890
    tls:
      insecure: true
  otlp:
    endpoint: localhost:4317
    tls:
      insecure: true
extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/data-prepper]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/data-prepper]



##############  DATA-PREPPER ###########################
entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - otel_trace_raw:
  sink:
    - opensearch:
        hosts: [ "https://domain.com:9200" ]
        cert:  /opt/opensearch-data-prepper/config/root-ca.pem
        username: "admin"
        password: "changeit"
        index_type: trace-analytics-raw
        max_retries: 20
        bulk_size: 4
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: [ "https://domain.com:9200" ]
        cert: /opt/opensearch-data-prepper/config/root-ca.pem
        username: "admin"
        password: "changeit"
        max_retries: 20
        bulk_size: 4
        index_type: trace-analytics-service-map

Results:

image
image1883×910 55.5 KB

image
image1889×819 112 KB

image
image1881×920 89.3 KB

Resources Used:

Hope this might help someone in the future :+1:

7

@Gsmitt,

Thank you for sharing your findings.

Are you using FluentBit to send log data? I do not see a log pipeline in your configuration files.

After researching I found out I need Fluent bit to send data to OpenTelemetry then send it to Data Prepper which sends that data to OpenSearch.

You should not need to have FluentBit send to Data Prepper. See the Log analytics use case documentation. This shows an architecture you can use for log analytics using FluentBit. It also has some sample pipelines.

Also, check out the Trace analytics use case documentation. If you are only doing trace analytics you shouldn’t need FluentBit at all.

1 Like
8

Hey @dlv

First Thank you for the reply :+1: and I’m fairly new at this.

I’m using other means to test that out. I was able to execute logs using OpenTelemtry → Data -Prepper

Im still having difficulties with Trace/Log analysis. I did read Opensearch main doc’s but I found it more helpful on GitHub examples of starters.

As for Fluent-bit… its only example logs AKA dummy.log’s to find out how all this works or find a better way of setting it up. I was looking into MySql/MariaDb, Apache2, etc for trace analytics but I dont know how yet. Feel as thou I have a lot to learn.

image
image1872×713 70.1 KB

image
image1628×686 58.7 KB

The following is in order flow.

fluent-bit.conf 
[INPUT]
  name                  tail
  refresh_interval      5
  path                  /var/log/apache2/*.log
  read_from_head        true
  tag apache2
[INPUT]
  Name                 node_exporter_metrics
  Tag                  node_metrics
  Scrape_interval      4
[INPUT]
  Name                 event_type
  Type                 traces
  Tag                  traces
[OUTPUT]
  Name                 opentelemetry
  Match                *
  Host                 localhost
  Port                 4318
  Metrics_uri          /v1/metrics
  Logs_uri             /v1/logs
  Traces_uri           /v1/traces
  Log_response_payload True
  Tls                  off
  Tls.verify           Off
  # add user-defined labels
  add_label           add  my_bit
Open-Telemetry.yaml 
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlphttp:
    endpoint: localhost:4318
    tls:
      insecure: true
  otlp/data-prepper:
    endpoint: localhost:21890
    tls:
      insecure: true
  otlp:
    endpoint: localhost:4317
    tls:
      insecure: true
extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/data-prepper]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/data-prepper]
Data-Prepper.yaml 
entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - otel_trace_raw:
  sink:
    - opensearch:
        hosts: [ "https://ansible.enseva-labs.net:9200" ]
        cert:  /opt/opensearch-data-prepper/config/root-ca.pem
        username: "admin"
        password: "changeit"
        index_type: trace-analytics-raw
        max_retries: 20
        bulk_size: 4
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: [ "https://ansible.enseva-labs.net:9200" ]
        cert: /opt/opensearch-data-prepper/config/root-ca.pem
        username: "admin"
        password: "changeit"
        max_retries: 20
        bulk_size: 4
        index_type: trace-analytics-service-map

Once that was completed I created a Index-pattern. as shown below.

image
image1548×837 150 KB

I did completed setting up metrics, using Prometheus to grab remote Graylog metrics.

image
image1844×566 75.8 KB

I spent 12 days reading, googling and taking notes to get this far.

Again @dlv thanks for the reply

EDIT I forgot to add,my new lesson is that Data-prepper can only have 1 source BUT it can have multiple YAMl file’s in /pipelines directory :wink:

root@ansible:/opt/opensearch-data-prepper/pipelines# ls -al
total 24
drwxr-xr-x  2 root root 4096 Jul 13 23:47 .
drwxr-xr-x 11 root root 4096 Jul 13 17:02 ..
-rw-r--r--  1 root root  989 Jul 13 23:47 http-pipeline.yaml
-rw-r--r--  1 root root  426 Jul 13 18:55 metrics-pipeline.yaml
-rw-r--r--  1 root root 1145 Jul 13 20:46 pipelines.yaml
-rw-r--r--  1 root root   60 Jul 13 16:25 README
root@ansible:/opt/opensearch-data-prepper/pipelines# cat metrics-pipeline.yaml
metrics-pipeline:
  source:
    otel_metrics_source:
      ssl: false
      authentication:
        unauthenticated:
  processor:
    - otel_metrics_raw_processor:
  sink:
    - stdout:
    - opensearch:
        hosts: [ "https://domain.com:9200" ]
        cert: /opt/opensearch-data-prepper/config/root-ca.pem
        username: "admin"
        password: "changeit"
        index: "metrics-otel-v1-%{yyyy.MM.dd}"
root@ansible:/opt/opensearch-data-prepper/pipelines#
9

Easiest way to get some traces I know are ansible playbooks and docker builds. Also very easy to get these to jaeger. If you have some service mesh and/or using kubernetes, then there’s many ebpf collectors being developed that might be interesting. After those it might be manual instrumentation.

I tested “otel collector → data-prepper → opensearch” last winter and also spent around two weeks to get all working but compared to just using prometheus for metrics and logstash/opensearch for logs it was pretty hard.

10

Hey @jnum

I agree, unfortantly the higher ups dont want docker, I can make as many VM’s I want with unlimited resources.

I havent worked with that yet, but i did see that in the doc’s.

Same here, im on my second week. Thank you for the response @jnum

11

@Gsmitt ,

It appears that you have traces going into Fluent Bit, then OTel Collector, then Data Prepper. I do not believe you need Fluent Bit to handle traces.

A typical design supporting traces and logs will look like the following:

Log: application log files → Fluent Bit → Data Prepper
Trace: your application → OTel Collector → Data Prepper

Right now, it looks like you are trying to get traces going. What is the source of your span data? Can you route those directly to the OTel Collector?

1 Like
12

Hey @dlv

Ok i understand now.

Im trying, your reply just gave me a better idea what I need to do. Since it looks like I can make multiple Data-Prepper YAML files for Fluent Bit && OTel which I thought I could only use one Data-Prepper YAML file this makes more sense. BTW this is where i was confused. I was assuming I could only have one Data-Prepper file since the doc’s stated there could only be one source. Going to try you suggestion

Log: application log files → Fluent Bit → Data Prepper
Trace: your application → OTel Collector → Data Prepper

Thanks @dlv :+1: I just need a little push.

13

Hey @dlv

That was my fix :+1: , thank for jumping in this post much apperciated.

14

@Gsmitt,

I’m glad to hear that resolved your issue. If you hit other issues, please continue to ask questions on the forum.

1 Like
15

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.