Can't configure Data Prepper with Opensearch

mbreb · December 13, 2023, 10:56am

Hello,

I’m trying to put together a simple example of a pipeline using docker images

Pipeline is configured as follows:

log-pipeline:
  source:
    random:
  sink:
    - stdout:
    - opensearch:
        hosts: ["https://localhost:9200"]
        insecure: true
        username: admin
        password: admin
        index: test

The setup is:

docker-compose file defined as

version: '3'
services:
  opensearch:
    image: opensearchproject/opensearch:2.11.1
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  dashboards:
    image: opensearchproject/opensearch-dashboards:2.11.1
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch:9200"]'
    depends_on:
      - opensearch
    networks:
      - opensearch-net


networks:
  opensearch-net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 10.0.0.0/24

Sequence of steps:

Start opensearch cluster and dashboards with:

docker-compose up

Confirm server and dasboards are available:

spin up data prepper using:

	docker run -p 4900:4900 \
	-v ${PWD}/log_pipeline.yaml:/usr/share/data-prepper/pipelines/log_pipeline.yaml  \
	-v ${PWD}/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml \
	--network "opensearch_experiments_opensearch-net" \
	opensearchproject/data-prepper:latest

Logs returned:
docker run -p 4900:4900
-v /home/mara/opensearch_experiments/log_pipeline.yaml:/usr/share/data-prepper/pipelines/log_pipeline.yaml
-v /home/mara/opensearch_experiments/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml
–network “opensearch_experiments_opensearch-net”
opensearchproject/data-prepper:latest
Reading pipelines and data-prepper configuration files from Data Prepper home directory.
/usr/bin/java
Found openjdk version of 17.0
2023-12-13T10:45:54,676 [main] INFO org.opensearch.dataprepper.DataPrepperArgumentConfiguration - Command line args: /usr/share/data-prepper/pipelines,/usr/share/data-prepper/config/data-prepper-config.yaml
2023-12-13T10:45:54,753 [main] INFO org.opensearch.dataprepper.parser.PipelinesDataflowModelParser - Reading pipeline configuration from log_pipeline.yaml
2023-12-13T10:45:55,408 [main] INFO org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension - Applying Kafka Cluster Config Extension.
2023-12-13T10:45:55,668 [main] INFO org.opensearch.dataprepper.parser.PipelineTransformer - Building pipeline [log-pipeline] from provided configuration
2023-12-13T10:45:55,669 [main] INFO org.opensearch.dataprepper.parser.PipelineTransformer - Building [random] as source component for the pipeline [log-pipeline]
2023-12-13T10:45:55,746 [main] INFO org.opensearch.dataprepper.parser.PipelineTransformer - Building buffer for the pipeline [log-pipeline]
2023-12-13T10:45:55,753 [main] INFO org.opensearch.dataprepper.parser.PipelineTransformer - Building processors for the pipeline [log-pipeline]
2023-12-13T10:45:55,755 [main] INFO org.opensearch.dataprepper.parser.PipelineTransformer - Building sinks for the pipeline [log-pipeline]
2023-12-13T10:45:55,756 [main] INFO org.opensearch.dataprepper.parser.PipelineTransformer - Building [stdout] as sink component
2023-12-13T10:45:55,758 [main] INFO org.opensearch.dataprepper.parser.PipelineTransformer - Building [opensearch] as sink component
2023-12-13T10:45:55,784 [main] INFO org.opensearch.dataprepper.parser.PipelineTransformer - Constructing MultiBufferDecorator with [0] secondary buffers for pipeline [log-pipeline]
2023-12-13T10:45:55,845 [main] WARN org.opensearch.dataprepper.pipeline.server.config.DataPrepperServerConfiguration - Creating data prepper server without authentication. This is not secure.
2023-12-13T10:45:55,845 [main] WARN org.opensearch.dataprepper.pipeline.server.config.DataPrepperServerConfiguration - In order to set up Http Basic authentication for the data prepper server, go here: data-prepper/docs/core_apis.md at main · opensearch-project/data-prepper · GitHub
2023-12-13T10:45:55,919 [main] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Initiating pipeline execution
2023-12-13T10:45:55,922 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - sink is not ready for execution, retrying
2023-12-13T10:45:55,922 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2023-12-13T10:45:55,931 [main] WARN org.opensearch.dataprepper.pipeline.server.HttpServerProvider - Creating Data Prepper server without TLS. This is not secure.
2023-12-13T10:45:55,932 [main] WARN org.opensearch.dataprepper.pipeline.server.HttpServerProvider - In order to set up TLS for the Data Prepper server, go here: data-prepper/docs/configuration.md at main · opensearch-project/data-prepper · GitHub
2023-12-13T10:45:55,935 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2023-12-13T10:45:55,949 [main] INFO org.opensearch.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server running at :4900
2023-12-13T10:45:56,076 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2023-12-13T10:45:56,266 [log-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: Connection refused
2023-12-13T10:45:56,268 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] Waiting for Sink to be ready
2023-12-13T10:45:57,269 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - sink is not ready for execution, retrying
2023-12-13T10:45:57,270 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2023-12-13T10:45:57,270 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2023-12-13T10:45:57,270 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2023-12-13T10:45:57,276 [log-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: Connection refused
2023-12-13T10:45:58,278 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - sink is not ready for execution, retrying
2023-12-13T10:45:58,278 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2023-12-13T10:45:58,279 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2023-12-13T10:45:58,280 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2023-12-13T10:45:58,287 [log-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: Connection refused
2023-12-13T10:45:59,290 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - sink is not ready for execution, retrying
2023-12-13T10:45:59,291 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2023-12-13T10:45:59,293 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2023-12-13T10:45:59,294 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2023-12-13T10:45:59,299 [log-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: Connection refused


Can someone help ?

mbreb · December 13, 2023, 2:21pm

Managed to fix this by replacing the OpenSearch sink config with the following:

- opensearch:

hosts: [“https://:9200”]

    hosts: ["https://<name-of-the-container>:9200"]
    insecure: true
    username: "admin"
    password: "admin"
    index: "opensearch_dashboards_sample_data_logs"

Either of the hosts config up works

How to get that info:

Spin up opensearch container and dashboards using docker-compose up
Get the network name by running : docker network ls. Should return something like:

cb5621821fb2 opensearch_experiments_opensearch-net bridge local

Inspect the network to get the info by running docker network inspect opensearch_experiments_opensearch-net

This should return info about all attached containers to the network

[
    {
        "Name": "opensearch_experiments_opensearch-net",
        "Id": "cb5621821fb2aa7df5571b7f7e3faa0c1736b6ef249e6486508d6117534a3d61",
        "Created": "2023-12-13T14:44:01.4387966+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "1f89b5c0da4921d4970f8a93c297d01c84a2df686eaa14b9251e330e6dd4523b": {
                "Name": "opensearch_experiments_opensearch_1",
                "EndpointID": "",
                "MacAddress": "",
                "IPv4Address": "10.0.0.2/24",
                "IPv6Address": ""
            },
            "5cff15aabc393afd9fc606f5dd4aec61510a6b2109a4e7ce9a8da27a48716905": {
                "Name": "opensearch_experiments_dashboards_1",
                "EndpointID": "",
                "MacAddress": "",
                "IPv4Address": "10.0.0.3/24",
                "IPv6Address": ""
            },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "opensearch-net",
            "com.docker.compose.project": "opensearch_experiments",
            "com.docker.compose.version": "1.29.2"
        }
    }
]

Find there the ip/name of the container and replace it in your pipeline config

system · February 11, 2024, 2:22pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failed Initialization of Opensearch sink with data prepper OpenSearch	1	659	December 8, 2023
Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>[""], :thread=>"#<Thread:0x239d6b0 run>"} OpenSearch	1	340	September 22, 2022
Data-preeper plugin Data Prepper troubleshoot	8	970	January 24, 2022
Data Prepper not starting Data Prepper	24	4081	November 11, 2021
Writing pipelines for data prepper Data Prepper discuss , configure	1	56	September 7, 2024

Can't configure Data Prepper with Opensearch

hosts: [“https://:9200”]

Related topics