My docker opensearch data prepper can't find pipelines

Data Prepper
1

I’m trying to configure the OpenSearch Data Prepper log collector with Winlogbeat to write to OpenSearch.

My OpenSearch is deployed on a host. And the OpenSearch Data Prepper is in a Docker container.

I have the following configuration files:

/opt/data-prepper/config/data-prepper-config.yaml

ssl: true
keyStoreFilePath: "/usr/share/data-prepper/keystore.p12"
keyStorePassword: "$KEYSTORE_PASSWORD"
serverPort: 4900
authentication:
  http_basic:
    username: your_secure_username
    password: your_secure_password

/opt/data-prepper/pipelines/pipelines.yaml

secure-pipeline:
  source:
    http:
      port: 2021
      path: "/log/ingest"
      ssl:
        certificateFilePath: "/usr/share/data-prepper/tls/http.crt"
        privateKeyFilePath: "/usr/share/data-prepper/tls/http.key"
      authentication:
        http_basic:
          username: "beats_ingest"
          password: "Vbd^32DFC3vqwe@"
      health_check_service: true
      unauthenticated_health_check: true
      request_timeout: "10000"
      max_request_length: "10mb"
  sink:
    - opensearch:
        hosts: ["https://192.168.69.226:9200"]
        insecure: true
        username: "admin"
        password: "BalaBO34L"
        index: "windows-%{yyyy.MM.dd}"
        action: index

And I’m running the Docker container via the following compose file:

services:
  data-prepper:
    image: opensearchproject/data-prepper:latest
    container_name: data-prepper
    restart: none
    ports:
      - "4900:4900"
      - "2021:2021"
    environment:
      - KEYSTORE_PASSWORD
    volumes:
      - /opt/data-prepper/config/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml:ro
      - /opt/data-prepper/pipelines/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml:ro
      - /opt/data-prepper/security/keystore.p12:/usr/share/data-prepper/keystore.p12:ro
      - /opt/data-prepper/security/tls:/usr/share/data-prepper/tls:ro
    networks:
      - data-prepper-network

networks:
  data-prepper-network:
    driver: bridge

My container crashes with an error:

[main] WARN org.springframework.context.support.AbstractApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘dataPrepper’ defined in URL [jar:file:/usr/share/data-prepper/lib/data-prepper-core-2.16.0.jar!/org/opensearch/dataprepper/core/DataPrepper.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.opensearch.dataprepper.core.DataPrepper]: Constructor threw exception; nested exception is java.lang.RuntimeException: No valid pipeline is available for execution, exiting

I have files:

tree /opt/data-prepper/

/opt/data-prepper/
├── compose.yml
├── config
│   └── data-prepper-config.yaml
├── pipelines
│   └── pipelines.yaml
└── security
    ├── keystore.p12
    └── tls
        ├── http.crt
        └── http.key

What am I doing wrong?

2

@NordeN Do you see the pipelines. yaml file inside the data-prepper container?

3

@NordeN, I’ve reproduced your issue. I’ve found two misconfigurations.

The pipeline configuration in /opt/data-prepper/pipelines/pipelines.yaml is incorrect.

You must use the following instead.

  source:
    http:
      port: 2021
      path: "/log/ingest"
      ssl: true
      ssl_certificate_file: "/usr/share/data-prepper/tls/http.crt"
      ssl_key_file: "/usr/share/data-prepper/tls/http.key"
      ...

In /opt/data-prepper/config/data-prepper-config.yaml, you use an environment variable.
keyStorePassword does not support environment variables and expects a string by design. You must also add privateKeyPassword, which supports only strings.

Please use this instead.

ssl: true
keyStoreFilePath: "/usr/share/data-prepper/keystore.p12"
keyStorePassword: "your_password_in_clear_text"
privateKeyPassword: "your_password_in_clear_text"
...