Running multiple pipelines using different YML files

Data Prepper
1

Hi,
I am trying to create multiple pipelines in Data Prepper. I have the below YML file which creates 2 pipelines. But, is there a way to create two separate YML files, each for one pipeline, and run them both?

pipeline-1:
  workers: 2
  delay: 1000
  source:
    random:
  processor:
    - string_converter:
        upper_case: true
  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        insecure: true
        username: admin
        password: admin
        index: data-prepper-1

pipeline-2:
  workers: 2
  delay: 1000
  source:
    random:
  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        insecure: true
        username: admin
        password: admin
        index: data-prepper
1 Like
2

Hi @harshit_patel

The example configuration you provided is the correct way to run multiple pipelines since Data Prepper is configured to have 1 Pipeline configuration YAML. You can checkout some log ingestion examples here: data-prepper/log_analytics.md at main · opensearch-project/data-prepper · GitHub

3

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.