Data Prepper - Pipeline Creations in Runtime

[mann]

Versions (Relevant - Data Prepper):

Hi,

I am trying to understand the functionalities provided by data prepper. As per my need, I wanted to automated the data prepper pipelines creation in runtime?

Like in running data prepper instance how can add multiple pipelines time to time?

Also that will be great if I can enable/disable the running pipeline etc.

[graytaylor0]

Hi @mann,

Thanks for creating this post. Currently, data prepper does not support enabling/disabling pipelines at runtime. At this time, the only way to modify the pipeline config is to stop and start data prepper.

This seems like a helpful feature to have, so it would be great if you could create a github issue for this. Let me know if you have any further questions. Thanks!

[mann]

Thanks @graytaylor0,

So does this possible in any other data prepper alternative like Logstash or FluentD?

[mann]

@graytaylor0 I have created the issue for the same here, it would be great to have this in your roadmap. Ref - Data Prepper - Pipeline Creations in Runtime · Issue #2301 · opensearch-project/data-prepper · GitHub.

Also, I was exploring more on this and by using “reload” tool can we add new pipelines in runtime?

As the “reload” feature allows you to update your Data Prepper configuration file and have those changes take effect without restarting the Data Prepper process. To use this feature, you’ll need to make the necessary changes to your configuration file, and then send a SIGHUP signal to the running Data Prepper process.

[asif]

@mann Thanks for creating the issue. We will try to add this to our roadmap. There is also an existing issue for this [Idea] Data Prepper can reload pipeline configuration without restarting · Issue #973 · opensearch-project/data-prepper · GitHub.

Thanks for the suggestions, it would help us later if you can add this reload option as a proposed solution in the issue.