Development environment setup for developing data-prepper

Data Prepper
1

I am new to Data-Prepper. I have a windows 10 laptop and I want to start developing a data-prepper plugin. I did not get much information on internet about how to setup a development environment for plugin development. Following are my queries.:

  1. I have cloned the data-prepper code from Git which has a folder data-prepper-plugins. Do I need to create the new data-prepper plugin project folder under this folder all the time?

  2. Once I have developed a data-prepper plugin (For e.g. Processor), do I need to build this project from the root buy executing build.gradle or the build.gradle from the new plugin-folder

  3. How do I execute the new plugin? Do i need to run an ubuntu docker container that will have data prepper and I copy the plugin code from windows to this instance and execute the plugin there?

These are but a few of the initial queries that I have, because these are not very clearly mentioned anywhere.

2

Hello Pradeep. Iā€™m happy to hear about your interest in creating a Data Prepper plugin.

First of all, what is your goal for this plugin? Would you like to make your own plugin for your own use-case, or are you interested in making a plugin to be part of the Data Prepper distribution?

If you wish to create one in the Data Prepper distribution, I strongly suggest that you start by searching the issues for any existing feature requests. You can even search just for processor requests. If it is not in there, then you should create a feature request first.

Data Prepper supports custom plugins which are not part of the build. This is ideal for use-cases that are narrowly tailored.

You can read some of the documentation for plugin development. It is applicable for either approach.

I created a sample Data Prepper plugin which you can look at to see how to create your own plugin which is not part of the Data Prepper distribution. Unfortunately, it is working against Data Prepper 1.3. There were some changes in 2.0 that break a few things here.

To answer some of your specific questions.

If your plugin is to be part of the Data Prepper distribution, then yes. It needs to be under the data-prepper-plugins directory. You also need to add it into the settings.gradle file along with the other plugins.

You will need to be able to run the full build. But, to speed work up, you can also run only a specific plugin. The following command runs a clean and build only on the opensearch plugin.

./gradlew -p data-prepper-plugins/opensearch clean build

This can be much faster for most builds. But, if you wish to create a PR, please be sure to run the full build before pushing.

This will vary depending on your approach.

Please let us know more about your use-case and we can try to help you further.

1 Like