Hi Friends,
I need clarification regarding Data Prepper inputs.
Can Data Prepper consume/reading from external endpoints like *Beat, Kafka, SQS, Syslog etc?
Thank you.
Regards,
Giacomo
Hello Giacomo,
Data Prepper has a pluggable design for inputs (Data Prepper calls these Sources). Right now, Data Prepper has sources for OTel Trace, OTel Metrics, and HTTP (e.g. FluentBit).
The maintainers are currently working on a Source to read from S3 via an SQS queue. This is planned for the upcoming 1.5 release.
The roadmap also has a list of proposed sources. Feel free to comment or upvote any sources that you are particularly wanting. You can also create an issue if the one you want isn’t listed.
You can also contribute your own Source plugin to Data Prepper. Feel free to reach out. The other maintainers and I would be happy to help.