Today we’re happy to announce the release of Data Prepper 1.1.0, and we want to say huge “Thank You!” for your continued support. This release provides security enhancements for the OpenTelemetry (OTEL) Collector Source and Peer Forwarder processors, AWS Cloud Map integration for peer discovery when running as a cluster in AWS infrastructure, less hard-coded configuration, and better performance when running trace analytics workloads.
Some of the key features we’ve introduced are an integration with AWS Certificate Manager (ACM) and S3 for both the OTEL Trace Source and Peer Forwarder plugins. These allow private keys and certificates to be fetched from ACM or S3 when establishing TLS connections. Another feature is AWS Cloud Map integration with the Peer Forwarder plugin. Now in addition to creating Data Prepper clusters through static files or DNS Discovery you can integrate with AWS Cloud Map to perform node discovery, reducing the amount of configuration and problems that can occur in static files or customer DNS discovery.
We’ve also introduced some crucial configurability enhancements in Data Prepper 1.1.0. First, the peer forwarder target port is no longer hard coded. Secondly, the AWS IAM roll used by the OpenSearch sink is now configurable. The OTEL Collector source configuration can now accept application/json in addition to gRPC. In addition, you can define multiple Micrometer registries so that in addition to Prometheus, you can now register Amazon CloudWatch as a destination for performance and availability metrics. These all ensure Data Prepper is easier to configure and use in your infrastructure.
You can download Data Prepper 1.1 in source code or as a Docker container, and reference the readme here. If you’re interested in learning more, have a specific question, or just want to provide feedback and thoughts, please visit OpenSearch.org, open an issue on GitHub, or post in the forums.
We are so excited about the progress so far! The entire community of contributors should be incredibly proud of the accomplishment of reaching 1.1.0 together.
Authors: @laneholloway @RajivTaori @qchea