Data Prepper can read gzip encoded content?

Data Prepper
,
1

Hi Community,
I have to to send log in gzip format from FluentBit to Data Prepper.
Data Prepper can read gzip encoded content? And if it’s possibile, how I have to configure it?

If it isn’t possibile there is a workaround to send .gz files from to Fluentbit to Data Prepper and parsing them?

Thank you.

2

Hi @yago82,

Currently, the http source of Data Prepper (data-prepper/data-prepper-plugins/http-source at main · opensearch-project/data-prepper · GitHub), which can receive data from FluentBit, does not have support for reading gzip encoded content. I have created a Github issue to add support for this use case (Support gzip encoded content in http source · Issue #1399 · opensearch-project/data-prepper · GitHub). Unfortunately, the only workaround for this until it is supported by Data Prepper is to decode the gzip content before sending to Data Prepper, and then to send the decoded data straight to the http source. An s3 source is currently in development as well, and it will contain support for reading gzip encoded content, and this functionality should be reusable between the s3 source and http source. If you need gzip support sooner, we are very appreciative and open to any community contributions.

Let me know if you have any further questions. Thanks!

1 Like
3

First of all thanks for your answer. To your knowledge there may be another way to compress / decompress data from Fluentbit to Data Prepper?

Or do you think it would be possible to modify and adapt this plugin (Gzip_lines codec plugin | Logstash Reference [8.4] | Elastic) to be used by data-prepper?

Thank you.

4

It is not possible to use Logstash plugins directly in Data Prepper (you could use Logstash in front of Data Prepper, use the gzip codec, and then send from Logstash’s http output plugin to Data Prepper’s http source, but this workflow isn’t great. I do think that adapting a similar plugin for gzip codec in Data Prepper is possible, and something that we plan on supporting in a future release.

5

Thank you for the answer.

I read that Fluentd probably support gzip compression.
According to you it might be a solution using fluentbit-fluentd-opensearch (so fluentd instead of data-prepper)?

Thank you.

6

Yes I would think that using Fluentd in place of Data Prepper would work for your use case (depending on the processing that you need to do within Fluentd)