Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Latest versions of both data-prepper and fluent-bit
Describe the issue:
I’m trying to use some real life data for my PoC implementation. I encountered an issue where using large files, logs causes errors.
The issue seem to originate from http server used by data-prepper and also fluent bit.
Configuration:
A simple 1 data-prepper and 1 fluent bit server running. I also have opensearch nodes running but lets ignore that for now.
Relevant Logs or Screenshots:
[2024/04/01 04:40:06] [error] [input:tail:tail.0] file=var/log/server.4.log requires a larger buffer size, lines are too long. Skipping file.
[2024/04/01 04:40:06] [error] [input:tail:tail.0] file=var/log/server.log requires a larger buffer size, lines are too long. Skipping file.
[2024/04/01 04:40:23] [error] [output:http:http.0] data-prepper-server:2021, HTTP status=408
Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 6710, timed out waiting for slots.
My log_pipeline.yaml sets buffers at the following:
buffer_size: 40096 # max number of events the buffer will accept
batch_size: 10024 # max number of events the buffer will drain for each read
In docker-compose, I’m setting request size to 500MB
environment:
- MAX_REQUEST_SIZE=500m
In fluent-bit.conf,
[SERVICE]
Mem_Buf_Limit 500MB # Adjust the buffer size as needed
If I don’t set these value this high, I get error 413 (from http), and way more failures.
[2024/04/01 04:29:26] [error] [output:http:http.0] data-prepper-server:2021, HTTP status=413
Buffer capacity too small for the number of records: 6851
Note: Due to failure, reties happen causing data duplication. I’ve left my server running overnight at some points and my barely 500 MB of data is showing an index size of over 100GB (might be due to Allow output plugins to configure a max chunk size · Issue #1938 · fluent/fluent-bit · GitHub).
I’ve tried to add a filte for trying to reduce the number of entries sent by fluent bit, but I haven’t succeeded there either (I tried using outputs created by chatgpt and based of documentation).