Try the understand the error "Failed to write the request of size 4580301 due to: Buffer capacity too small for the number of records: 18005"

Data Prepper
1

the infrastructure is:
fluentbit → data-prepper:2 → opensearch

Describe the issue:
i tail multiple and large logs with tail in FluentBit to Data prepper and i got this error.

Failed to write the request of size 4580301 due to: Buffer capacity too small for the number of records: 18005

when i put to the directory the logs and i write to log the small couple of lines i don’t get this error.

Configuration:
[SERVICE]
flush 1
log_Level info
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 50M

[SERVICE]
flush 1
log_level info
Parsers_File /fluent-bit/etc/parsers_multiline.conf

[INPUT]
name tail
refresh_interval 5
path /logs//mazorx.log, /logs///mazorx.log, /logs////mazorx.log
multiline.parser multiline-regex-test
Tag mazor_log
Path_Key filename
Buffer_Max_Size 1M
Buffer_Chunk_Size 1M
Mem_Buf_Limit 100MB
storage.type filesystem
Static_Batch_Size 5MB

[OUTPUT]
Name http
Match mazor_log
Host master01.ent.core.medtronic.com
Port 443
URI /data-prepper/log/ingest
Format json
tls On
tls.verify Off
storage.total_limit_size 50M

[FILTER]
Name parser
Match mazor_log
Key_Name log
Reserve_Data true
Parser mazor_log

[FILTER]

Add the record_modifier filter

Name record_modifier
Match mazor_log
Record log_type mazor_log

Relevant Logs or Screenshots:

2

i found the configuration on the Buffer_Chunk_Size is solved the problem.
it need smaller enough from the Buffer_Max_Size to work properly

3

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.