# Try the understand the error "Failed to write the request of size 4580301 due to: Buffer capacity too small for the number of records: 18005"

**URL:** <https://forum.opensearch.org/t/try-the-understand-the-error-failed-to-write-the-request-of-size-4580301-due-to-buffer-capacity-too-small-for-the-number-of-records-18005/20664>\
**Category:** Data Prepper\
**Tags:** troubleshoot\
**Created:** [August 4, 2024, 7:04am UTC](https://forum.opensearch.org/t/try-the-understand-the-error-failed-to-write-the-request-of-size-4580301-due-to-buffer-capacity-too-small-for-the-number-of-records-18005/20664 "2024-08-04T07:04:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Israel](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/israel/32/7912_2.png) [@Israel](https://forum.opensearch.org/u/Israel)\
**Post date:** [August 4, 2024, 7:04am UTC](https://forum.opensearch.org/t/try-the-understand-the-error-failed-to-write-the-request-of-size-4580301-due-to-buffer-capacity-too-small-for-the-number-of-records-18005/20664/1 "2024-08-04T07:04:22Z")

</div>

the infrastructure is:  
fluentbit → data-prepper:2 → opensearch

**Describe the issue** :  
i tail multiple and large logs with tail in FluentBit to Data prepper and i got this error.

Failed to write the request of size 4580301 due to: Buffer capacity too small for the number of records: 18005

when i put to the directory the logs and i write to log the small couple of lines i don’t get this error.

**Configuration** :  
[SERVICE]  
flush 1  
log\_Level info  
storage.path /var/log/flb-storage/  
storage.sync normal  
storage.checksum off  
storage.backlog.mem\_limit 50M

[SERVICE]  
flush 1  
log\_level info  
Parsers\_File /fluent-bit/etc/parsers\_multiline.conf

[INPUT]  
name tail  
refresh\_interval 5  
path /logs/ **/mazorx.log, /logs/** / **/mazorx.log, /logs/** / **/** /mazorx.log  
multiline.parser multiline-regex-test  
Tag mazor\_log  
Path\_Key filename  
Buffer\_Max\_Size 1M  
Buffer\_Chunk\_Size 1M  
Mem\_Buf\_Limit 100MB  
storage.type filesystem  
Static\_Batch\_Size 5MB

[OUTPUT]  
Name http  
Match mazor\_log  
Host [master01.ent.core.medtronic.com](http://master01.ent.core.medtronic.com)  
Port 443  
URI /data-prepper/log/ingest  
Format json  
tls On  
tls.verify Off  
storage.total\_limit\_size 50M

[FILTER]  
Name parser  
Match mazor\_log  
Key\_Name log  
Reserve\_Data true  
Parser mazor\_log

[FILTER]

# Add the record\_modifier filter

Name record\_modifier  
Match mazor\_log  
Record log\_type mazor\_log

**Relevant Logs or Screenshots** :

---

<div class="post-metadata">

**Author:** ![Israel](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/israel/32/7912_2.png) [@Israel](https://forum.opensearch.org/u/Israel)\
**Post date:** [August 6, 2024, 6:41am UTC](https://forum.opensearch.org/t/try-the-understand-the-error-failed-to-write-the-request-of-size-4580301-due-to-buffer-capacity-too-small-for-the-number-of-records-18005/20664/2 "2024-08-06T06:41:55Z")

</div>

i found the configuration on the Buffer\_Chunk\_Size is solved the problem.  
it need smaller enough from the Buffer\_Max\_Size to work properly

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [October 5, 2024, 6:41am UTC](https://forum.opensearch.org/t/try-the-understand-the-error-failed-to-write-the-request-of-size-4580301-due-to-buffer-capacity-too-small-for-the-number-of-records-18005/20664/3 "2024-10-05T06:41:58Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.