# Data Prepper performance

**URL:** <https://forum.opensearch.org/t/data-prepper-performance/12866>\
**Category:** Data Prepper\
**Created:** [March 2, 2023, 9:08am UTC](https://forum.opensearch.org/t/data-prepper-performance/12866 "2023-03-02T09:08:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![harshit\_patel](https://avatars.discourse-cdn.com/v4/letter/h/e47c2d/32.png) [@harshit\_patel](https://forum.opensearch.org/u/harshit_patel)\
**Post date:** [March 2, 2023, 9:08am UTC](https://forum.opensearch.org/t/data-prepper-performance/12866/1 "2023-03-02T09:08:18Z")

</div>

Hi everyone,

Given my ingestion rate of 250GB per day from one source and that I want to support 100 such sources, what buffer\_size and batch\_size would be most appropriate for data-prepper? Alternatively, is data-prepper able to effectively process such a high volume of data?

I am thinking of keeping buffer\_size = 1500000 and batch\_size = 31250. Is there any downside for keeping such high buffer size?

@graytaylor0

---

<div class="post-metadata">

**Author:** ![graytaylor0](https://avatars.discourse-cdn.com/v4/letter/g/ecc23a/32.png) [@graytaylor0](https://forum.opensearch.org/u/graytaylor0)\
**Post date:** [March 6, 2023, 3:37pm UTC](https://forum.opensearch.org/t/data-prepper-performance/12866/2 "2023-03-06T15:37:49Z")

</div>

If you have the memory available, keeping buffer size and batch size to a large value will have better performance, and will not have a downside. Here is an example of performance testing that was run for log ingestion ([data-prepper/latest\_performance\_test\_results.md at main · opensearch-project/data-prepper · GitHub](https://github.com/opensearch-project/data-prepper/blob/main/docs/latest_performance_test_results.md)). Note that the buffer size was 200,000 in this case, but 1.5M is fine too depending on the amount of memory available.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [May 5, 2023, 3:38pm UTC](https://forum.opensearch.org/t/data-prepper-performance/12866/3 "2023-05-05T15:38:01Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.