# High write throughput for merge operation

**URL:** <https://forum.opensearch.org/t/high-write-throughput-for-merge-operation/17442>\
**Category:** OpenDistro\
**Created:** [January 16, 2024, 2:34pm UTC](https://forum.opensearch.org/t/high-write-throughput-for-merge-operation/17442 "2024-01-16T14:34:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ypeleg](https://avatars.discourse-cdn.com/v4/letter/y/9de0a6/32.png) [@ypeleg](https://forum.opensearch.org/u/ypeleg)\
**Post date:** [January 16, 2024, 2:34pm UTC](https://forum.opensearch.org/t/high-write-throughput-for-merge-operation/17442/1 "2024-01-16T14:34:49Z")

</div>

Hi,  
I am analyzing our opendistro system with performance analyzer dashboards.  
I am seeing that 99.99% of the write throughput is used for merge and about 30% of the cpu.  
I tried to disable index merging from the warm node but still it remained the same.  
Any idea why opendistro spends so much resources on merge?  
opendistro version: 1.12

 ![2024-01-11 17_06_44-Metrics Reference - Open Distro Documentation — Mozilla Firefox](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/8/8ed1fef31c5419e95817185caeec78988efddd63.png)

---

<div class="post-metadata">

**Author:** ![radu.gheorghe](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/radu.gheorghe/32/3516_2.png) [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)\
**Post date:** [January 16, 2024, 3:30pm UTC](https://forum.opensearch.org/t/high-write-throughput-for-merge-operation/17442/2 "2024-01-16T15:30:01Z")

</div>

With the default configuration, if you write at full throttle, it’s normal that most CPU time goes on merging.

You can tweak the merge policy to control the general balance between how expensive writes are on one hand and how expensive full text search is and how large (i.e. un-compacted) the index becomes. Here’s an oldie but goldie on the topic (on Solr, but both use Lucene, so it’s the same): [Solr: Optimize Is (Not) Bad for You – Video & Slides](https://sematext.com/blog/solr-optimize-is-not-bad-for-you-lucene-solr-revolution/)

---

<div class="post-metadata">

**Author:** ![ypeleg](https://avatars.discourse-cdn.com/v4/letter/y/9de0a6/32.png) [@ypeleg](https://forum.opensearch.org/u/ypeleg)\
**Post date:** [January 18, 2024, 12:38pm UTC](https://forum.opensearch.org/t/high-write-throughput-for-merge-operation/17442/3 "2024-01-18T12:38:13Z")

</div>

Hi Radu,  
Thank you for your answer.  
I also saw the video and learned some new important things.  
I didn’t mention that I tried to increase the index refresh interval from 30s to 60s but didn’t see noticeable change.

I appreciate your help.

---

<div class="post-metadata">

**Author:** ![radu.gheorghe](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/radu.gheorghe/32/3516_2.png) [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)\
**Post date:** [January 18, 2024, 1:55pm UTC](https://forum.opensearch.org/t/high-write-throughput-for-merge-operation/17442/4 "2024-01-18T13:55:12Z")

</div>

You’re welcome. With the refresh interval you get diminishing returns, so it’s kind of normal not to see them at some point. 30s is quite high for high-ingestion use-cases.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [March 18, 2024, 1:55pm UTC](https://forum.opensearch.org/t/high-write-throughput-for-merge-operation/17442/5 "2024-03-18T13:55:30Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.