# Opensearch Replication & Recovery performance issue. tooks so long (100GiB -\> 2 ~ 3 Hours)

**URL:** <https://forum.opensearch.org/t/opensearch-replication-recovery-performance-issue-tooks-so-long-100gib-2-3-hours/27091>\
**Category:** OpenSearch\
**Tags:** troubleshoot\
**Created:** [September 21, 2025, 12:32pm UTC](https://forum.opensearch.org/t/opensearch-replication-recovery-performance-issue-tooks-so-long-100gib-2-3-hours/27091 "2025-09-21T12:32:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jameskim](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jameskim](https://forum.opensearch.org/u/jameskim)\
**Post date:** [September 21, 2025, 12:32pm UTC](https://forum.opensearch.org/t/opensearch-replication-recovery-performance-issue-tooks-so-long-100gib-2-3-hours/27091/1 "2025-09-21T12:32:29Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):

- Opensearch : 3.2.0
- Opensearch Operator : 2.8.0
- Environment : EKS 1.30
- Instance Type : m6g
- Volume : GP3 (IOPS, ThroughPut MAX)

**Describe the issue** :

Hi.

I`m migrating from elasticsearch to opensearch.

I am in the process of testing the speed of recovery and replication.

We are checking the replication speed with the following settings.

**Test Index is almost 100GiB.**

```auto
PUT /my_embedding_index/_settings
{
  "number_of_replicas": 1
}

```

What I’ve done: Change recovery speed, set file chunks, and more with Cluster Settings

Changing settings such as **max\_bytes\_per\_sec, node\_concurrent\_recoveries** , etc.

speeds up only in the beginning **and then slows down exponentially from a certain point in time**.

Nothing worked.

The peculiarities I found are as follows.

1. Network Bandwidth is not problem
2. Disk IO Exceeded and Throughput is not problem
3. **EBS Idle time is almost zero**

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/5/504890155840b935e09a35abcf168bb2a3bd927d.png)

* * *

**The difference between Elastic Search and Open Search**

[Opensearch : 3~4k]

Read IOPs are stuck at some point.

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/f/ffd64c6191fa3c40c0197a135fb3a52a8acfb140.png)

[ElasticSearch : 60k]

ElasticSearch is far more disk IOPs than Opensearch.

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/2/2dc3cb7fab607111f3e38ca6e1bd0b01989737d2.png)

```auto
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme2n1 0.00 0.00 1750.00 0.00 7000.00 0.00 8.00 0.98 0.56 0.56 0.00 0.57 99.00

avg-cpu: %user %nice %system %iowait %steal %idle
           2.59 0.00 1.05 5.97 0.00 90.39

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme2n1 0.00 8.00 1746.00 22.00 6984.00 120.00 8.04 1.01 0.57 0.56 1.50 0.54 96.00

```

I don`t know why opensearch uses so high Disk IOPS and So high Utilization.

**Configuration** :

ElasticSearch and Opensearch is almost same configuration and Computing spec.

**Relevant Logs or Screenshots** :

---

<div class="post-metadata">

**Author:** ![shs\_tech](https://avatars.discourse-cdn.com/v4/letter/s/ba8739/32.png) [@shs\_tech](https://forum.opensearch.org/u/shs_tech)\
**Post date:** [September 23, 2025, 5:16pm UTC](https://forum.opensearch.org/t/opensearch-replication-recovery-performance-issue-tooks-so-long-100gib-2-3-hours/27091/2 "2025-09-23T17:16:51Z")

</div>

I’m also experiencing similar issue in 3.2.0 . IOPS is touching whatever limit we provide. Sudden spikes are seen while snapshot is running which make the search & Kibana performance very poor.

---

<div class="post-metadata">

**Author:** ![jameskim](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jameskim](https://forum.opensearch.org/u/jameskim)\
**Post date:** [September 25, 2025, 4:31am UTC](https://forum.opensearch.org/t/opensearch-replication-recovery-performance-issue-tooks-so-long-100gib-2-3-hours/27091/3 "2025-09-25T04:31:04Z")

</div>

@shs_tech

> <https://github.com/opensearch-project/OpenSearch/issues/19156>
>
> \### Describe the bug
> 
> After upgrading our OpenSearch cluster from 2.18.0 to 3.1.…0, we observe a significant increase in disk read operations while write throughput/IOPS remain roughly the same. The workload is ingest-only (searches disabled). On 2.18.0, read IOPS stayed around 200–400 per node; on 3.1.0, under the same conditions, read IOPS jump to 400–3000+ per node.
> 
> This looks like a regression in indexing path / segment lifecycle that triggers substantially more background reads during ingestion (probably merge threads).
> 
> \*\*Requests for guidance\*\*
> Are there changes in 3.1.x that would increase background reads during indexing (e.g., segment lifecycle, merge behavior, replication strategy interactions, compaction/refresh defaults)?
> Any recommended settings in 3.1.x to restore 2.18-like read profiles for ingest-only use cases?
> 
> 
> \### Related component
> 
> Indexing:Performance
> 
> \### To Reproduce
> 
> \- Create a 3-node cluster on t4g.2xlarge with gp3 volumes (500 GB, 125 MB/s, 3000 IOPS) using image public.ecr.aws/opensearchproject/opensearch:3.1.0.
> \- Apply the cluster settings shown above.
> \- Create indices via the index template shown above.
> \- Ingest time-series logs at ~1.6k docs/sec per node; disable searches entirely (ingest-only).
> \- Retain only 2 days of indices, total size ~160 GB, reaching 1971 shards (650+ per node).
> \- Observe per-node disk metrics: read throughput 20–40 MB/s, read IOPS 400–3000+, while writes remain 15–60 MB/s, 200–600 IOPS.
> \- Repeat the same steps with OpenSearch 2.18.0 and note that read IOPS stay around 200–400 per node.
> 
> 
> \### Expected behavior
> 
> Read IOPS during ingest-only workload should be comparable to 2.18.0 (approximately 200–400 read IOPS per node), given identical hardware, shard layout, and ingestion rate.
> 
> \*\*Actual behavior\*\*
> On 3.1.0, read IOPS increase to 400–3000+ per node under the same workload and configuration, while write IOPS remain similar to 2.18.0.
> 
> \*\*Impact\*\*
> Higher disk reads lead to increased storage load and cost risk, potential saturation of gp3 baseline, and reduced indexing headroom.
> 
> \### Additional Details
> 
> \*\*Environment\*\*
> 
> test cluster setup:
> \*\*OpenSearch version:\*\* 3.1.0 (Docker image public.ecr.aws/opensearchproject/opensearch:3.1.0)
> \*\*Previous version (baseline):\*\* 2.18.0
> \*\*Cluster size:\*\* 3 data nodes (also cluster-manager/ingest)
> \*\*Instance type:\*\* t4g.2xlarge (8 vCPU, 32 GB RAM) on AWS
> \*\*Storage:\*\* gp3 500 GB per node, 125 MB/s baseline throughput, 3000 IOPS
> \*\*JVM opts:\*\* -Xms16g -Xmx16g -XX:MaxGCPauseMillis=400
> \*\*Workload:\*\* time-series log ingestion only (@timestamp field), searches disabled
> \*\*Data volume:\*\* indices retained 2 days, total size ~160 GB
> \*\*Shards:\*\* 1971 total (~650+ per node)
> 
> \*\*Observed metrics (per node)\*\*
> 
> \*\*Indexing rate:\*\* ~1.6k docs/sec
> \*\*Write throughput:\*\* 15–60 MB/s, 200–600 write IOPS
> \*\*Read throughput (problem):\*\* 20–40 MB/s, 400–3000+ peak read IOPS on 3.1.0
> (On 2.18.0: typically 200–400 read IOPS)
> 
> Cluster settings
> 
> \`\`\`
> OPENSEARCH\_JAVA\_OPTS = "-Xms16g -Xmx16g -XX:MaxGCPauseMillis=400"
> node.attr.temp = "hot"
> node.roles = "cluster\_manager,data,ingest,remote\_cluster\_client"
> plugins.security.ssl.http.enabled = "false"
> plugins.security.system\_indices.enabled = "false"
> plugins.security.ssl.http.clientauth\_mode = "NONE"
> plugins.security.protected\_indices.enabled = "false"
> indices.recovery.max\_bytes\_per\_sec = "60mb"
> cluster.routing.rebalance.enable = "all"
> cluster.routing.allocation.allow\_rebalance = "indices\_primaries\_active"
> cluster.routing.allocation.disk.threshold\_enabled = "false"
> cluster.routing.allocation.node\_initial\_primaries\_recoveries = "2"
> cluster.routing.allocation.node\_concurrent\_recoveries = "16"
> cluster.max\_shards\_per\_node = "3000"
> cluster.routing.allocation.balance.prefer\_primary = "true"
> cluster.indices.replication.strategy = "SEGMENT"
> \`\`\`
> 
> \*\*Index template:\*\*
> \`\`\`
> {
> "replication": { "type": "SEGMENT" },
> "allocation": { "max\_retries": "300" },
> "mapping": {
> "total\_fields": { "limit": "2000" },
> "depth": { "limit": "20" },
> "ignore\_malformed": "true"
> },
> "refresh\_interval": "120s",
> "translog": {
> "flush\_threshold\_size": "1024mb",
> "sync\_interval": "120s",
> "durability": "async"
> },
> "unassigned": {
> "node\_left": { "delayed\_timeout": "5m" }
> },
> "number\_of\_replicas": "1",
> "merge\_on\_flush": {
> "enabled": "false",
> "max\_full\_flush\_merge\_wait\_time": "30s",
> "policy": "default"
> },
> "codec": "default",
> "routing": {
> "allocation": {
> "require": { "temp": "hot" },
> "total\_shards\_per\_node": "3"
> }
> },
> "number\_of\_shards": "3",
> "use\_compound\_file": "false",
> "merge": {
> "scheduler": { "max\_thread\_count": "1" },
> "policy.max\_merge\_at\_once": "10",
> "policy": "log\_byte\_size"
> }
> }
> \`\`\`

you can see this !

i resoved issue by adding jvm flags

---

<div class="post-metadata">

**Author:** ![shs\_tech](https://avatars.discourse-cdn.com/v4/letter/s/ba8739/32.png) [@shs\_tech](https://forum.opensearch.org/u/shs_tech)\
**Post date:** [September 25, 2025, 1:14pm UTC](https://forum.opensearch.org/t/opensearch-replication-recovery-performance-issue-tooks-so-long-100gib-2-3-hours/27091/4 "2025-09-25T13:14:07Z")

</div>

Hello @jameskim ,

Thank you for sharing this. I’ve also added the JVM arg & it resolved my problem.

Cheers !