# Slow indexing performance **URL:** \ **Category:** OpenSearch\ **Tags:** troubleshoot, configure\ **Created:** [April 24, 2022, 1:00am UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366 "2022-04-24T01:00:44Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![theflakes](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@theflakes](https://forum.opensearch.org/u/theflakes)\ **Post date:** [April 24, 2022, 1:00am UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/1 "2022-04-24T01:00:44Z")
I moved from ELK 7.x about a month ago. So I’m still learning Opensearch. I am seeing slower indexing performance so far though. I am using Logstash with the Opensearch output plugin to push logs to Opensearch. I see the indexing fluctuate from 1600 to 3200 EPS consistently. ELK EPS was quite a bit higher and more consistent with the same underlying setup. We create three indexes per day with only one of them being quite large. Two are around 100MB per day and the other is anywhere from 40GB to 70GB per day. I’ve read what I can find and am not seeing how I up the EPS so far. I’m using the same index template as I did with ELK. Refresh interval is set to 5 minutes, one shard and no replicas. I am running in docker via docker swarm. Docker compose did not perform very well. Just one Opensearch v1.3.0 node and a dashboards docker container. It is running in an AWS memory optimized instance as was ELK. Any thoughts / tips? thanks ---
**Author:** ![mrweber](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/mrweber/32/1462_2.png) [@mrweber](https://forum.opensearch.org/u/mrweber)\ **Post date:** [April 24, 2022, 3:50am UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/2 "2022-04-24T03:50:23Z")
See [[BUG] Indexing Performance Degraded in OpenSearch 1.3.+ · Issue #2916 · opensearch-project/OpenSearch · GitHub](https://github.com/opensearch-project/OpenSearch/issues/2916). Enable G1GC and see if it helps. ---
**Author:** ![theflakes](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@theflakes](https://forum.opensearch.org/u/theflakes)\ **Post date:** [April 24, 2022, 1:20pm UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/3 "2022-04-24T13:20:26Z")
Thanks. How do I enable G1GC for the Docker version of Opensearch? I’ve tried the below but Opensearch will not start with these as the OPENSEARCH\_JAVA\_OPTS in the docker-compose.yml file. `-XX:+UseG1GC -XX:InitialHeapSize=32g -XX:MaxHeapSize=32g -XX:MaxGCPauseMillis=500 -XX:+DisableExplicitGC` Got this from reading: [JVM Tuning with G1 GC. A Garbage-First Garbage Collector… | by Mark Nienaber | Medium](https://marknienaber.medium.com/jvm-tuning-with-g1-gc-76f27535f054) ---
**Author:** ![mrweber](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/mrweber/32/1462_2.png) [@mrweber](https://forum.opensearch.org/u/mrweber)\ **Post date:** [April 24, 2022, 5:19pm UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/4 "2022-04-24T17:19:43Z")
OPENSEARCH\_JAVA\_OPTS is correct, you probably should try something like: ```auto -Xms30g -Xmx30g -XX:-UseConcMarkSweepGC -XX:-UseCMSInitiatingOccupancyOnly -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 ``` ---
**Author:** ![hugok](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/hugok/32/3116_2.png) [@hugok](https://forum.opensearch.org/u/hugok)\ **Post date:** [April 25, 2022, 6:56am UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/5 "2022-04-25T06:56:19Z")
We observed similar performance degradation while upgrading from 1.2.4 to 1.3. This OpenSearch cluster is deployed via Helm Chart on a k8s environment. We’ll try `+UseG1GC` . How to enable G1GC for helm chart managed cluster? Thanks for the thread. ```auto [opensearch@opensearch-cluster-client-57 ~]$ ./jdk/bin/java --version openjdk 11.0.14.1 2022-02-08 OpenJDK Runtime Environment Temurin-11.0.14.1+1 (build 11.0.14.1+1) OpenJDK 64-Bit Server VM Temurin-11.0.14.1+1 (build 11.0.14.1+1, mixed mode) ``` ```auto /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/tmp/opensearch-14041072601323771892 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Dclk.tck=100 -Djdk.attach.allowAttachSelf=true -Djava.security.policy=/usr/share/opensearch/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy -Dopensearch.cgroups.hierarchy.override=/ -Xmx8g -Xms8g -XX:MaxDirectMemorySize=4294967296 -Dopensearch.path.home=/usr/share/opensearch -Dopensearch.path.conf=/usr/share/opensearch/config -Dopensearch.distribution.type=tar -Dopensearch.bundled_jdk=true -cp /usr/share/opensearch/lib/* org.opensearch.bootstrap.OpenSearch -Ecluster.name=opensearch-cluster -Ediscovery.seed_hosts=opensearch-cluster-master-headless -Enode.roles=ingest,remote_cluster_client, -Enode.name=opensearch-cluster-client-0 -Enetwork.host=0.0.0.0 ``` ---
**Author:** ![hugok](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/hugok/32/3116_2.png) [@hugok](https://forum.opensearch.org/u/hugok)\ **Post date:** [April 25, 2022, 2:01pm UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/6 "2022-04-25T14:01:23Z")
We observed the performance degradation with OpenSearch 1.3 as well. @mrweber Your bug report helps a lot. The OpenSearch cluster we have is deployed with Helm Chart and running on top of k8s (managed by Rancher). To apply the G1GC collector is a bit tricky for helm chart. By adding the config map may or may not work. We updated the helm chart to obtain the config/jvm.options . I’m not Java people hence it takes awhile for me to apply the change. One key point is the number in the fron of each line. That means which **JDK version** to apply the line. The default is 14 for G1CG. OpenSearch 1.3 uses JDK 11. Hence you need to change the version range for the G1CG otherwise, it’ll go SerialGC which has very poor performance. ```auto ## GC configuration 8-13:-XX:+UseConcMarkSweepGC 8-13:-XX:CMSInitiatingOccupancyFraction=75 8-13:-XX:+UseCMSInitiatingOccupancyOnly ## G1GC Configuration # NOTE: G1 GC is only supported on JDK version 10 or later # to use G1GC, uncomment the next two lines and update the version on the # following three lines to your version of the JDK 10-13:-XX:-UseConcMarkSweepGC 10-13:-XX:-UseCMSInitiatingOccupancyOnly 10-14:-XX:+UseG1GC 10-14:-XX:G1ReservePercent=25 10-14:-XX:InitiatingHeapOccupancyPercent=30 ``` As you can see from the OpenSearch exporter, the indexing rate is much higher and the GC latency and times are way lower. This change along with the refresh\_interval increasing brings the performance back as OpenSearch 1.2.4. Or maybe even better since we had refresh\_interval as 5sec before. It’s 60s now. ![GC-tuning](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/7/78ee0d322a839567d798b75d87c2e9818230282a.jpeg) @mrweber @theflakes theflakes I’m curious how do you measure the indexing performance in general? ---
**Author:** ![dblock](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/dblock/32/1350_2.png) [@dblock](https://forum.opensearch.org/u/dblock)\ **Post date:** [April 25, 2022, 6:34pm UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/7 "2022-04-25T18:34:14Z")
First, thanks @mrweber for reporting this and finding the root cause. If anyone is interested in how this regression slipped through, or interested in contributing to improving performance benchmarking, please see [[BUG] Performance tests for 1.3 unable to detect indexing performance degradation · Issue #2985 · opensearch-project/OpenSearch · GitHub](https://github.com/opensearch-project/OpenSearch/issues/2985). ---
**Author:** ![louzadod](https://avatars.discourse-cdn.com/v4/letter/l/ea5d25/32.png) [@louzadod](https://forum.opensearch.org/u/louzadod)\ **Post date:** [May 6, 2022, 7:53pm UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/8 "2022-05-06T19:53:51Z")
Hello @hugok , I’m using the same environment as you (K8S + Rancher). I tried to put in my helm values file: ```auto opensearchJavaOpts: "-Xmx10G -Xms10G -XX:-UseConcMarkSweepGC -XX:-UseCMSInitiatingOccupancyOnly -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30" ``` But it did not work. I’m using OpenSearch Helm version 1.0.4. Which Helm version are you using? Thank you. ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [May 6, 2022, 8:53pm UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/9 "2022-05-06T20:53:43Z")
Hey @louzadod, you may need to to update the Helm chart to obtain the config/jvm.options, as per this comment [Slow indexing performance - #6 by hugok](https://forum.opensearch.org/t/slow-indexing-performance/9366/6). Hope it helps. ---
**Author:** ![louzadod](https://avatars.discourse-cdn.com/v4/letter/l/ea5d25/32.png) [@louzadod](https://forum.opensearch.org/u/louzadod)\ **Post date:** [May 9, 2022, 1:13pm UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/10 "2022-05-09T13:13:12Z")
Thanks a lot @reta and @hugok . After running the updated helm, the configuration was updated as bellow: ```auto cd config [opensearch@logs-corporativos-data-3 config]$ cat jvm.options ## GC configuration 8-13:-XX:+UseConcMarkSweepGC 8-13:-XX:CMSInitiatingOccupancyFraction=75 8-13:-XX:+UseCMSInitiatingOccupancyOnly ## G1GC Configuration # NOTE: G1 GC is only supported on JDK version 10 or later # to use G1GC, uncomment the next two lines and update the version on the # following three lines to your version of the JDK 10-13:-XX:-UseConcMarkSweepGC 10-13:-XX:-UseCMSInitiatingOccupancyOnly 10-14:-XX:+UseG1GC 10-14:-XX:G1ReservePercent=25 10-14:-XX:InitiatingHeapOccupancyPercent=30 ``` Is it the expected result? Thank you again. Regards. ---
**Author:** ![Mr\_Hedgehog](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@Mr\_Hedgehog](https://forum.opensearch.org/u/Mr_Hedgehog)\ **Post date:** [May 27, 2023, 7:15pm UTC](https://forum.opensearch.org/t/slow-indexing-performance/9366/11 "2023-05-27T19:15:31Z")
Hi there - I know some time has passed since the last comment on this thread. I would also be interested to know, in general how indexing performance might be measured and how I could identify a lag/buildup in documents and possibly alert on it. I’d also interested to know where that grafana dashboard has come from? I don’t recognise it from the opensearch prometheus exporter. Thanks, Will.