We observed the performance degradation with OpenSearch 1.3 as well. @mrweber Your bug report helps a lot. The OpenSearch cluster we have is deployed with Helm Chart and running on top of k8s (managed by Rancher).
To apply the G1GC collector is a bit tricky for helm chart. By adding the config map may or may not work. We updated the helm chart to obtain the config/jvm.options . I’m not Java people hence it takes awhile for me to apply the change. One key point is the number in the fron of each line. That means which JDK version to apply the line. The default is 14 for G1CG. OpenSearch 1.3 uses JDK 11. Hence you need to change the version range for the G1CG otherwise, it’ll go SerialGC which has very poor performance.
## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly
## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
10-13:-XX:-UseConcMarkSweepGC
10-13:-XX:-UseCMSInitiatingOccupancyOnly
10-14:-XX:+UseG1GC
10-14:-XX:G1ReservePercent=25
10-14:-XX:InitiatingHeapOccupancyPercent=30
As you can see from the OpenSearch exporter, the indexing rate is much higher and the GC latency and times are way lower. This change along with the refresh_interval increasing brings the performance back as OpenSearch 1.2.4. Or maybe even better since we had refresh_interval as 5sec before. It’s 60s now.
@mrweber @theflakes theflakes I’m curious how do you measure the indexing performance in general?