Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch Version: 3.7.0
Describe the issue:
We upgraded to OpenSearch version 3.7.0 recently and observed the initialization is very slow as compared to its predecessor versions.
Configuration:
jvm.options:
## JVM configuration
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################
## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
# --- startup timing diagnostics (remove after investigation) ---
# inline '#' comments are NOT allowed on option lines in jvm.options,
# so each flag must be on its own line with nothing after it.
-Xlog:startuptime
# class-load timing goes to its own file so it doesn't flood the console
-Xlog:class+load=info:file=/tmp/opensearch/classload.log:utctime,level,tags:filecount=2,filesize=20m
# --- end diagnostics ---
11-:-XX:+UseG1GC
11-:-XX:G1ReservePercent=25
11-:-XX:InitiatingHeapOccupancyPercent=30
## JVM temporary directory
-Djava.io.tmpdir=${OPENSEARCH_TMPDIR}
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/tmp/opensearch/heapdump.hprof
# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/tmp/opensearch/hs_err.log
# JDK 9+ GC logging
11-:-Xlog:gc*=warning:file=/tmp/opensearch/gc.log:utctime,pid,tags,level:filecount=2,filesize=2m
# JDK 20+ Incubating Vector Module for SIMD optimizations;
# disabling may reduce performance on vector optimized lucene
20-:--add-modules=jdk.incubator.vector
# See please https://bugs.openjdk.org/browse/JDK-8341127 (openjdk/jdk#21283)
23:-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.setAsTypeCache
23:-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.asTypeUncached
21-:-javaagent:agent/opensearch-agent.jar
Relevant Logs or Screenshots:
3.7.0 Logs. In the below logs, we can clearly see that from “Created opensearch keystore” to “version[3.7.0]” takes easily more than 30s whereas by the time the ports are bound it even crosses 60+ seconds, sometimes taking 70s as well. This slowness was neither observed in 3.5.0 nor in 2.19.4
2026-07-08T16:04:12.900118195Z ls: cannot access '/opt/opensearch/is-certManagerSecret': No such file or directory
2026-07-08T16:04:12.902604321Z cp: cannot stat '/etc/opensearch/opensearch-security/*': No such file or directory
2026-07-08T16:04:12.905443257Z sed: can't read /etc/opensearch/securityconfig/config.yml: No such file or directory
2026-07-08T16:04:12.906787059Z chmod: cannot access '/etc/opensearch/securityconfig/*': No such file or directory
2026-07-08T16:04:17.891192222Z Created opensearch keystore in /etc/opensearch/config/opensearch.keystore
2026-07-08T16:04:47.820267563Z WARNING: Using incubator modules: jdk.incubator.vector
2026-07-08T16:04:55.824540970Z {"type":"log","host":"ild-bssc-indexsearch-data-1.ram-system","container":"is-data","level":"WARN","systemid":"BSSC-1234","system":"BSSC","time": "2026-07-08T16:04:55.800854Z","logger":"stderr","timezone":"UTC","marker":"","log":{"message":"Jul 08, 2026 4:04:55 PM org.opensearch.javaagent.bootstrap.AgentPolicy setPolicy"}}
2026-07-08T16:04:55.824852413Z {"type":"log","host":"ild-bssc-indexsearch-data-1.ram-system","container":"is-data","level":"WARN","systemid":"BSSC-1234","system":"BSSC","time": "2026-07-08T16:04:55.824594Z","logger":"stderr","timezone":"UTC","marker":"","log":{"message":"INFO: Policy attached successfully: org.opensearch.bootstrap.OpenSearchPolicy@7d3fb0ef"}}
2026-07-08T16:04:55.911503423Z {"type":"log","host":"ild-bssc-indexsearch-data-1.ram-system","container":"is-data","level":"INFO","systemid":"BSSC-1234","system":"BSSC","time": "2026-07-08T16:04:55.911151Z","logger":"o.o.n.Node","timezone":"UTC","marker":"","log":{"message":"version[3.7.0], pid[402], build[rpm/72121f014083f9ca010fd5a7da83b2ec4886027f/2026-06-03T23:30:23.512631638Z], OS[Linux/6.6.3-0-virt/amd64], JVM[Red Hat, Inc./OpenJDK 64-Bit Server VM/21.0.11/21.0.11+10-LTS]"}}
2026-07-08T16:04:55.911576683Z {"type":"log","host":"ild-bssc-indexsearch-data-1.ram-system","container":"is-data","level":"INFO","systemid":"BSSC-1234","system":"BSSC","time": "2026-07-08T16:04:55.911452Z","logger":"o.o.n.Node","timezone":"UTC","marker":"","log":{"message":"JVM home [/opt/jdk-minimal], using bundled JDK/JRE [false]"}}
2026-07-08T16:04:55.911773716Z {"type":"log","host":"ild-bssc-indexsearch-data-1.ram-system","container":"is-data","level":"INFO","systemid":"BSSC-1234","system":"BSSC","time": "2026-07-08T16:04:55.911571Z","logger":"o.o.n.Node","timezone":"UTC","marker":"","log":{"message":"JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,CLDR, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-10621479840062898917, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/tmp/opensearch/heapdump.hprof, -XX:ErrorFile=/tmp/opensearch/hs_err.log, -Xlog:gc*=warning:file=/tmp/opensearch/gc.log:utctime,pid,tags,level:filecount=2,filesize=2m, --add-modules=jdk.incubator.vector, -javaagent:agent/opensearch-agent.jar, -Dopensearch.cgroups.hierarchy.override=/, -Xms2g, -Xmx2g, -XX:MaxDirectMemorySize=1073741824, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/etc/opensearch/config, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]"}}
2026-07-08T16:04:57.164616859Z {"type":"log","host":"ild-bssc-indexsearch-data-1.ram-system","container":"is-data","level":"WARN","systemid":"BSSC-1234","system":"BSSC","time": "2026-07-08T16:04:57.158563Z","logger":"stderr","timezone":"UTC","marker":"","log":{"message":"Jul 08, 2026 4:04:57 PM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider <init>"}}
2026-07-08T16:04:57.164700438Z {"type":"log","host":"ild-bssc-indexsearch-data-1.ram-system","container":"is-data","level":"WARN","systemid":"BSSC-1234","system":"BSSC","time": "2026-07-08T16:04:57.158823Z","logger":"stderr","timezone":"UTC","marker":"","log":{"message":"INFO: Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled"}}
2026-07-08T16:04:59.736795707Z {"type":"log","host":"ild-bssc-indexsearch-data-1.ram-system","container":"is-data","level":"INFO","systemid":"BSSC-1234","system":"BSSC","time": "2026-07-08T16:04:59.724250Z","logger":"o.o.p.p.PrometheusExporterPlugin","timezone":"UTC","marker":"","log":{"message":"starting Prometheus exporter plugin"}}