Update from 2.19 to 3.4.0-1 fails to start

OpenSearch
, ,
1

Versions 3.4.0-1

Describe the issue:
I have a cluster of 5 nodes with Opensearch 2.19.4-1 installed and and about 40TB.
I would like to update the cluster to v.3-x.

I started with one data node.

First I have updated java to version 21. The I updated opensearch to 3.4.0-1.
I am using the jvm.options generated after the installation.
Opensearch fails to start with error:

`[2026-01-27T12:44:31,531][ERROR][o.o.b.Bootstrap          ] [elastic-sdm-01] Exception`
`java.lang.IllegalArgumentException: Version id 7090199 must contain OpenSearch mask`
`at org.opensearch.Version.fromId(Version.java:211) ~[opensearch-core-3.4.0.jar:3.4.0]`
`at org.opensearch.common.settings.Setting.lambda$versionSetting$1(Setting.java:1274) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.common.settings.Setting.get(Setting.java:496) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.common.settings.Setting.get(Setting.java:490) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.cluster.metadata.IndexMetadata.indexCreated(IndexMetadata.java:2565) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.cluster.metadata.IndexMetadata$Builder.build(IndexMetadata.java:2190) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.cluster.metadata.IndexMetadata$Builder.fromXContent(IndexMetadata.java:2525) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.cluster.metadata.IndexMetadata.fromXContent(IndexMetadata.java:1500) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.gateway.PersistedClusterStateService.lambda$loadOnDiskState$1(PersistedClusterStateService.java:470) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.gateway.PersistedClusterStateService.consumeFromType(PersistedClusterStateService.java:515) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.gateway.PersistedClusterStateService.loadOnDiskState(PersistedClusterStateService.java:469) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:374) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:143) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.node.Node.start(Node.java:1874) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:347) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:421) [opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) [opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) [opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) [opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-3.4.0.jar:3.4.0]`
`at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) [opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) [opensearch-3.4.0.jar:3.4.0]`
`[2026-01-27T12:44:31,532][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [elastic-sdm-01] uncaught exception in thread [main]`
`org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Version id 7090199 must contain OpenSearch mask`
`at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:172) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.4.0.jar:3.4.0]`
`at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) ~[opensearch-3.4.0.jar:3.4.0]`
`Caused by: java.lang.IllegalArgumentException: Version id 7090199 must contain OpenSearch mask`
`at org.opensearch.Version.fromId(Version.java:211) ~[opensearch-core-3.4.0.jar:3.4.0]`
`at org.opensearch.common.settings.Setting.lambda$versionSetting$1(Setting.java:1274) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.common.settings.Setting.get(Setting.java:496) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.common.settings.Setting.get(Setting.java:490) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.cluster.metadata.IndexMetadata.indexCreated(IndexMetadata.java:2565) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.cluster.metadata.IndexMetadata$Builder.build(IndexMetadata.java:2190) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.cluster.metadata.IndexMetadata$Builder.fromXContent(IndexMetadata.java:2525) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.cluster.metadata.IndexMetadata.fromXContent(IndexMetadata.java:1500) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.gateway.PersistedClusterStateService.lambda$loadOnDiskState$1(PersistedClusterStateService.java:470) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.gateway.PersistedClusterStateService.consumeFromType(PersistedClusterStateService.java:515) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.gateway.PersistedClusterStateService.loadOnDiskState(PersistedClusterStateService.java:469) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:374) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:143) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.node.Node.start(Node.java:1874) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:347) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:421) ~[opensearch-3.4.0.jar:3.4.0]`
`at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) ~[opensearch-3.4.0.jar:3.4.0]`
`… 6 more`

I don’t understand what means: “Version id 7090199 must contain OpenSearch mask"

Can you please help me/ give me some advices?

I have data at

path.data: /es_data/opensearch

Configuration:

cat /etc/opensearch/jvm.options

JVM configuration

################################################################

IMPORTANT: JVM heap size

################################################################



You should always set the min and max JVM heap

size to the same value. For example, to set

the heap to 4 GB, set:



-Xms4g

-Xmx4g



See https://opensearch.org/docs/opensearch/install/important-settings/

for more information



################################################################

Xms represents the initial size of total heap space

Xmx represents the maximum size of total heap space

-Xms4g
-Xmx4g

################################################################

Expert settings

################################################################


All settings below this section are considered

expert settings. Don’t tamper with them unless

you understand what you are doing



################################################################

GC configuration

8-10:-XX:+UseConcMarkSweepGC
8-10:-XX:CMSInitiatingOccupancyFraction=75
8-10:-XX:+UseCMSInitiatingOccupancyOnly

G1GC Configuration

NOTE: G1GC is the default GC for all JDKs 11 and newer

11-:-XX:+UseG1GC

See https://github.com/elastic/elasticsearch/pull/46169 for the history

behind these settings, but the tl;dr is that default values can lead

to situations where heap usage grows enough to trigger a circuit breaker

before GC kicks in.

11-:-XX:G1ReservePercent=25
11-:-XX:InitiatingHeapOccupancyPercent=30

JVM temporary directory

-Djava.io.tmpdir=${OPENSEARCH_TMPDIR}

heap dumps

generate a heap dump when an allocation from the Java heap fails

heap dumps are created in the working directory of the JVM

-XX:+HeapDumpOnOutOfMemoryError

specify an alternative path for heap dumps; ensure the directory exists and

has sufficient space

-XX:HeapDumpPath=/var/lib/opensearch

specify an alternative path for JVM fatal error logs

-XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log

JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/opensearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

JDK 9+ GC logging

9-:-Xlog:gc\*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m

JDK 20+ Incubating Vector Module for SIMD optimizations;

disabling may reduce performance on vector optimized lucene

20-:–add-modules=jdk.incubator.vector

See please https://bugs.openjdk.org/browse/JDK-8341127 (openjdk/jdk#21283)

23:-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.setAsTypeCache
23:-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.asTypeUncached

21-:-javaagent:agent/opensearch-agent.jar
21-:–add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED

For cases with high memory-mapped file counts, a lower value can improve stability and

prevent issues like “leaked” maps or performance degradation. A value of 1 effectively

disables the shared Arena pooling and uses a confined Arena for each MMapDirectory

-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1

Relevant Logs or Screenshots:

2

@evelinabuttitta Is this the only error in your logs? Was 2.19 cluster upgraded from 1.x version previously or it is clean 2.19 installation?

3

Hi,
The cluster run for a year version 2.17. I updated successfully the all nodes to 2.19 today (as I read in the the docu about update to v 3).
I found only another error :

[2026-01-27T18:58:31,310][ERROR][o.o.p.c.c.PluginSettings ] [elastic-sdm-01] Loading config file /etc/opensearch/opensearch-performance-analyzer/performance-analyzer.properties failed with error: org.opensearch.performanceanalyzer.commons.config.ConfigFileException: java.io.FileNotFoundException: /etc/opensearch/opensearch-performance-analyzer/performance-analyzer.properties (No such file or directory). Disabling plugin.

Here more logs

[2026-01-27T18:58:23,819][INFO ][o.o.s.a.r.AuditMessageRouter] [elastic-sdm-01] Closing AuditMessageRouter
[2026-01-27T18:58:23,819][INFO ][o.o.n.Node               ] [elastic-sdm-01] stopping …
[2026-01-27T18:58:23,821][INFO ][o.o.s.a.s.SinkProvider   ] [elastic-sdm-01] Closing InternalOpenSearchSink
[2026-01-27T18:58:23,822][INFO ][o.o.s.a.s.SinkProvider   ] [elastic-sdm-01] Closing DebugSink
[2026-01-27T18:58:23,853][INFO ][o.o.n.Node               ] [elastic-sdm-01] stopped
[2026-01-27T18:58:23,853][INFO ][o.o.n.Node               ] [elastic-sdm-01] closing …
[2026-01-27T18:58:23,860][INFO ][o.o.s.a.i.AuditLogImpl   ] [elastic-sdm-01] Closing AuditLogImpl
[2026-01-27T18:58:23,879][INFO ][o.o.n.Node               ] [elastic-sdm-01] closed
[2026-01-27T18:58:27,743][WARN ][stderr                   ] [elastic-sdm-01] Jan 27, 2026 6:58:27 PM org.opensearch.javaagent.bootstrap.AgentPolicy setPolicy
[2026-01-27T18:58:27,746][WARN ][stderr                   ] [elastic-sdm-01] INFO: Policy attached successfully: org.opensearch.bootstrap.OpenSearchPolicy@10bea4
[2026-01-27T18:58:27,759][INFO ][o.o.n.Node               ] [elastic-sdm-01] version[3.4.0], pid[731174], build[rpm/00336141f90b2456d7aa35e9052fd6baf7147423/2025-12-15T21:42:54.003439444Z], OS[Linux/4.18.0-553.94.1.el8_10.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/25.0.1/25.0.1+8-LTS]
[2026-01-27T18:58:27,759][INFO ][o.o.n.Node               ] [elastic-sdm-01] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2026-01-27T18:58:27,759][INFO ][o.o.n.Node               ] [elastic-sdm-01] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,CLDR, -Xms4g, -Xmx4g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-9514615570534820402, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/opensearch, -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, --add-modules=jdk.incubator.vector, -javaagent:agent/opensearch-agent.jar, --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED, -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1, -XX:MaxDirectMemorySize=2147483648, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/etc/opensearch, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2026-01-27T18:58:27,803][WARN ][o.o.p.PluginsService     ] [elastic-sdm-01] Missing plugin [workload-management], dependency of [opensearch-security]
[2026-01-27T18:58:27,803][WARN ][o.o.p.PluginsService     ] [elastic-sdm-01] Some features of this plugin may not function without the dependencies being installed.

[2026-01-27T18:58:27,945][WARN ][stderr                   ] [elastic-sdm-01] Jan 27, 2026 6:58:27 PM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider 
[2026-01-27T18:58:27,945][WARN ][stderr                   ] [elastic-sdm-01] INFO: Java vector incubator API enabled; uses preferredBitSize=512; FMA enabled
[2026-01-27T18:58:28,749][INFO ][o.o.s.s.t.SSLConfig      ] [elastic-sdm-01] SSL dual mode is disabled
[2026-01-27T18:58:28,750][INFO ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] OpenSearch Config path is /etc/opensearch
[2026-01-27T18:58:29,948][INFO ][o.o.s.s.SslSettingsManager] [elastic-sdm-01] TLS HTTP Provider                    : JDK
[2026-01-27T18:58:29,949][INFO ][o.o.s.s.SslSettingsManager] [elastic-sdm-01] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2026-01-27T18:58:29,950][INFO ][o.o.s.s.SslSettingsManager] [elastic-sdm-01] TLS Transport Client Provider             : JDK
[2026-01-27T18:58:29,950][INFO ][o.o.s.s.SslSettingsManager] [elastic-sdm-01] TLS Transport Server Provider             : JDK
[2026-01-27T18:58:29,950][INFO ][o.o.s.s.SslSettingsManager] [elastic-sdm-01] Enabled TLS protocols for Transport layer : [TLSv1.3, TLSv1.2]
[2026-01-27T18:58:30,306][INFO ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] Clustername: elk-sdm-new
[2026-01-27T18:58:30,329][WARN ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] File /etc/opensearch/elastic-sdm-01.pem has insecure file permissions (should be 0600)
[2026-01-27T18:58:30,330][WARN ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] File /etc/opensearch/opensearch.keystore has insecure file permissions (should be 0600)
[2026-01-27T18:58:30,330][WARN ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] File /etc/opensearch/isrgrootx1.pem has insecure file permissions (should be 0600)
[2026-01-27T18:58:30,330][WARN ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] File /etc/opensearch/jvm.options.old has insecure file permissions (should be 0600)
[2026-01-27T18:58:30,330][WARN ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] File /etc/opensearch/opensearch.yml has insecure file permissions (should be 0600)
[2026-01-27T18:58:30,331][WARN ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] File /etc/opensearch/elastic-sdm-01.pem.puppet-bak has insecure file permissions (should be 0600)
[2026-01-27T18:58:31,310][ERROR][o.o.p.c.c.PluginSettings ] [elastic-sdm-01] Loading config file /etc/opensearch/opensearch-performance-analyzer/performance-analyzer.properties failed with error: org.opensearch.performanceanalyzer.commons.config.ConfigFileException: java.io.FileNotFoundException: /etc/opensearch/opensearch-performance-analyzer/performance-analyzer.properties (No such file or directory). Disabling plugin.
[2026-01-27T18:58:31,311][INFO ][o.o.p.c.c.PluginSettings ] [elastic-sdm-01] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: false, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2026-01-27T18:58:31,480][ERROR][o.o.p.c.c.StatsCollector ] [elastic-sdm-01] Error in loading metadata for folderLocation: /etc/opensearch/opensearch-performance-analyzer/, fileLocation: plugin-stats-metadata
java.io.FileNotFoundException: /etc/opensearch/opensearch-performance-analyzer/plugin-stats-metadata (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method) ~[?:?]
at java.base/java.io.FileInputStream.open(FileInputStream.java:185) ~[?:?]
at java.base/java.io.FileInputStream.(FileInputStream.java:139) ~[?:?]
at java.base/java.io.FileInputStream.(FileInputStream.java:109) ~[?:?]
at org.opensearch.performanceanalyzer.commons.collectors.StatsCollector.loadMetadata(StatsCollector.java:158) [performance-analyzer-commons-2.0.0.jar:?]
at org.opensearch.performanceanalyzer.commons.collectors.StatsCollector.instance(StatsCollector.java:73) [performance-analyzer-commons-2.0.0.jar:?]
at org.opensearch.performanceanalyzer.PerformanceAnalyzerPlugin.scheduleRcaCollectors(PerformanceAnalyzerPlugin.java:264) [opensearch-performance-analyzer-3.4.0.0.jar:3.4.0.0]
at org.opensearch.performanceanalyzer.PerformanceAnalyzerPlugin.(PerformanceAnalyzerPlugin.java:213) [opensearch-performance-analyzer-3.4.0.0.jar:3.4.0.0]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:864) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:815) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:613) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.plugins.PluginsService.(PluginsService.java:227) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.node.Node.(Node.java:555) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.node.Node.(Node.java:483) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:250) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:250) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-3.4.0.jar:3.4.0]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) [opensearch-3.4.0.jar:3.4.0]
[2026-01-27T18:58:31,485][INFO ][o.o.p.c.SearchBackPressureStatsCollector] [elastic-sdm-01] SearchBackPressureStatsCollector started
[2026-01-27T18:58:31,731][INFO ][o.o.t.g.GrpcPlugin       ] [elastic-sdm-01] Loading 1 QueryBuilderProtoConverter extensions from other plugins
[2026-01-27T18:58:31,732][INFO ][o.o.t.g.GrpcPlugin       ] [elastic-sdm-01] Discovered QueryBuilderProtoConverter extension: org.opensearch.knn.grpc.proto.request.search.query.KNNQueryBuilderProtoConverter (handles: KNN)
[2026-01-27T18:58:31,732][INFO ][o.o.t.g.GrpcPlugin       ] [elastic-sdm-01] Successfully loaded 1 QueryBuilderProtoConverter extensions
[2026-01-27T18:58:31,733][INFO ][o.o.t.g.GrpcPlugin       ] [elastic-sdm-01] Found 0 gRPC interceptor providers, will initialize during component creation
[2026-01-27T18:58:31,733][INFO ][o.o.t.g.GrpcPlugin       ] [elastic-sdm-01] Successfully loaded 0 GrpcServiceFactory extensions
[2026-01-27T18:58:31,735][INFO ][o.o.i.r.ReindexModulePlugin] [elastic-sdm-01] ReindexPlugin reloadSPI called
[2026-01-27T18:58:31,736][INFO ][o.o.i.r.ReindexModulePlugin] [elastic-sdm-01] Unable to find any implementation for RemoteReindexExtension
[2026-01-27T18:58:31,875][INFO ][o.o.s.r.ResourceActionGroupsHelper] [elastic-sdm-01] Registered 3 action-groups for forecaster
[2026-01-27T18:58:31,877][INFO ][o.o.s.r.ResourceActionGroupsHelper] [elastic-sdm-01] Registered 3 action-groups for anomaly-detector
[2026-01-27T18:58:31,879][INFO ][o.o.s.r.ResourceActionGroupsHelper] [elastic-sdm-01] Registered 3 action-groups for workflow_state
[2026-01-27T18:58:31,879][INFO ][o.o.s.r.ResourceActionGroupsHelper] [elastic-sdm-01] Registered 3 action-groups for workflow
[2026-01-27T18:58:31,880][INFO ][o.o.s.r.ResourceActionGroupsHelper] [elastic-sdm-01] Registered 3 action-groups for ml-model-group
[2026-01-27T18:58:31,883][INFO ][o.o.j.JobSchedulerPlugin ] [elastic-sdm-01] Loaded scheduler extension: opensearch_time_series_analytics, index: .opendistro-anomaly-detector-jobs
[2026-01-27T18:58:31,883][INFO ][o.o.j.JobSchedulerPlugin ] [elastic-sdm-01] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2026-01-27T18:58:31,884][INFO ][o.o.j.JobSchedulerPlugin ] [elastic-sdm-01] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap–job
[2026-01-27T18:58:31,911][INFO ][o.o.j.JobSchedulerPlugin ] [elastic-sdm-01] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2026-01-27T18:58:31,912][INFO ][o.o.j.JobSchedulerPlugin ] [elastic-sdm-01] Loaded scheduler extension: scheduled_experiment_job, index: .search-relevance-scheduled-experiment-jobs
[2026-01-27T18:58:31,913][INFO ][o.o.j.JobSchedulerPlugin ] [elastic-sdm-01] Loaded scheduler extension: opensearch_ml_commons_jobs, index: .plugins-ml-jobs
[2026-01-27T18:58:31,914][INFO ][o.o.j.JobSchedulerPlugin ] [elastic-sdm-01] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource

[2026-01-27T18:58:31,962][INFO ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting ‘http.compression: true’ in opensearch.yml
[2026-01-27T18:58:31,989][INFO ][o.o.e.NodeEnvironment    ] [elastic-sdm-01] using [1] data paths, mounts [[/es_data (/dev/mapper/vg02-lv_es_data)]], net usable_space [29.8tb], net total_space [43.6tb], types [xfs]
[2026-01-27T18:58:31,990][INFO ][o.o.e.NodeEnvironment    ] [elastic-sdm-01] heap size [4gb], compressed ordinary object pointers [true]
[2026-01-27T18:58:32,688][INFO ][o.o.n.Node               ] [elastic-sdm-01] node name [elastic-sdm-01], node ID [WIAEzo7OTTGw0v99o8OBUQ], cluster name [elk-sdm-new], roles [remote_cluster_client, data]
[2026-01-27T18:58:32,787][INFO ][o.o.e.ExtensionsManager  ] [elastic-sdm-01] ExtensionsManager initialized
[2026-01-27T18:58:34,181][WARN ][stderr                   ] [elastic-sdm-01] SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”.
[2026-01-27T18:58:34,182][WARN ][stderr                   ] [elastic-sdm-01] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2026-01-27T18:58:34,182][WARN ][stderr                   ] [elastic-sdm-01] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

[2026-01-27T18:58:36,736][WARN ][stderr                   ] [elastic-sdm-01] SLF4J(W): No SLF4J providers were found.
[2026-01-27T18:58:36,736][WARN ][stderr                   ] [elastic-sdm-01] SLF4J(W): Defaulting to no-operation (NOP) logger implementation
[2026-01-27T18:58:36,736][WARN ][stderr                   ] [elastic-sdm-01] SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
[2026-01-27T18:58:36,736][WARN ][stderr                   ] [elastic-sdm-01] SLF4J(W): Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
[2026-01-27T18:58:36,736][WARN ][stderr                   ] [elastic-sdm-01] SLF4J(W): Ignoring binding found at [jar:file:/usr/share/opensearch/plugins/repository-s3/log4j-slf4j-impl-2.21.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[2026-01-27T18:58:36,736][WARN ][stderr                   ] [elastic-sdm-01] SLF4J(W): See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
[2026-01-27T18:58:36,999][INFO ][o.o.s.e.ExperimentTaskManager] [elastic-sdm-01] ExperimentTaskManager initialized with max 24 concurrent tasks (processors: 48)
[2026-01-27T18:58:37,023][INFO ][o.o.r.m.c.i.SdkClientFactory] [elastic-sdm-01] Using local opensearch cluster as metadata store.
[2026-01-27T18:58:37,177][WARN ][o.o.s.p.SQLPlugin        ] [elastic-sdm-01] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information

[2026-01-27T18:58:37,877][INFO ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] plugin-additional-permissions.yml not found on classpath for plugin plugin:org.opensearch.jobscheduler.JobSchedulerPlugin, using empty permissions
[2026-01-27T18:58:37,881][INFO ][o.o.s.OpenSearchSecurityPlugin] [elastic-sdm-01] plugin-additional-permissions.yml not found on classpath for plugin plugin:org.opensearch.flowframework.FlowFrameworkPlugin, using empty permissions
[2026-01-27T18:58:38,720][INFO ][o.o.t.NettyAllocator     ] [elastic-sdm-01] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=512kb, suggested_max_allocation_size=512kb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=2mb}]
[2026-01-27T18:58:38,730][INFO ][o.o.s.s.t.SSLConfig      ] [elastic-sdm-01] SSL dual mode is disabled
[2026-01-27T18:58:38,827][INFO ][o.o.d.DiscoveryModule    ] [elastic-sdm-01] using discovery type [zen] and seed hosts providers [settings]
[2026-01-27T18:58:39,461][WARN ][o.o.g.DanglingIndicesState] [elastic-sdm-01] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2026-01-27T18:58:39,494][WARN ][o.o.m.a.p.TransportPredictionStreamTaskAction] [elastic-sdm-01] StreamTransportService is not available.
[2026-01-27T18:58:39,929][WARN ][o.o.m.a.e.TransportExecuteStreamTaskAction] [elastic-sdm-01] StreamTransportService is not available.
[2026-01-27T18:58:40,416][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [elastic-sdm-01] PerformanceAnalyzer Enabled: false
[2026-01-27T18:58:40,449][INFO ][o.o.n.Node               ] [elastic-sdm-01] initialized
[2026-01-27T18:58:40,450][INFO ][o.o.n.Node               ] [elastic-sdm-01] starting …


[2026-01-27T18:58:40,699][INFO ][o.o.t.TransportService   ] [elastic-sdm-01] publish_address {141.52.56.17:9300}, bound_addresses {[fe80::5097:11ff:fe56:f959]:9300}, {141.52.56.17:9300}, {[2a00:1398:4:9e0a::1d]:9300}
[2026-01-27T18:58:40,701][INFO ][o.o.t.TransportService   ] [elastic-sdm-01] Remote clusters initialized successfully.
[2026-01-27T18:58:41,100][ERROR][o.o.b.Bootstrap          ] [elastic-sdm-01] Exception
java.lang.IllegalArgumentException: Version id 7090199 must contain OpenSearch mask
at org.opensearch.Version.fromId(Version.java:211) ~[opensearch-core-3.4.0.jar:3.4.0]
at org.opensearch.common.settings.Setting.lambda$versionSetting$1(Setting.java:1274) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.common.settings.Setting.get(Setting.java:496) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.common.settings.Setting.get(Setting.java:490) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cluster.metadata.IndexMetadata.indexCreated(IndexMetadata.java:2565) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cluster.metadata.IndexMetadata$Builder.build(IndexMetadata.java:2190) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cluster.metadata.IndexMetadata$Builder.fromXContent(IndexMetadata.java:2525) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cluster.metadata.IndexMetadata.fromXContent(IndexMetadata.java:1500) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.gateway.PersistedClusterStateService.lambda$loadOnDiskState$1(PersistedClusterStateService.java:470) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.gateway.PersistedClusterStateService.consumeFromType(PersistedClusterStateService.java:515) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.gateway.PersistedClusterStateService.loadOnDiskState(PersistedClusterStateService.java:469) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:374) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:143) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.node.Node.start(Node.java:1874) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:347) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:421) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-3.4.0.jar:3.4.0]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) [opensearch-3.4.0.jar:3.4.0]
[2026-01-27T18:58:41,102][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [elastic-sdm-01] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Version id 7090199 must contain OpenSearch mask
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:172) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.4.0.jar:3.4.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) ~[opensearch-3.4.0.jar:3.4.0]
Caused by: java.lang.IllegalArgumentException: Version id 7090199 must contain OpenSearch mask
at org.opensearch.Version.fromId(Version.java:211) ~[opensearch-core-3.4.0.jar:3.4.0]
at org.opensearch.common.settings.Setting.lambda$versionSetting$1(Setting.java:1274) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.common.settings.Setting.get(Setting.java:496) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.common.settings.Setting.get(Setting.java:490) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cluster.metadata.IndexMetadata.indexCreated(IndexMetadata.java:2565) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cluster.metadata.IndexMetadata$Builder.build(IndexMetadata.java:2190) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cluster.metadata.IndexMetadata$Builder.fromXContent(IndexMetadata.java:2525) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.cluster.metadata.IndexMetadata.fromXContent(IndexMetadata.java:1500) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.gateway.PersistedClusterStateService.lambda$loadOnDiskState$1(PersistedClusterStateService.java:470) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.gateway.PersistedClusterStateService.consumeFromType(PersistedClusterStateService.java:515) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.gateway.PersistedClusterStateService.loadOnDiskState(PersistedClusterStateService.java:469) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:374) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:143) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.node.Node.start(Node.java:1874) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:347) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:421) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) ~[opensearch-3.4.0.jar:3.4.0]
… 6 more
4

Hi,

I found in the project files where is generated this error:

It is somehow related to lucene version with the opensearch version???

Do you have other ideas?

In my server I have installed java-11-openjdk-devel

[root@elastic-sdm-01 ~]# yum list installed | grep java
java-11-openjdk.x86_64 1:11.0.25.0.9-2.el8 @rhel-8-for-x86_64-appstream-rpms
java-11-openjdk-devel.x86_64 1:11.0.25.0.9-2.el8 @rhel-8-for-x86_64-appstream-rpms
java-11-openjdk-headless.x86_64 1:11.0.25.0.9-2.el8 @rhel-8-for-x86_64-appstream-rpms
java-21-openjdk.x86_64 1:21.0.10.0.7-1.el8 @rhel-8-for-x86_64-appstream-rpms
java-21-openjdk-devel.x86_64 1:21.0.10.0.7-1.el8 @rhel-8-for-x86_64-appstream-rpms
java-21-openjdk-headless.x86_64 1:21.0.10.0.7-1.el8 @rhel-8-for-x86_64-appstream-rpms
javapackages-filesystem.noarch 5.3.0-1.module+el8+2447+6f56d9a6 @rhel-8-for-x86_64-appstream-rpms
tzdata-java.noarch

And version of opensearch is 3.4.0-1

[root@elastic-sdm-01 ~]# yum list installed | grep opensearch
opensearch.x86_64 3.4.0-1

5

@evelinabuttitta How did you deploy your cluster? Is it a virtual machine or Docker container?

6

Hi, the cluster is deployed on 5 bare-metal servers (RHEL 8)

7

I encountered this issue when upgrading an older cluster as well.

The root cause turned out to be a index that was created all the way back on Elasticsearch 7.10, specifically a opendistro configuration index. Part of the reason this happened may have been that it was a hidden index, which may

The index name was restored_.opendistro-alerting-config so I belive that this for us was a leftover from a S3 backup restore test.

In the end deleting this index made the update work. I did also have to delete the entire data directory of the node that failed to start and had it recover from the other nodes.. but if you make a note of the index uid of the old index beforehand I belive you can figure out which directory specifically belings to that index. (Of course any changes to the data directory is on your own risk).

8

@albgus thanks for your comment.
Also in my case the problem is due to old indices present in my data (indices created with ES 7.9 or Opensearch 1.x). This causes the above error updating to opensearch v.3
In the logs “Version id 7090199” means ES 7.9.
I need first to find out all old indices, then reindex or delete them and try again the update to v3.

I am still working on this…since I have a lot of data.
For more info see ticket [BUG] Bootstrap error not clear when starting OpenSearch 3.x with an unsupported index version · Issue #20499 · opensearch-project/OpenSearch · GitHub

9

Glad you sorted it out. It seems like the strict version check in 3.0 is catching a lot of people off guard with these legacy system indices. Deleting the node data to force a sync from the master is a bit drastic, but often the quickest way to get a stuck node back online.