Migrate OpenSearch 2.9.12 to 3.x

General Feedback
1

Hello,

currently there is an update problem OpenSearch version 2.9.12 to 3.2.0 (or general v3.). We are running on Ubuntu 24.04 with docker containers and can run without problems on version 2.19.2.

Maybe someone have more insights?

If we move to version 3.2.0 we will face:

org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[.opendistro_security/_rXP91ceQ6WSx4YgM5KHHA]] was created with version [1.3.1] but the minimum compatible version is [2.0.0]. It should be re-indexed in OpenSearch 2.x before upgrading to 3.2.0.

So we tried to the solution of Incompatible index on upgrade to v3.0.0 - #2 by jasonrojas but this is not possible because opensearch is not available even the port is opened:

docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2d29e1f55cb8 opensearchproject/opensearch-dashboards:3.2.0 “./opensearch-dashbo…” About an hour ago Up 4 minutes 0.0.0.0:5601->5601/tcp, [::]:5601->5601/tcp opensearch-dashboards
89627e440f94 opensearchproject/opensearch:3.2.0 “./opensearch-docker…” About an hour ago Up 4 minutes 0.0.0.0:9200->9200/tcp, [::]:9200->9200/tcp, 9300/tcp, 0.0.0.0:9600->9600/tcp, [::]:9600->9600/tcp, 9650/tcp opensearch-node1

Tried to connect via curl from the host maschine itself leads to:

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:9200

Next try within the container:

[opensearch@89627e440f94 tools]$ curl -v -k --cert /usr/share/opensearch/config/kirk.pem --key /usr/share/opensearch/config/kirk-key.pem -XPOST ‘https://localhost:9200/_reindex?wait_for_completion=true’ -H “Content-type:application/json” -d ‘{“source”:{“index”:“.opendistro_security”}, “dest”:{“index”:“.opendistro_security-re”}}’
Note: Unnecessary use of -X or --request, POST is already inferred.

  • Host localhost:9200 was resolved.
  • IPv6: ::1
  • IPv4: 127.0.0.1
  • Trying [::1]:9200…
  • connect to ::1 port 9200 from ::1 port 56490 failed: Connection refused
  • Trying 127.0.0.1:9200…
  • connect to 127.0.0.1 port 9200 from 127.0.0.1 port 34646 failed: Connection refused
  • Failed to connect to localhost port 9200 after 0 ms: Could not connect to server
  • closing connection #0
    curl: (7) Failed to connect to localhost port 9200 after 0 ms: Could not connect to server

Or another attemp to reset the whole security indices, without losing data:

./securityadmin.sh -cd ../securityconfig/ -icl -nhnv -cacert ../../../config/root-ca.pem -cert ../../../config/kirk.pem -key ../../../config/kirk-key.pem
Security Admin v7
Will connect to localhost:9200
ERR: Seems there is no OpenSearch running on localhost:9200 - Will exit

Some troubleshooting information:

netstat -tulpen | grep 9200
tcp 0 0 0.0.0.0:9200 0.0.0.0:* LISTEN 0 13277103 67995/docker-proxy
tcp6 0 0 :::9200 :::* LISTEN 0 13277104 68002/docker-proxy

Last log lines related to opensearch-node1:

…

opensearch-node1 | [2025-09-11T09:37:36,247][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [windows_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,248][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [waf_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,248][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [vpcflow_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,248][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [test_windows_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,249][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [s3_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,249][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_web_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,249][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_proxy_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,249][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_macos_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,250][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_compliance_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,250][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_cloud_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,250][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_apt_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,251][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [others_application_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,251][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [okta_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,251][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [network_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,251][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [netflow_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,252][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [m365_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,252][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [linux_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,252][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [gworkspace_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,252][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [github_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,253][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [dns_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,253][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [cloudtrail_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,254][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [azure_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,254][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [apache_access_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,254][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-node1] Loaded [ad_ldap_logtype.json] log type
opensearch-node1 | [2025-09-11T09:37:36,406][INFO ][o.o.t.TransportService ] [opensearch-node1] publish_address {172.18.0.2:9300}, bound_addresses {[::]:9300}
opensearch-node1 | [2025-09-11T09:37:36,408][INFO ][o.o.t.TransportService ] [opensearch-node1] Remote clusters initialized successfully.
opensearch-node1 | uncaught exception in thread [main]
opensearch-node1 | [2025-09-11T09:37:36,759][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-node1] uncaught exception in thread [main]
opensearch-node1 | org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[.opendistro_security/_rXP91ceQ6WSx4YgM5KHHA]] was created with version [1.3.1] but the minimum compatible version is [2.0.0]. It should be re-indexed in OpenSearch 2.x before upgrading to 3.2.0.
opensearch-node1 | at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:172) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | Caused by: java.lang.IllegalStateException: The index [[.opendistro_security/_rXP91ceQ6WSx4YgM5KHHA]] was created with version [1.3.1] but the minimum compatible version is [2.0.0]. It should be re-indexed in OpenSearch 2.x before upgrading to 3.2.0.
opensearch-node1 | at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:143) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:114) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:339) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:321) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:195) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.node.Node.start(Node.java:1860) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:346) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:420) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) ~[opensearch-3.2.0.jar:3.2.0]
opensearch-node1 | … 6 more
opensearch-node1 | java.lang.IllegalStateException: The index [[.opendistro_security/_rXP91ceQ6WSx4YgM5KHHA]] was created with version [1.3.1] but the minimum compatible version is [2.0.0]. It should be re-indexed in OpenSearch 2.x before upgrading to 3.2.0.
opensearch-node1 | at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:143)
opensearch-node1 | at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:114)
opensearch-node1 | at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:339)
opensearch-node1 | at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:321)
opensearch-node1 | at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:195)
opensearch-node1 | at org.opensearch.node.Node.start(Node.java:1860)
opensearch-node1 | at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:346)
opensearch-node1 | at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:420)
opensearch-node1 | at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168)
opensearch-node1 | at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159)
opensearch-node1 | at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)
opensearch-node1 | at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
opensearch-node1 | at org.opensearch.cli.Command.main(Command.java:101)
opensearch-node1 | at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125)
opensearch-node1 | at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91)
opensearch-node1 | For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log

So there are two problems

  1. .opendistro_security indices needs to be updated to compatible indices version >=2.0.0

  2. Migration is not possible because the API via 9200 is not working

The mentioned log file from the last log ouput is not available - Known problem

Regards

2

Hi @ved144kumar ,

In order to go from 1.x to 3.x you will first need to go upgrade it on 2.x.

Back up the security indices first, then we will delete them and restore them. This will recreate the versions on 2.19.2. Which afterwards the upgrade will run successfully giving the version is no longer restricting.

You can backup and restore using the following with security admin -

Backup

"/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -backup "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv

Restore

"/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv

Then after upgrading you can run the following to confirm versions of the new indices -

curl -XGET --cert ./kirk.pem --key ./kirk-key.pem --cacert ./root-ca.pem "https://localhost:9200/.opendistro_security/_settings?human"
{".opendistro_security":{"settings":{"index":{"replication":{"type":"DOCUMENT"},"creation_date_string":"2025-10-06T11:14:00.019Z","number_of_shards":"1","auto_expand_replicas":"0-all","provided_name":".opendistro_security","creation_date":"1759749240019","number_of_replicas":"1","uuid":"ZDKyfhKCTfaKuqIdhxfOtA","version":{"upgraded":"137237827","created_string":"2.19.2","upgraded_string":"3.2.0","created":"136408027"}}}}}

You can see the created_string is updated from backup, delete, and restore in 2.19.2, which was 1.3.20. Which enabled the upgrade to 3.2.0.

Leeroy.

3

Hi Leeroy,

thank you for the reply. Tried that but the “created_string” value is unchanged after the restore attempt.

[opensearch@4b24c0bd0790 config]$ "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=kirk,OU=client,O=client,L=test,C=de"
OpenSearch Version: 2.19.3
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch-cluster
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/config/opensearch-security/
Will update '/config' with /usr/share/opensearch/config/opensearch-security/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /usr/share/opensearch/config/opensearch-security/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /usr/share/opensearch/config/opensearch-security/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /usr/share/opensearch/config/opensearch-security/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /usr/share/opensearch/config/opensearch-security/audit.yml
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /usr/share/opensearch/config/opensearch-security/allowlist.yml
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
Done with success




[opensearch@4b24c0bd0790 config]$ curl -XGET --cert ./kirk.pem --key ./kirk-key.pem --cacert ./root-ca.pem "https://localhost:9200/.opendistro_security/_settings?human"
{".opendistro_security":{"settings":{"index":{"creation_date_string":"2022-03-10T12:59:27.661Z","number_of_shards":"1","auto_expand_replicas":"0-all","provided_name":".opendistro_security","creation_date":"1646917167661","number_of_replicas":"0","uuid":"b3C4GdHySNKATWe_QmK6gQ","version":{"upgraded":"136408127","created_string":"1.2.4","upgraded_string":"2.19.3","created":"1

Error message while starting with V3.0.0 image:

The index [[.opendistro-reports-instances/yNM6po2DRh2bJUcqjLpcCg]] was created with version [1.3.2] but the minimum compatible version is [2.0.0]. It should be re-indexed in OpenSearch 2.x before upgrading to 3.3.0.

4

Hi @thsul ,

Did you delete them after backing up before restoring? otherwise the old will remain.

Leeroy.

5

Thank you, after deleting .opendistro* and .admin* as well as restoring the backup, the new version is working.

1 Like