Failed upgrade from 1.3.6 to 2.4.0 but cannot go back

OpenSearch
,
1

hello, tried to upgrade in place from 1.3.6 to 2.4.0, it complained about old indices indexed by ES 6.8.11:

The index [[gl-events_22/06X3HOBYSLqduZ7Ee7_y3g]] was created with version [6.8.11] but the minimum compatible version is OpenSearch 1.0.0 (or Elasticsearch 7.0.0). It should be re-indexed in OpenSearch 1.x (or Elasticsearch 7.x) before upgrading to 2.4.0.

so i went back to 1.3.6, cleaned up and deleted some old ones (I’m not proficient with jq to check via json/rest api and those were old closed indices), tried again, it complained again, so went back to 1.3.6 again but now 1.3.6 says:

java.lang.IllegalArgumentException: indexCreatedVersionMajor is in the future: 9

guessing my indices are in some kind of limbo between 1.3 and 2.0 which makes them unreadable by both, any idea on how can i recover those data?

EDIT: i found out which indices prevented opensearch 2.* from starting, deleted corresponding dirs now getting following errors:

[2022-12-09T12:59:03,009][INFO ][o.o.c.c.JoinHelper       ] [4473a4e919fb] failed to join {4473a4e919fb}{BZsKa72CQim7i76BsbaGsg}{HHn_WmZcS12iKWvbWsMYYQ}{172.22.0.3}{172.22.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={4473a4e919fb}{BZsKa72CQim7i76BsbaGsg}{HHn_WmZcS12iKWvbWsMYYQ}{172.22.0.3}{172.22.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=18384, optionalJoin=Optional[Join{term=18385, lastAcceptedTerm=61, lastAcceptedVersion=51599, sourceNode={4473a4e919fb}{BZsKa72CQim7i76BsbaGsg}{HHn_WmZcS12iKWvbWsMYYQ}{172.22.0.3}{172.22.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={4473a4e919fb}{BZsKa72CQim7i76BsbaGsg}{HHn_WmZcS12iKWvbWsMYYQ}{172.22.0.3}{172.22.0.3:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
org.opensearch.transport.RemoteTransportException: [4473a4e919fb][172.22.0.3:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
        at org.opensearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1722) ~[opensearch-2.4.0.jar:2.4.0]
[...]
Caused by: java.lang.IllegalStateException: index [graylog_1616/2z-1n4NrTx6e9DznxWWDrA] version not supported: 6.8.11 minimum compatible index version is: 7.0.0      
[...]

it still complains about old inexistent indices, i know i shouldn’t delete them offline but i had no other choice, is there any way to make opensearch just ignore them at startup?

2

Hey @maniel

And no I’m not stalking you from Graylog :laughing:

I noticed this in the log file.

Caused by: org.opensearch.cluster.coordination.FailedToCommitClusterStateException: publication failed

Found something similar here.

EDIT:
I missed this section

Caused by: java.lang.IllegalStateException: index [graylog_1616/2z-1n4NrTx6e9DznxWWDrA] version not supported: 6.8.11 minimum compatible index version is: 7.0.0

If you don’t mind losing data, you could execute a curl command

root# curl -XDELETE http://localhost:9200/graylog_1616/

Or

Hope that helps

3

the thing is API isn’t available, those messages are shown before node (run as single) initializes, also as i said, index called graylog_1616 doesn’t exist physically, i deleted it’s directory, the metadata about it are saved somewhere, Monday i’ll play with opensearch-node command on a backup

4

Hey

If your running Graylog infront to Opensearch then the metadata would be in MongoDb.

db.index_sets.find()

OR

db.index_sets.find( { writeConcern: <document> } )
5

yeah, I’m aware of where it’s stored in graylog’s infrastructure, but i’m talking about opensearch itself, i’m not even launching graylog at this point