Unknown setting [index.replication.type] while trying to create index during upgrade

OpenSearch
1

Versions:
OpenSearch - 1.2.4 to 2.9.0

Describe the issue:
Hi,
We created two docker images. One with OpenSearch 1.2.4 and one with 2.9.0
We created respective helm charts by ourselves with respective docker images.
Then, we’re trying to perform rolling upgrade from OpenSearch 1.2.4 to 2.9.0 (Deployment has 4 data, 3 master).
Since it’s rolling upgrade, old pods are terminated ad new pods are spanned one by one.
So for sometime, both version of OpenSearch co-exists in same cluster.

During this time, new index (with 5 primary shards) creation failed with error in the console log saying

failing shard [failed shard, shard [test-logs-2024.01.24][3], node[yp-zXhOiToOZNiA0PUH49g], [P], recovery_source[new shard recovery], s[INITIALIZING], a[id=JphFN1hUTuaHYuqDo5DfnA], unassigned_info[[reason=INDEX_CREATED], at[2024-02-14T09:59:08.993Z], delayed=false, allocation_status[no_attempt]], message [failed to create index], failure [IllegalArgumentException[unknown setting [index.replication.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings]], markAsStale [true]]

New index is created but few shards are in UNASSIGNED state and few assigned are assigned to new data pods.

Did anyone faced this kind of issue?

2

Tested with this helm chart too (Upgraded from 1.9.0 to 2.9.0) helm-charts/charts/opensearch at main · opensearch-project/helm-charts · GitHub by keeping replica as 5 and number of primary shards as 5.
It’s throwing same error.

[2024-02-22T11:14:33,831][WARN ][o.o.c.r.a.AllocationService] [opensearch-cluster-master-4] failing shard [failed shard, shard [test1-137][3], node[y8c2aPyNRwiwXC015XV07Q], [P], recovery_source[new shard recovery], s[INITIALIZING], a[id=UX2iCc5uQ6euEt5dnfcz4g], unassigned_info[[reason=INDEX_CREATED], at[2024-02-22T11:14:32.947Z], delayed=false, allocation_status[no_attempt]], message [failed to create index], failure [IllegalArgumentException[unknown setting [index.replication.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings]], markAsStale [true]]
java.lang.IllegalArgumentException: unknown setting [index.replication.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:589) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:530) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:500) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.indices.IndicesService.createIndexService(IndicesService.java:698) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.indices.IndicesService.createIndex(IndicesService.java:611) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.indices.IndicesService.createIndex(IndicesService.java:192) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:510) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:261) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:591) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:578) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:546) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:469) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.cluster.service.ClusterApplierService.access$000(ClusterApplierService.java:81) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:180) ~[opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:275) [opensearch-2.9.0.jar:2.9.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:238) [opensearch-2.9.0.jar:2.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
3

This looks like a bug. index.replication.type is a new setting introduced in 2.x. It does look like OpenSearch 2.x will unconditionally add that setting to the index settings upon index creation, falling back to the default if it is not actually specified (OpenSearch/server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java at main · opensearch-project/OpenSearch · GitHub). As you’ve noted this causes a problem during a rolling upgrade as 1.x will not understand this setting. @manoj I’ll open a bug in GitHub for this, thanks!

4

@manoj Upon thinking about this a bit more, I’m not sure this is actually a bug. We generally don’t guarantee forward compatibility. In this case, the issue is that an index created by OpenSearch 2.x cannot be used by OpenSearch 1.x, which is not a compatibility guarantee offered by the system.

5

Hi @andrross ,
I guess, according to Rolling Upgrade - OpenSearch Documentation, we should able to continue to index (including new index creation) and query data.