Upgrade to opensearch 2.17.1 blocked because index.knn is set to false

Plugins k-NN
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

  • OpenSearch: 2.17.1
  • Running in GKE

Describe the issue:
When upgrading opensearch from 2.12.0 to 2.17.1, we had the bad surprise of seeing this error message showing up and blocking the upgrade from being successful:

Caused by: org.opensearch.index.mapper.MapperParsingException: Failed to parse mapping [_doc]: Cannot set modelId or method parameters when index.knn setting is false
    at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:479) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:465) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.index.mapper.MapperService.merge(MapperService.java:451) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:248) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:121) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:298) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:280) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:198) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.node.Node.start(Node.java:1623) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) ~[opensearch-2.17.1.jar:2.17.1]
    at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]

The index in question was not created with index.knn set to “true” and we have one mapping with type “knn_vector”. Now I’m wondering whether or not we need to recreate the index completely with this value set to “true” or if there’s a way around it… We also can’t get rid of the knn vector mapping.

As far as I could see this value can’t be updated so I feel like there’s no other choice but that but at the same time I’m not very well versed in opensearch.

Any feedback will be appreciated, thanks!

2

@SilentSib index.knn is an index dynamic setting. Therefore it can’t be changed while the index is opened.
You need to first close the index, make the change and open the index again.
Please be aware that once the index is closed no data can be ingested, indexed or queried until the index is opened again.
I suggest to have a snapshot of the index before applying these steps.

POST /test-index/_close

PUT /test-index/_settings
{
    "index": {
      "knn": true
  }
}

POST /test-index/_open
3

Thanks a lot @pablo, that worked like a charm, chose your reply as the solution.

1 Like
4

This looks like a bug on our side. Will follow up with a GH issue.

Basically, before 2.17, an index could specify index.knn=false and also a method or model_id in the mapping. However, this method or model would be a noop because no ANN structures would be built with index.knn=false.

In 2.17, we added this validation to block this case: k-NN/src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java at 499273660065403e13e9781e3b9a9931b59f1bf6 · opensearch-project/k-NN · GitHub. However, this caused a break in BWC. Will follow up with GH issue.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.