Knn Doesnt coexist with Seismic Sparse

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.4

Describe the issue: The SEISMIC sparse model, which is an advancement over traditional sparse model, seems to have a plugin conflict when tried to enable knn as well. This means, we cannot index both seismic sparse and knn embeddings on the same index which prohibits hybrid search.

But traditional sparse and knn have no issues, when ingested on same index.

Attaching the screenshot.

Configuration:

index.sparse: true

index.knn: true

Relevant Logs or Screenshots:

image
image1920×941 212 KB

2

Any clarifications on this would be highly appreciated, Thanks!

3

Hi @Nagpraveen ,

At the moment, due to the way both plugins implemented, it is not possible to create an index that has knn and seismic sparse enabled at the same time. We are working towards lifting this limitation off (see please [1]), but it may take some time. Thank you.

[1] Introduce CodecRegistry and EnginePlugin::getAdditionalCodecs hook to allow additional Codec registration by reta · Pull Request #20411 · opensearch-project/OpenSearch · GitHub

4

@reta Amazing! Thank you for proposing the patch.

5

@reta : Can we expect this to be part of 3.5 release?

6

@Nagpraveen unfortunately not, as I mentioned it make take some time (I would expect multiple changes / release to fully address the problem).

7

@reta : Understood, Thank you !