Increasing "lucene" max number of dimensions

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:

At opensearch-project/k-NN/pull/490, the limits for faiss and nmslib were increased.

The lucene plugin limits remained aligned with FloatVectorValues.MAX_DIMENSIONS.

However Lucene devs, AFAICT, do not consider increasing this limit at the moment.See Increase KnnByteVectorField limit on dimensions to 2048 by benwtrent · Pull Request #12191 · apache/lucene · GitHub.

Would you folks consider increasing the lucene plugin limits like ElasticSearch did at PR-95257?

I haven’t wrapped my head around how exactly you included the Lucene k-NN, but I am very curious if anyone could give me an indication of how I could go around patching OpenSearch to increase these limits.

@FranciscoBorgesSFDC is there any reason why you are using lucene engine? If you need more dimensions then why not try faiss or nmslib? Opensearch supports these 2 more engines

@Navneet many use cases we have require pre-filtering to avoid truncation problems. Hence the desire to stick with the lucene engine.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.