Increasing "lucene" max number of dimensions

Plugins k-NN
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.6.0

Describe the issue:

At opensearch-project/k-NN/pull/490, the limits for faiss and nmslib were increased.

The lucene plugin limits remained aligned with FloatVectorValues.MAX_DIMENSIONS.

However Lucene devs, AFAICT, do not consider increasing this limit at the moment.See Increase KnnByteVectorField limit on dimensions to 2048 by benwtrent · Pull Request #12191 · apache/lucene · GitHub.

Would you folks consider increasing the lucene plugin limits like ElasticSearch did at PR-95257?

I haven’t wrapped my head around how exactly you included the Lucene k-NN, but I am very curious if anyone could give me an indication of how I could go around patching OpenSearch to increase these limits.

2

@FranciscoBorgesSFDC is there any reason why you are using lucene engine? If you need more dimensions then why not try faiss or nmslib? Opensearch supports these 2 more engines

3

@Navneet many use cases we have require pre-filtering to avoid truncation problems. Hence the desire to stick with the lucene engine.