What parameters range to consider to experiment with Lucene; Faiss engine types using HNSW algorithm

Open Source Elasticsearch and Kibana
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch 2.17

Describe the issue: I want to know the range of parameters to consider or experiment with.

For Lucene-HNSW
ef_construction, m, while index creation and K in search query

For Faiss-Lucene
ef_construction, m,and ef_search while index creation and K in search query

I have index containing15M documents and we have to scale to 90M with embeddings dimension as 768. I want to know what impact these parameters might have and what would be the ideal parameters to use or a range to experiment with.

Can anyone share their experience with the parameters and what worked for them with how many documents and embeddings size would be greatly appreciated

2

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.