Load existing Faiss index

Plugins k-NN
1

Hello,

Can I use KNN plugin to load existing Faiss index? If possible what’s the procedure?

Thanks,
Wei

2

What is the meaning of existing faiss index? Is it like created outside opensearch and you want to load that index?

@Wei2023

3

Hi Naveet,

Yes the index is created outside opensearch and I want to load it. Also how is Faiss index built in opensearch, is it separated with the lucene index segments?

Thanks,
Wei

4

So currently there is no way to load that Faiss index to opensearch index.

To ans other question we use lucene segments as a base to generate the faiss index. So you can think of if lucene has 10 segments per shard(i can check on this more, my knowledge here is little outdated) then we will have like 10 faiss index… This is on very very high level lucene segments are used to generate the faiss index.

@Wei2023

5

Thanks Naveet. We have an offline process to build Faiss index, so we are looking at using the faiss JNI lib to load the prebuild Faiss index for query serving. If each lucene segment will generate a Faiss index, is each Faiss index searched sequentially and then the results from all Faiss indexes merged per shard? What will happen when Lucene segments merge, will the corresponding Faiss indexes also merge?

6

@Wei2023
So the faiss index which are created per segment are searched sequentially in a shard. But multiple shards are searched in parallel.

When lucene segments are merged the underline faiss indices are also merged.

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.