We have 3 nodes of 60GB Machines where 50% i.e. 30GB is for Heap and 30 GB is for File Cache. We have 3 indexes of 1 Primary and 2 Replicas each. We have calculated the KNN Graph Memory required for them is around 28GB in total (across 3).
What should we do once the graph grows out of that?
Should we Scale Horizontally by adding more Nodes of 60GB RAM or should we Scale Vertically by Adding Machines with higher RAM per node?
Also, where does the KNN Graph reside? in Heap or File Cache? Should I give more RAM to Heap or file Cache?
Make sure you include replicas in this calculation.
What should we do once the graph grows out of that?
Interesting, one recommendation I have would be to start out with more primary shards than number of nodes. For instance, if you start with 12 primaries, you could scale up to 12 nodes and the shards would be distributed very well.
Another recommendation I have would be to create some kind of roll over index policy. For instance, if you are ingesting new vectors every day, you could create a new index everyday. Then, you could search over all of the indices with 1 query using wild cards:
GET /myindex-*/_search
This would also allow you to “retire” data based on date.
I would recommend scaling horizontally when possible. It is easier to add/remove nodes as opposed to upgrading/downgrading nodes.
Either way, it would be helpful to first estimate the maximum number of vectors you will want to search at once.