What is the purpose of load API?

Plugins Machine Learning
1

Hi,

I’m trying to setup a ML node into my cluster. I have my models trained and was able to run predict APIs on them. I want to know the purpose of Load API. I was able to call predict APIs without calling the Load API. Then what’s the purpose of loading a Model ? Will it help to reduce the latency of the predict requests ?

Second question: If a model is loaded in a node and if the node is restarted for some reason, will the loaded gets lost ? Do I have to load it back again ?

Thanks

2

@cheemscode1 Good question. We support two kinds of models in ml-commons: models trained by build-in algorithms and custom models.

  • models trained by build-in algorithms:
    We support some algorithms like K-Means, Linear Regression etc (refer to Supported Algorithms - OpenSearch documentation). You can train a model and predict without loading such models as their size generally is small. If don’t load model first, every predict request will read model from model index then initialize model instance and run predict. If load model (read model from index, init model instance) first, predict request will just use the model instance in memory directly. So you are right, the purpose of loading a model is to reduce latency.

  • Custom model (we only support text-embedding NLP model by 2.5):
    User can upload their own NLP model to ml-commons. Generally these model size is big, i.e 100MB+. If we don’t load model first, the predict request will read model from index which may takes a long time, and the time will increase if the model size is bigger, so not scalable. So for custom model, user have to load it first, then run predict.

If a model is loaded in a node and if the node is restarted for some reason, will the loaded gets lost ? Do I have to load it back again ?

We are going to support model auto-reload feature in 2.6, which will automatically check which model previously loaded on this node, then reload it when node re-starts. You can check this issue [FEATURE] Auto reload model when cluster rebooted/node rejoin · Issue #577 · opensearch-project/ml-commons · GitHub

1 Like