# Error When Loading Embedding Model Into Memory **URL:** \ **Category:** Machine Learning\ **Created:** [August 1, 2023, 4:32am UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351 "2023-08-01T04:32:55Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![jsukup](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jsukup](https://forum.opensearch.org/u/jsukup)\ **Post date:** [August 1, 2023, 4:32am UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/1 "2023-08-01T04:32:55Z")
**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser): - AWS OpenSearch Service 2.7 Dashboard - Google Chrome **Describe the issue** : I’m attempting to load a pretrained embedding model into memory using: POST /\_plugins/\_ml/models/\_register { “name”: “huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2”, “version”: “1.0.1”, “model\_format”: “TORCH\_SCRIPT” } This completes successfully and I’m provided with a model ID when I run this: GET /\_plugins/\_ml/tasks/k4w9r4kBBiUmBL-z11AC Response: { “model\_id”: “lIw9r4kBBiUmBL-z3FC-”, “task\_type”: “DEPLOY\_MODEL”, “function\_name”: “TEXT\_EMBEDDING”, “state”: “COMPLETED”, “worker\_node”: [ “l8xgkWdfQNGhuez3MPMAqw” ], “create\_time”: 1690862212827, “last\_update\_time”: 1690862310966, “is\_async”: true } However, when I try to load this model into memory using this: POST /\_plugins/\_ml/models/lIw9r4kBBiUmBL-z3FC-/\_load I’m given this response: { “model\_id”: “lIw9r4kBBiUmBL-z3FC-”, “task\_type”: “DEPLOY\_MODEL”, “function\_name”: “TEXT\_EMBEDDING”, “state”: “FAILED”, “worker\_node”: [ “l8xgkWdfQNGhuez3MPMAqw”, “NNPgI\_0\_QUaENi\_weYIkNQ” ], “create\_time”: 1690863161315, “last\_update\_time”: 1690863312176, **“error”: “”“{“l8xgkWdfQNGhuez3MPMAqw”:“model content changed”,“NNPgI\_0\_QUaENi\_weYIkNQ”:“model content changed”}”“”,** “is\_async”: true } I cannot find any information regarding what this error message means. Also, I was able to successfully load this model before and use it to create vector embeddings which I still have saved in an index. ---
**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\ **Post date:** [August 1, 2023, 7:20am UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/2 "2023-08-01T07:20:10Z")
Are you running on MacOS? There is a known issue [[BUG]Model content hash can't match original hash value · Issue #844 · opensearch-project/ml-commons · GitHub](https://github.com/opensearch-project/ml-commons/issues/844) ---
**Author:** ![jsukup](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jsukup](https://forum.opensearch.org/u/jsukup)\ **Post date:** [August 1, 2023, 3:56pm UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/3 "2023-08-01T15:56:35Z")
No, this in being run on AWS. I saw this post your reference as well, but I think it is regarding something different. ---
**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\ **Post date:** [August 1, 2023, 4:45pm UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/4 "2023-08-01T16:45:44Z")
Got it, can you share your cluster settings? Like how many data nodes, the EC2 instance types. We need to reproduce the error to dive deep. ---
**Author:** ![jsukup](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jsukup](https://forum.opensearch.org/u/jsukup)\ **Post date:** [August 1, 2023, 4:57pm UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/5 "2023-08-01T16:57:31Z")
![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/5/57df237d5cccba619e9613c669bf1d284068b881.png) NOTE: I just changed this to 4 nodes but it was 2 nodes during the issue outlined. ---
**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\ **Post date:** [August 1, 2023, 5:07pm UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/6 "2023-08-01T17:07:59Z")
Did you use dedicated master node? > NOTE: I just changed this to 4 nodes but it was 2 nodes during the issue outlined. Changing to 4 nodes can solve the problem? ---
**Author:** ![jsukup](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jsukup](https://forum.opensearch.org/u/jsukup)\ **Post date:** [August 2, 2023, 7:43pm UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/7 "2023-08-02T19:43:25Z")
The problem seems intermittent – sometimes the model load successfully, other times not. I’ve also tried using the “\_deploy” method rather than “\_load” which I read somewhere as an alternative approach, but it doesn’t seem to matter. ---
**Author:** ![dhrubo](https://avatars.discourse-cdn.com/v4/letter/d/9e8a1a/32.png) [@dhrubo](https://forum.opensearch.org/u/dhrubo)\ **Post date:** [August 2, 2023, 7:50pm UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/8 "2023-08-02T19:50:37Z")
I was able to reproduce the issue in my end. In my case first time when invoked `_load` api, it was partially loaded and I was able to generate embedding. But when I invoked `_unload` api and then invoke `_load` api, I was able to see the issue. I’ll try to deep dive more into this issue. In the meantime, I tried to reproduce this issue with bigger instances, but couldn’t reproduce there. We are transitioning to `_deploy` from `_load`. In the long run `_load` will be deprecated. ---
**Author:** ![jsukup](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jsukup](https://forum.opensearch.org/u/jsukup)\ **Post date:** [August 2, 2023, 8:33pm UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/9 "2023-08-02T20:33:14Z")
On a related note, another thing I’ve noticed is that sometimes a model loaded into memory (verified using `GET /_plugins/_ml/profile/models`) will sometimes seemingly `_unload` on its own after a given amount of time (again, verified using `GET /_plugins/_ml/profile/models` with an empty response of `{}`). - Is this behavior expected? - Is this due to idle time? - Is it best practice to `_unload` a model after use? ---
**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\ **Post date:** [August 2, 2023, 10:11pm UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/10 "2023-08-02T22:11:28Z")
@dhrubo may help reproduce and dive deep for this problem too. I haven’t done any research yet. But I guess that may be caused by small EC2 instance type. `t3.small.search` has just 2 vCPU, and 2 GB memory. That looks too constrained to run model. Guess that maybe cause some unexpected error and model unloaded. ---
**Author:** ![jsukup](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jsukup](https://forum.opensearch.org/u/jsukup)\ **Post date:** [August 3, 2023, 3:20am UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/11 "2023-08-03T03:20:30Z")
@ylwu Yes, I upgraded the instance and it seems to be performing better without throwing the error this time. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\ **Post date:** [October 2, 2023, 3:21am UTC](https://forum.opensearch.org/t/error-when-loading-embedding-model-into-memory/15351/12 "2023-10-02T03:21:15Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.