# Model is Partially responding in case of non-ML node restart

**URL:** <https://forum.opensearch.org/t/model-is-partially-responding-in-case-of-non-ml-node-restart/17030>\
**Category:** Machine Learning\
**Created:** [December 11, 2023, 5:12pm UTC](https://forum.opensearch.org/t/model-is-partially-responding-in-case-of-non-ml-node-restart/17030 "2023-12-11T17:12:04Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![andrii](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@andrii](https://forum.opensearch.org/u/andrii)\
**Post date:** [December 11, 2023, 5:12pm UTC](https://forum.opensearch.org/t/model-is-partially-responding-in-case-of-non-ml-node-restart/17030/1 "2023-12-11T17:12:05Z")

</div>

OpenSearch 2.11.1  
Cluster configuration:  
3 master nodes  
3 ingest nodes  
3 data nodes  
3 ML nodes  
The cluster is deployed on Kubernetes.

opensearch.yaml has the following section:

```auto
    plugins.ml_commons.only_run_on_ml_node: true
    plugins.ml_commons.task_dispatch_policy: round_robin
    plugins.ml_commons.max_ml_task_per_node: 10
    plugins.ml_commons.max_model_on_node: 10
    plugins.ml_commons.sync_up_job_interval_in_seconds: 3
    plugins.ml_commons.monitoring_request_count: 100
    plugins.ml_commons.max_register_model_tasks_per_node: 10
    plugins.ml_commons.max_deploy_model_tasks_per_node: 10
    plugins.ml_commons.allow_registering_model_via_url: false
    plugins.ml_commons.native_memory_threshold: 90
    plugins.ml_commons.model_auto_redeploy.enable: true
    plugins.ml_commons.model_auto_redeploy.lifetime_retry_times: 5

```

For some reason, a cluster is trying to deploy a model on any non-ML node once it is restarted.  
and the model becomes Partially responding:

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/e/eb318c383084f3ecbdbc790243fa3ae4c1bf3431.png)

What is interesting is that it doesn’t matter how many nodes will be restarted. Only one (the last one) is mentioned on the Model status UI.

is the any option that prevents such a behavior?

---

<div class="post-metadata">

**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\
**Post date:** [December 11, 2023, 7:49pm UTC](https://forum.opensearch.org/t/model-is-partially-responding-in-case-of-non-ml-node-restart/17030/2 "2023-12-11T19:49:44Z")

</div>

@zane_neo , Zan can you help take a look ?

---

<div class="post-metadata">

**Author:** ![zane\_neo](https://avatars.discourse-cdn.com/v4/letter/z/0ea827/32.png) [@zane\_neo](https://forum.opensearch.org/u/zane_neo)\
**Post date:** [December 12, 2023, 3:07am UTC](https://forum.opensearch.org/t/model-is-partially-responding-in-case-of-non-ml-node-restart/17030/3 "2023-12-12T03:07:34Z")

</div>

@andrii , can you check the model planning worker nodes and deploy\_to\_all infos in the index by invoking `/_ml/models/_search` or `/_ml/models/{your_model_id}?return_content=false` API?

---

<div class="post-metadata">

**Author:** ![andrii](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@andrii](https://forum.opensearch.org/u/andrii)\
**Post date:** [December 12, 2023, 2:55pm UTC](https://forum.opensearch.org/t/model-is-partially-responding-in-case-of-non-ml-node-restart/17030/4 "2023-12-12T14:55:14Z")

</div>

@zane_neo here is an output of the command:

```auto
{
  "name": "huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1",
  "model_group_id": "T1MSRowBZ52_YC-EEcLS",
  "algorithm": "TEXT_EMBEDDING",
  "model_version": "1",
  "model_format": "TORCH_SCRIPT",
  "model_state": "DEPLOYED",
  "model_content_size_in_bytes": 91794759,
  "model_content_hash_value": "51f09df55d16debce6caaf381004b243a9131c4e64245cbe4bf51e788dc20196",
  "model_config": {
    "model_type": "bert",
    "embedding_dimension": 384,
    "framework_type": "SENTENCE_TRANSFORMERS",
    "all_config": """{"_name_or_path":"nreimers/MiniLM-L6-H384-uncased","architectures":["BertModel"],"attention_probs_dropout_prob":0.1,"gradient_checkpointing":false,"hidden_act":"gelu","hidden_dropout_prob":0.1,"hidden_size":384,"initializer_range":0.02,"intermediate_size":1536,"layer_norm_eps":1e-12,"max_position_embeddings":512,"model_type":"bert","num_attention_heads":12,"num_hidden_layers":6,"pad_token_id":0,"position_embedding_type":"absolute","transformers_version":"4.8.2","type_vocab_size":2,"use_cache":true,"vocab_size":30522}"""
  },
  "created_time": 1701982684193,
  "last_updated_time": 1702334722824,
  "last_registered_time": 1701982695005,
  "last_deployed_time": 1702334722823,
  "auto_redeploy_retry_times": 0,
  "total_chunks": 10,
  "planning_worker_node_count": 3,
  "current_worker_node_count": 3,
  "planning_worker_nodes": [
    "fNjI1HT0RBCd3Pd17j4pLg",
    "N1bKhyYGTle6NukCtdEI3w",
    "Q9_T0r2yTmuEh9xF6xZCPw"
  ],
  "deploy_to_all_nodes": true
}

```

In planning\_worker\_nodes I see only ML nodes:

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/3438828e3e842e67738be68143e86c8862ceae58.png)

---

<div class="post-metadata">

**Author:** ![zane\_neo](https://avatars.discourse-cdn.com/v4/letter/z/0ea827/32.png) [@zane\_neo](https://forum.opensearch.org/u/zane_neo)\
**Post date:** [December 13, 2023, 5:40am UTC](https://forum.opensearch.org/t/model-is-partially-responding-in-case-of-non-ml-node-restart/17030/5 "2023-12-13T05:40:08Z")

</div>

Thanks, @andrii, current model status looks correct, this should be a temporary issue, I’ll try to reproduce this issue and get back.

---

<div class="post-metadata">

**Author:** ![zane\_neo](https://avatars.discourse-cdn.com/v4/letter/z/0ea827/32.png) [@zane\_neo](https://forum.opensearch.org/u/zane_neo)\
**Post date:** [December 18, 2023, 12:46am UTC](https://forum.opensearch.org/t/model-is-partially-responding-in-case-of-non-ml-node-restart/17030/6 "2023-12-18T00:46:35Z")

</div>

Hi @andrii , this issue is bug which only affects profile API/ML dashboard, it’s a metadata issue and service won’t actually try to deploy model to data node, I’ve created  
this issue to track/fix this issue: [Machine Learning Dashboard shows partially response after restarting non-ml nodes · Issue #1774 · opensearch-project/ml-commons · GitHub](https://github.com/opensearch-project/ml-commons/issues/1774)

---

<div class="post-metadata">

**Author:** ![andrii](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@andrii](https://forum.opensearch.org/u/andrii)\
**Post date:** [December 18, 2023, 5:19pm UTC](https://forum.opensearch.org/t/model-is-partially-responding-in-case-of-non-ml-node-restart/17030/7 "2023-12-18T17:19:40Z")

</div>

thanks a lot @zane_neo

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [February 16, 2024, 5:20pm UTC](https://forum.opensearch.org/t/model-is-partially-responding-in-case-of-non-ml-node-restart/17030/8 "2024-02-16T17:20:24Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.