Can't generate embedding with a ML Huggingface model successfully deployed "Model no deployed"

[Adrien]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.7.0 OpenSearch
2.7.0 OpenSearch Dashboard
Docker-Compose 1.29.2
Firefox 113.0.1
Hugging Face model: microsoft/GODEL-v1_1-large-seq2seq
microsoft/GODEL-v1_1-large-seq2seq at main
Describe the issue:

I can’t generate embedding in Opensearch via opensearch-ml-py and the REST API. It displays 'TransportError(500, ‘m_l_exception’, ‘m_l_exception: model not deployed.’
I tried the version 2.8.0 of Opensearch, the error become worst… I also tried different opensearch-ml-py version but still got the same error display.

I investigated and I found that the error come from ml-commons/ml-algorithms/src/main/java/org/opensearch/ml/engine/algorithms/DLModel.java at 2.7 · opensearch-project/ml-commons · GitHub.
Actually, the API involve that my model isn’t deploy whereas it is. So the “predictor” variable is still null because the function load isn’t called. I can’t understand because I realized the deployment and the load step from the python API and the REST API…

I tried the example in the Demo Notebook to trace Sentence Transformers model — Opensearch-py-ml 1.0.0 documentation (opensearch-project.github.io).
Still doesn’t work…
I suppose 3 possible issues:

• Error when I convert the .bin model (Misconfiguration)
• Error to find the model path in Opensearch
• Error in the 2 last versions?

If you have any suggestions, let me know.

Configuration:
Docker-compose file:
version: ‘3’
3 nodes including 1 ML node, and 2 default nodes (data, cluster_manager…)

Model converter - py file

CLUSTER_URL = 'localhost:9200'

def get_os_client(cluster_url = CLUSTER_URL,
                  username='admin',
                  password='admin'):
    '''
    Get OpenSearch client
    :param cluster_url: cluster URL like ://ml-te-netwo--ff1736fa7db98ff2.elb.us-west-2.amazonaws.com:443
    :return: OpenSearch client
    '''
    client = OpenSearch(
        hosts=[cluster_url],
        http_auth=(username, password),
        verify_certs=False
    )
    return client

client = get_os_client()

import opensearch_py_ml as oml
from opensearch_py_ml.ml_commons import MLCommonClient
ml_client = MLCommonClient(client)

pre_trained_model = SentenceTransformerModel(folder_path = 'MicrosoftBot/', overwrite = True, model_id = "microsoft/GODEL-v1_1-large-seq2seq")

model_path = pre_trained_model.save_as_pt(model_id = "microsoft/GODEL-v1_1-large-seq2seq", save_json_folder_path = "MicrosoftBot/",model_name = "GODEL-v1_1-large-seq2seq", sentences=["This is an example sentence", "Each sentence is converted"])

print(model_path)
model_config_path = pre_trained_model.make_model_config_json(embedding_dimension=1024, verbose=True)
print(model_config_path)

Upload model:

#connect to ml_common client with OpenSearch client
from opensearch_py_ml.ml_commons import MLCommonClient
import warnings
warnings.filterwarnings('ignore', category=DeprecationWarning)
warnings.filterwarnings("ignore", message="Unverified HTTPS request")
from opensearchpy import OpenSearch

CLUSTER_URL = 'localhost:9200'

def get_os_client(cluster_url = CLUSTER_URL,
                  username='admin',
                  password='admin'):
    '''
    Get OpenSearch client
    :param cluster_url: cluster URL like https://ml-te-netwo-1s12ba42br23v-ff1736fa7db98ff2.elb.us-west-2.amazonaws.com:443
    :return: OpenSearch client
    '''
    client = OpenSearch(
        hosts=[cluster_url],
        http_auth=(username, password),
        verify_certs=False
    )
    return client

client = get_os_client()
ml_client = MLCommonClient(client)

model_path = '/home/adrien/Documents/OpenSearch/MicrosoftBot/GODEL-v1_1-large-seq2seq.zip'
model_config_path = '/home/adrien/Documents/OpenSearch/MicrosoftBot/ml-commons_model_config.json'

modelId = ml_client.upload_model(model_path, model_config_path, isVerbose=True)

print("model id:" + modelId)
model_info = ml_client.get_model_info(modelId)
print(model_info)

Relevant Logs or Screenshots:
Terminal logs:

warnings.warn(
No sentence-transformers model found with name /home/adrien/.cache/torch/sentence_transformers/microsoft_GODEL-v1_1-large-seq2seq. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /home/adrien/.cache/torch/sentence_transformers/microsoft_GODEL-v1_1-large-seq2seq were not used when initializing T5EncoderModel: ['lm_head.weight']
- This IS expected if you are initializing T5EncoderModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing T5EncoderModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
model file is saved to  MicrosoftBot/GODEL-v1_1-large-seq2seq
zip file is saved to  MicrosoftBot/GODEL-v1_1-large-seq2seq.zip 

MicrosoftBot/GODEL-v1_1-large-seq2seq.zip
reading config file from: MicrosoftBot/config.json
generating ml-commons_model_config.json file...

{'name': 'microsoft/GODEL-v1_1-large-seq2seq', 'version': 1, 'model_format': 'TORCH_SCRIPT', 'model_task_type': 'TEXT_EMBEDDING', 'model_config': {'model_type': 't5', 'embedding_dimension': 1024, 'framework_type': 'sentence_transformers', 'all_config': '{"_name_or_path": "/home/adrien/.cache/torch/sentence_transformers/microsoft_GODEL-v1_1-large-seq2seq", "architectures": ["T5EncoderModel"], "d_ff": 4096, "d_kv": 64, "d_model": 1024, "decoder_start_token_id": 0, "dense_act_fn": "relu", "dropout_rate": 0.1, "eos_token_id": 1, "feed_forward_proj": "relu", "initializer_factor": 1.0, "is_encoder_decoder": true, "is_gated_act": false, "layer_norm_epsilon": 1e-06, "model_type": "t5", "n_positions": 512, "num_decoder_layers": 24, "num_heads": 16, "num_layers": 24, "output_past": true, "pad_token_id": 0, "relative_attention_max_distance": 128, "relative_attention_num_buckets": 32, "task_specific_params": {"summarization": {"early_stopping": true, "length_penalty": 2.0, "max_length": 200, "min_length": 30, "no_repeat_ngram_size": 3, "num_beams": 4, "prefix": "summarize: "}, "translation_en_to_de": {"early_stopping": true, "max_length": 300, "num_beams": 4, "prefix": "translate English to German: "}, "translation_en_to_fr": {"early_stopping": true, "max_length": 300, "num_beams": 4, "prefix": "translate English to French: "}, "translation_en_to_ro": {"early_stopping": true, "max_length": 300, "num_beams": 4, "prefix": "translate English to Romanian: "}}, "torch_dtype": "float32", "transformers_version": "4.30.1", "use_cache": true, "vocab_size": 32102}'}}
ml-commons_model_config.json file is saved at :  MicrosoftBot/ml-commons_model_config.json

Dev tools console – model info:
Model deployed.

[dhrubo]

Hi @Adrien,

Thanks for your interest in Opensearch.

The save_as_pt function currently only support tracing Sentence transformers model.

Unfortunately, for microsoft/GODEL-v1_1-large-seq2seq we don’t have support for tracing. This is why you are seeing those warning.

If you think we should add support for any other models, please cut an issue here

For now, you need to trace the model by yourself and then you can upload the model.

[sribalajivelan]

Hi Team,

I deployed few times, after I deleted and try other models, it is getting below error, Now I can’t deploy any model on my open search, I’m using Open Search 2.9.0

POST /_plugins/_ml/models/_register
{
  "name": "huggingface/sentence-transformers/msmarco-distilbert-base-tas-b",
  "version": "1.0.1",
  "model_format": "TORCH_SCRIPT"
}

{
  "task_type": "REGISTER_MODEL",
  "function_name": "TEXT_EMBEDDING",
  "state": "FAILED",
  "worker_node": [
    "XqhzY-VESQeNYLAUZGqyww"
  ],
  "create_time": 1692967710823,
  "last_update_time": 1692967711601,
  "error": """Cannot invoke "java.util.List.contains(Object)" because "versions" is null""",
  "is_async": true
}

[dhrubo]

Could you please try again? I tried and it worked as expected.

Thanks
Dhrubo