Can't register a pre-trained model

OpenSearch
1

On an AWS managed service, version 2.17

Describe the issue:

trying to use one of the pretrained models listed on the docs:

the model is huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1
version 1.0.1
trying TORCH_FORMAT and ONNX

getting this error on the task:
“This model is not in the pre-trained model list, please check your parameters.”

what could i be doing wrong here?
thanks!

2

Hey, I just started getting this same issue yesterday and noticed the smallest change in the docs. The version for my model and your model is now 1.0.2 instead of 1.0.1.

I’m also on AWS managed OpenSearch. When I switched to model 1.0.2 I could register and deploy a pre-trained model again.

Don’t know why they’re actively blocking the old model versions… put on your tinfoil hats for that. But 1.0.2 should work.

3

Thanks for the reply!

You’re right but check this out, when using the TORCH_SCRIPT format on v 1.0.2 the registration succeeds but deploying throws this error:

“error”: “{"IDv1tSDKQvW1wgPGETaMWw":"\nUnknown builtin op: aten::scaled_dot_product_attention.\nHere are some suggestions: \n\taten::_scaled_dot_product_attention\n\nThe original call is:\n File \"code/torch/transformers/models/bert/modeling_bert.py\", line 165\n x1 = torch.view(_36, [_38, int(_39), 12, 32])\n value_layer = torch.permute(x1, [0, 2, 1, 3])\n attn_output = torch.scaled_dot_product_attention(query_layer, key_layer, value_layer, attention_mask)\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <— HERE\n attn_output0 = torch.transpose(attn_output, 1, 2)\n input = torch.reshape(attn_output0, [_26, _27, 384])\n"}”,

ONNX version works for registering and deploying, seems like something is up indeed, tinfoil hat firmly in place.

4

We are also experiencing this issue. We use the Docker image [opensearchproject/opensearch:2.19.2] as part of our CI/CD pipeline to run integration tests, and we’ve encountered the problem described above. We are able to register the model if we update the version to 1.0.2, but the model deployment then fails.

Just to also confirm, when I change the model format to ONNX I can register and deploy the model.

POST /_plugins/_ml/models/_register
{
  "name": "huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
  "version": "1.0.2",
  "model_format": "ONNX"
}

I’m assuming they are interchangeable but would guess the embedding values (floating points) could be different? :man_shrugging:

5

As far as I know both formats represent the same weights so I assume the outputted values would also be the same… I could be wrong tho

6

I have recently started using OpenSearch after I abandoned Elastic some years ago due to license changes. I am using both OpenSearch 3.0 and 2.17 and was using 2.17 (2 Node deployment) for the TORCH_FORMAT and encountered the issues using Python as it would download but not deploy. I have solved the issue for me by doing the following. But before I do that just some background on the tech stack used so you can see if your conditions approximate mine. I am running a host on hypervisor (in my case proxmox) enough RAM 20G, 8 Xeon cores, SSD disks but old tech. There is no GPU installed, OS Ubuntu 22.04LTS. My steps I took was as follows. The data frames API issue was my first problem and moved to “pandas==1.5.3”, Second I don’t have a GPU so installed the CPU version for torch “pip install torch==1.13.1 --index-url https://download.pytorch.org/whl/cpu” then the transformer version had to be changed as well “pip install transformers==4.37.2”. You will notice these are kinda old version of these packages and the reason for this is to try and match the version that were used for the version of OpenSearch you are using. Having said that I have not yet tested this for OpenSearch 3.0 and this will happen later for me once done with the training and when I want to use it in production. Hope this helps somebody with the same issues.