# Deploying huggingface\_transformers on opensearch cluster
**URL:** https://forum.opensearch.org/t/deploying-huggingface-transformers-on-opensearch-cluster/21896
**Category:** Machine Learning
**Tags:** troubleshoot
**Created:** [October 9, 2024, 8:26pm UTC](https://forum.opensearch.org/t/deploying-huggingface-transformers-on-opensearch-cluster/21896 "2024-10-09T20:26:03Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@zbislaw.tabor](https://forum.opensearch.org/u/zbislaw.tabor)
#### Post date: [October 9, 2024, 8:26pm UTC](https://forum.opensearch.org/t/deploying-huggingface-transformers-on-opensearch-cluster/21896/1 "2024-10-09T20:26:03Z")
**Describe the issue** :
I am trying to deploy on opensearch cluster a transformer model but not a sentence transformer model. The minimal code for saving a model in torchscript together with tokenizer is:
################################################################
# [TorchScript](https://huggingface.co/docs/transformers/torchscript)
from transformers import BertTokenizer, BertConfig, BertModel
device = “cpu”
modelPreTrained = BertModel.from\_pretrained(“bert-base-uncased”,torchscript=True)
modelPreTrained.to(device)
modelPreTrained.eval()
tokenizer = BertTokenizer.from\_pretrained(‘bert-base-uncased’, do\_lower\_case=True)
text = “[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]”
tokenized\_text = tokenizer.tokenize(text)
# Masking one of the input tokens
masked\_index = 8
tokenized\_text[masked\_index] = “[MASK]”
indexed\_tokens = tokenizer.convert\_tokens\_to\_ids(tokenized\_text)
segments\_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]
# Creating a dummy input
tokens\_tensor = torch.tensor([indexed\_tokens])
segments\_tensors = torch.tensor([segments\_ids])
traced\_model = torch.jit.trace(modelPreTrained, [tokens\_tensor, segments\_tensors])
torch.jit.save(traced\_model, “traced\_bert.pt”)
with open(“tokenizer.json”, ‘w’) as f:
json.dump(tokenizer.get\_vocab(), f)
################################################################
After running the code I got to files: “traced\_bert.pt” and “tokenizer.json” which I pack to a single zip file “TransformerModel.zip” and compute shasum:
shasum -a 256 TransformerModel.zip
Then I prepare a “config.json” file e.g.:
{
“name”: “my\_model”,
“version”: “1.0.0”,
“model\_format”: “TORCH\_SCRIPT”,
“model\_content\_hash\_value”: “dbb914064ed6cc9617d72747b00616865388a18dc9210dba381fa41be091b9f5”,
“model\_config”: {
“model\_type”: “bert”,
“embedding\_dimension”: 768,
“framework\_type”: “huggingface\_transformers”
}
}
Given “TransformerModel.zip” and “config.json” I can register the model on opensearch cluster in some model group, which I have created earlier:
import opensearch\_py\_ml as oml
from opensearchpy import OpenSearch
from opensearch\_py\_ml.ml\_commons import MLCommonClient
host = ‘[sime.host.com](http://sime.host.com)’
port = 3005
auth = (‘joedoe’, ‘qwerty’)
client = OpenSearch(
hosts = [{‘host’: host, ‘port’: port}],
http\_auth = auth,
use\_ssl = True,
verify\_certs = False,
ssl\_assert\_hostname = False,
ssl\_show\_warn = False,
)
ml\_client = MLCommonClient(client)
model\_path = ‘./TransformerModel.zip’
model\_config\_path = ‘./config.json’
ml\_client.register\_model(model\_path, model\_config\_path, model\_group\_id = “JzMOKJIBC9ZdJM8aKaCz”,isVerbose=True,deploy\_model=False,wait\_until\_deployed=False)
The model is correctly uploaded to the cluster, registered and receives ID. Here is the end of the log from registering:
uploading chunk 39 of 41
Model id: {‘status’: ‘Uploaded’}
uploading chunk 40 of 41
Model id: {‘status’: ‘Uploaded’}
uploading chunk 41 of 41
Model id: {‘status’: ‘Uploaded’}
Model registered successfully
‘CTPrcpIBC9ZdJM8a-rme’
Now I go to the cluster and want to deploy the model. First I run:
GET /\_plugins/\_ml/models/CTPrcpIBC9ZdJM8a-rme
and get response:
{
“name”: “my\_model”,
“model\_group\_id”: “JzMOKJIBC9ZdJM8aKaCz”,
“algorithm”: “TEXT\_EMBEDDING”,
“model\_version”: “6”,
“model\_format”: “TORCH\_SCRIPT”,
“model\_state”: “REGISTERED”,
“model\_content\_size\_in\_bytes”: 405489551,
“model\_content\_hash\_value”: “dbb914064ed6cc9617d72747b00616865388a18dc9210dba381fa41be091b9f5”,
“model\_config”: {
“model\_type”: “bert”,
“embedding\_dimension”: 768,
“framework\_type”: “HUGGINGFACE\_TRANSFORMERS”
},
“created\_time”: 1728504920734,
“total\_chunks”: 41,
“is\_hidden”: false
}
So I run deploying:
POST /\_plugins/\_ml/models/CTPrcpIBC9ZdJM8a-rme/\_deploy
and got error “DEPLOY\_FAILED”
{
“name”: “my\_model”,
“model\_group\_id”: “JzMOKJIBC9ZdJM8aKaCz”,
“algorithm”: “TEXT\_EMBEDDING”,
“model\_version”: “6”,
“model\_format”: “TORCH\_SCRIPT”,
“model\_state”: “DEPLOY\_FAILED”,
“model\_content\_size\_in\_bytes”: 405489551,
…
Deploying sentence transformer models works well, according to opensearch-py-ml tutorial.
But how to correctly deploy transformer model like BertModel.from\_pretrained(“bert-base-uncased”,torchscript=True)?
I will appreciate any suggestions.
---
### Author:  [@system](https://forum.opensearch.org/u/system)
#### Post date: [December 8, 2024, 8:26pm UTC](https://forum.opensearch.org/t/deploying-huggingface-transformers-on-opensearch-cluster/21896/2 "2024-12-08T20:26:05Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.