# How to register sparse encoding model in AWS OpenSearch

**URL:** <https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112>\
**Category:** Machine Learning\
**Tags:** troubleshoot\
**Created:** [December 15, 2023, 4:04pm UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112 "2023-12-15T16:04:09Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![grunt-solaces.0h](https://avatars.discourse-cdn.com/v4/letter/g/b9e5f3/32.png) [@grunt-solaces.0h](https://forum.opensearch.org/u/grunt-solaces.0h)\
**Post date:** [December 15, 2023, 4:04pm UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/1 "2023-12-15T16:04:09Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):

AWS OpenSearch 2.11

**Describe the issue** :

I’m trying to deploy the pretrained model `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1` on AWS OpensSearch to use it for Neural Sparse Search but it doesn’t seem to work.

The full request:

```auto
POST /_plugins/_ml/models/_register
{
    "name": "amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1",
    "version": "1.0.0",
    "model_group_id": "<group-id>",
    "description": "This is a neural sparse encoding model: It transfers text into sparse vector, and then extract nonzero index and value to entry and weights. It serves only in ingestion and customer should use tokenizer model in query.",
    "model_format": "TORCH_SCRIPT",
    "function_name": "SPARSE_ENCODING",
    "model_content_hash_value": "9a41adb6c13cf49a7e3eff91aef62ed5035487a6eca99c996156d25be2800a9a",
    "url": "https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.0/torch_script/opensearch-neural-sparse-encoding-doc-v1-1.0.0-torch_script.zip"
}

```

Which causes OpenSearch to return the following error:

```auto
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "To upload custom model user needs to enable allow_registering_model_via_url settings. Otherwise please use opensearch pre-trained models."
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "To upload custom model user needs to enable allow_registering_model_via_url settings. Otherwise please use opensearch pre-trained models."
  },
  "status": 400
}

```

When I try to set the `allow_registering_model_via_url` to `true`, I get the next one:

Full request:

```auto
PUT /_cluster/settings
{
    "persistent": {
        "plugins.ml_commons.allow_registering_model_via_url": true
    }
}

```

Error:

```auto
{
  "Message": "Your request: '/_cluster/settings' payload is not allowed."
}

```

I suspect AWS doesn’t allow updating that setting.

Any ideas on how the model can be deployed?

Thank you!

---

<div class="post-metadata">

**Author:** ![dhrubo](https://avatars.discourse-cdn.com/v4/letter/d/9e8a1a/32.png) [@dhrubo](https://forum.opensearch.org/u/dhrubo)\
**Post date:** [December 15, 2023, 6:01pm UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/2 "2023-12-15T18:01:42Z")

</div>

> When I try to set the `allow_registering_model_via_url` to `true`, I get the next one:

@grunt-solaces.0h , Yes, in managed service we don’t allow this setting yet.

@xinyual could you please take a look at this issue?

Thanks  
Dhrubo

---

<div class="post-metadata">

**Author:** ![xinyual](https://avatars.discourse-cdn.com/v4/letter/x/65b543/32.png) [@xinyual](https://forum.opensearch.org/u/xinyual)\
**Post date:** [December 17, 2023, 2:32am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/3 "2023-12-17T02:32:17Z")

</div>

Hi @grunt-solaces.0h, thanks for your interests on neural sparse. In AOS, we cannot register via URL and in 2.11 you have to deploy model on sagemaker. See `CloudFormation Integration` in AOS [here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/cfn-template.html). Although we haven’t updated the doc, integration with sparse model is supported there.

---

<div class="post-metadata">

**Author:** ![grunt-solaces.0h](https://avatars.discourse-cdn.com/v4/letter/g/b9e5f3/32.png) [@grunt-solaces.0h](https://forum.opensearch.org/u/grunt-solaces.0h)\
**Post date:** [December 19, 2023, 9:48am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/4 "2023-12-19T09:48:42Z")

</div>

Thanks @dhrubo and @xinyual for taking the time to respond!

I deployed the model via SageMaker and now it works.

However, the throughput of the ingestion pipeline is quite low.

I’m using the following setup:

- AOS cluster: single node of `r6g.xlarge.search`, w/ `200GB` of `General Purpose (SSD) - gp3`
- SageMaker setup: `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1` deployed on 1 instance of `ml.r5.4xlarge`
- The documents are pushed in bulk and I’ve setup an ingestion pipeline which uses the model from SageMaker
- the index’s `refresh_interval` is set to `60s`

From what I can tell the docs are processed sequentially. This seem to be confirmed by the SageMaker metrics: CPU stays at ~1K% and memory at ~25% regardless of how many documents I include in the bulk request.

What could be done to speed it up?

Thank you!

---

<div class="post-metadata">

**Author:** ![rs-search](https://avatars.discourse-cdn.com/v4/letter/r/e274bd/32.png) [@rs-search](https://forum.opensearch.org/u/rs-search)\
**Post date:** [December 19, 2023, 9:59pm UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/5 "2023-12-19T21:59:02Z")

</div>

@xinyual Is CloudFormation Integration the only way to access `opensearch-neural-sparse-encoding-v1` in AWS OpenSearch? Is there any way in SageMaker to register the model directly by its [URL](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.0/torch_script/opensearch-neural-sparse-encoding-doc-v1-1.0.0-torch_script.zip) or is the use of CloudFormation a requirement here?

About CloudFormation, should I expect to see neural-sparse-encoding as an available option when following the steps under [Amazon SageMaker Template Integration](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/cfn-template.html#cfn-template-sm)? Any additional detail on the setup process for sparse encodings in OpenSearch service would be appreciated. I ran into the exact same problem and questions described in @grunt-solaces.0h 's first post and was not able to find answers in docs.

---

<div class="post-metadata">

**Author:** ![grunt-solaces.0h](https://avatars.discourse-cdn.com/v4/letter/g/b9e5f3/32.png) [@grunt-solaces.0h](https://forum.opensearch.org/u/grunt-solaces.0h)\
**Post date:** [December 20, 2023, 1:53pm UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/6 "2023-12-20T13:53:57Z")

</div>

@xinyual @dhrubo we’ve made some progress on the indexing performance by using a machine with more CPUs for AOS.

The reasoning behind: it seems like the thread pool `opensearch_ml_predict` is used for handling the calls to SageMaker and it seems like it’s being sized based on the number of CPUs the AOS instance has.

Is there a way to increase the number of threads for this thread pool via settings? Or is there another way to increase the parallelism of the ingestion pipeline? It would be ideal for us since at the moment the ingestion pipeline seems to be the bottleneck and the SageMaker endpoint underutilized.

Thank you!

---

<div class="post-metadata">

**Author:** ![xinyual](https://avatars.discourse-cdn.com/v4/letter/x/65b543/32.png) [@xinyual](https://forum.opensearch.org/u/xinyual)\
**Post date:** [December 21, 2023, 1:33am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/7 "2023-12-21T01:33:54Z")

</div>

Hi @rs-search , we could deploy sparse model via sagemaker and remote inference. See [here](https://opensearch.org/docs/latest/ml-commons-plugin/remote-models/index/) to use remote model. But we couldn’t register model via url. I think Neural sparse button is already available in AOS integration.

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/e/ec36abd1c3bb17e9f18bc34f07defedf594483ad.png)

---

<div class="post-metadata">

**Author:** ![xinyual](https://avatars.discourse-cdn.com/v4/letter/x/65b543/32.png) [@xinyual](https://forum.opensearch.org/u/xinyual)\
**Post date:** [December 21, 2023, 1:40am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/8 "2023-12-21T01:40:40Z")

</div>

Hi @grunt-solaces.0h Unfortunately, improving thread number doesn’t have a influence on inference speed. Please check sagemaker endpoint CPU utilization to see whether it is fully used. Actually for a better ingestion throughput, we prefer to use GPU instance like g4dn, g5 or p3

---

<div class="post-metadata">

**Author:** ![grunt-solaces.0h](https://avatars.discourse-cdn.com/v4/letter/g/b9e5f3/32.png) [@grunt-solaces.0h](https://forum.opensearch.org/u/grunt-solaces.0h)\
**Post date:** [January 3, 2024, 2:35pm UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/9 "2024-01-03T14:35:05Z")

</div>

Hi @xinyual,

Thanks for the answer!

We were wondering about the thread number for the following reasons:

1. We switched to a larger machine, with more CPUs and as a result it seems the ingestion pipeline is sending more documents for inference, in parallel
2. At first we used a CPU instance for inference and it was being underutilised as it was processing only X documents, where X seemed to be the number of CPUs in the OS instance
3. We switched to a GPU instance for inference which is again underutilised. Indeed, it does the inference faster per document, but it seems like the bottleneck is still the ingestion pipeline which doesn’t send as many documents to the SageMaker instance as it is capable of handling

---

<div class="post-metadata">

**Author:** ![zane\_neo](https://avatars.discourse-cdn.com/v4/letter/z/0ea827/32.png) [@zane\_neo](https://forum.opensearch.org/u/zane_neo)\
**Post date:** [January 4, 2024, 5:29am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/10 "2024-01-04T05:29:18Z")

</div>

Hi @grunt-solaces.0h, the predict thread pool is originally designed for local model CPU deployment case. For remote case, there’s bottleneck on httpclient, e.g. if you only have one instance sending requests to remote model, the remote model instance could be underutilized. We have identified another performance issue on httpclient and working on fix these by enabling user to configure MAX\_CONNECTION count in httpclient. But I think increasing predict thread pool size can also increase the performance in your case, although after this you might encounter httpclient performance bottleneck issue. But it’s good to have a try, you can change the `node.processors` value in opensearch.yml file, the predict thread pool size is calculated with: `num(node.processors) * 2`.

---

<div class="post-metadata">

**Author:** ![grunt-solaces.0h](https://avatars.discourse-cdn.com/v4/letter/g/b9e5f3/32.png) [@grunt-solaces.0h](https://forum.opensearch.org/u/grunt-solaces.0h)\
**Post date:** [January 4, 2024, 10:12am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/11 "2024-01-04T10:12:53Z")

</div>

Thanks @zane_neo , this makes sense!

Will `MAX_CONNECTION ` be configurable via `PUT _cluster/settings`?

We can’t set the `node.processors` value as we use OpenSearch deployed in Amazon and we don’t have access to that config so we’ll have to wait for the httpclient fix.

---

<div class="post-metadata">

**Author:** ![zane\_neo](https://avatars.discourse-cdn.com/v4/letter/z/0ea827/32.png) [@zane\_neo](https://forum.opensearch.org/u/zane_neo)\
**Post date:** [January 5, 2024, 3:21am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/12 "2024-01-05T03:21:25Z")

</div>

In fact, our current http client implementation is not optimal, I’ve created this issue: [[FEATURE] Replace blocking httpclient with async httpclient in remote inference · Issue #1839 · opensearch-project/ml-commons · GitHub](https://github.com/opensearch-project/ml-commons/issues/1839) to track this. BTW, if you can scale more instances(no need high performance ones) and deploy your model to these instances, it can be a workaround to improve the parallelism performance.

---

<div class="post-metadata">

**Author:** ![grunt-solaces.0h](https://avatars.discourse-cdn.com/v4/letter/g/b9e5f3/32.png) [@grunt-solaces.0h](https://forum.opensearch.org/u/grunt-solaces.0h)\
**Post date:** [January 5, 2024, 10:29am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/13 "2024-01-05T10:29:42Z")

</div>

Great, thank you @zane_neo ! We really appreciate it!

---

<div class="post-metadata">

**Author:** ![darvel](https://avatars.discourse-cdn.com/v4/letter/d/9dc877/32.png) [@darvel](https://forum.opensearch.org/u/darvel)\
**Post date:** [February 16, 2024, 9:37am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/15 "2024-02-16T09:37:18Z")

</div>

> [@grunt-solaces.0h](#):
>
> I deployed the model via SageMaker and now it works.

How deployed the model via SageMaker, please guide me?

---

<div class="post-metadata">

**Author:** ![xinyual](https://avatars.discourse-cdn.com/v4/letter/x/65b543/32.png) [@xinyual](https://forum.opensearch.org/u/xinyual)\
**Post date:** [February 21, 2024, 1:20am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/16 "2024-02-21T01:20:59Z")

</div>

Hi @darvel , sorry for late reply. Which version of OS do you use? If you use Amazon Opensearch in AWS, you can find integration in UI and it will help you deploy the sparse model on sagemaker. For open source version, please make sure your sparse endpoint takes list of string as input and output a list of \<String, float\> dictionary.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [April 21, 2024, 1:21am UTC](https://forum.opensearch.org/t/how-to-register-sparse-encoding-model-in-aws-opensearch/17112/17 "2024-04-21T01:21:35Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.