Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): AWS OpenSearch 2.11 Describe the issue: I’m trying to deploy the pretrained model amazon/neural-sparse/opensearch-neural-sparse-encoding-v1 on AWS OpensSearch to use it for Neural Sparse Search but it doesn’t seem to work. The full re…

Hi @grunt-solaces.0h , thanks for your interests on neural sparse. In AOS, we cannot register via URL and in 2.11 you have to deploy model on sagemaker. See CloudFormation Integration in AOS here . Although we haven’t updated the doc, integration with sparse model is supported there.

When I try to set the allow_registering_model_via_url to true, I get the next one: @grunt-solaces.0h , Yes, in managed service we don’t allow this setting yet. @xinyual could you please take a look at this issue? Thanks Dhrubo

Thanks @dhrubo and @xinyual for taking the time to respond! I deployed the model via SageMaker and now it works. However, the throughput of the ingestion pipeline is quite low. I’m using the following setup: AOS cluster: single node of r6g.xlarge.search, w/ 200GB of General Purpose (SSD) - gp3 S…

@xinyual Is CloudFormation Integration the only way to access opensearch-neural-sparse-encoding-v1 in AWS OpenSearch? Is there any way in SageMaker to register the model directly by its URL or is the use of CloudFormation a requirement here? About CloudFormation, should I expect to see neural-spars…

@xinyual @dhrubo we’ve made some progress on the indexing performance by using a machine with more CPUs for AOS. The reasoning behind: it seems like the thread pool opensearch_ml_predict is used for handling the calls to SageMaker and it seems like it’s being sized based on the number of CPUs the A…

Hi @rs-search , we could deploy sparse model via sagemaker and remote inference. See here to use remote model. But we couldn’t register model via url. I think Neural sparse button is already available in AOS integration. [image]

Hi @grunt-solaces.0h Unfortunately, improving thread number doesn’t have a influence on inference speed. Please check sagemaker endpoint CPU utilization to see whether it is fully used. Actually for a better ingestion throughput, we prefer to use GPU instance like g4dn, g5 or p3

Hi @xinyual , Thanks for the answer! We were wondering about the thread number for the following reasons: We switched to a larger machine, with more CPUs and as a result it seems the ingestion pipeline is sending more documents for inference, in parallel At first we used a CPU instance for infere…

Hi @grunt-solaces.0h , the predict thread pool is originally designed for local model CPU deployment case. For remote case, there’s bottleneck on httpclient, e.g. if you only have one instance sending requests to remote model, the remote model instance could be underutilized. We have identified anot…

Thanks @zane_neo , this makes sense! Will MAX_CONNECTION be configurable via PUT _cluster/settings? We can’t set the node.processors value as we use OpenSearch deployed in Amazon and we don’t have access to that config so we’ll have to wait for the httpclient fix.

In fact, our current http client implementation is not optimal, I’ve created this issue: [FEATURE] Replace blocking httpclient with async httpclient in remote inference · Issue #1839 · opensearch-project/ml-commons · GitHub to track this. BTW, if you can scale more instances(no need high performance…

How to register sparse encoding model in AWS OpenSearch

Plugins Machine Learning

darvel February 16, 2024, 9:37am 15

How deployed the model via SageMaker, please guide me?

[RFC] neural sparse models improvement plan

Topic		Replies	Views
The new sparse encoding model is not deployable Machine Learning	3	146	August 31, 2024
Model weights for sparse encoders Machine Learning	19	709	December 11, 2023
How deployed the model via AWS SageMaker to AWS Opensearch? Machine Learning	1	591	February 20, 2024
What does a model have to return from sagemaker for opensearch to be able to use it? Machine Learning	12	864	May 13, 2024
How to scale neural sparse ingestion pipeline OpenSearch	3	377	May 7, 2024

How to register sparse encoding model in AWS OpenSearch

Related topics