@pablo , I found the root cause, I login into the container, try this, SSL failed, the container seems does not have this CA for artifacts.opensearch.org(CNAME to d2czzf4vq4ccyb.cloudfront.net)
curl -l https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/msmarco-distilbert-base-tas-b/1.0.1/onnx/sentence-transformers_msmarco-distilbert-base-tas-b-1.0.1-onnx.zip
curl: (60) SSL certificate problem: unable to get local issuer certificate
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
I use the CNAME target d2czzf4vq4ccyb.cloudfront.net, SSL works:
curl -l https://d2czzf4vq4ccyb.cloudfront.net/models/ml-models/huggingface/sentence-transformers/msmarco-distilbert-base-tas-b/1.0.1/onnx/sentence-transformers_msmarco-distilbert-base-tas-b-1.0.1-onnx.zip
Warning: Binary output can mess up your terminal. Use ââoutput -â to tell
Warning: curl to output it to your terminal anyway, or consider "âoutput
Warning: " to save to a file.
I change the url to use d2czzf4vq4ccyb.cloudfront.net and it works now when register the model. So this is because the container does not have the CA certificate for artifacts.opensearch.org, But I donât understand why you works since we use same docker compose file, the docker should be same
@pablo I found the root cause, I change the domain to the CNAME target d2czzf4vq4ccyb.cloudfront.net and it worked, So this is because the container does not have the CA root certificate for this domain artifacts.opensearch.org.
But I do know why you could work since we are using same docker compose file and the container should be same
I encounter another issue when deploy the model
âerrorâ: âââ{â4iTSMnXUTEedGfYGgdItewâ:âno onnxruntime in java.library.path: :/usr/share/opensearch/plugins/opensearch-knn/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/libâ,â_Bxw_WjwTBSh3jZ5M4JIQw":âno onnxruntime in java.library.path: :/usr/share/opensearch/plugins/opensearch-knn/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/libâ}ââ",
I think opensearch doc/support is not good, disappoint
@smith What is your Docker host OS? Iâm using Ubuntu 20.04 and 22.04
What user did you use to deploy the model?
@pablo ďźI am using mac OS and I am using a personal user to deploy this. I tried Ubuntu it works(also use a persoanl user to run the docker). This is docker, why it depends on the Host OS?
@smith I think the docker service might use OSâs certificate keystore to validate CAs in SSL connections.
@smith Did you get any solution for it?
It is working in my personal laptop, but giving the same issue in office laptop.
I have a similar problem, in my case I have opensearch with Certificate Authentication enabled. I can connect to opensearch with curl command to register the ml:
$ curl -k --cert {cert} --key {cert key} -X POST âhttps://localhost:9200/_plugins/_ml/models/_register?deploy=trueâ -H âContent-Type: application/jsonâ -d â{ânameâ:âamazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v2-distillâ,âversionâ:â1.0.0â,âmodel_formatâ:âTORCH_SCRIPTâ}â
Getting response:
{âtask_idâ:âgko0_ZEBO2TvFnwiBb5xâ,âstatusâ:âCREATEDâ}
Also I am able to connect with OpenSearch Dashboard and do the same. In both cases, I got the same error described above.
Does OpenSearch ML support Certificate Authentication? Is there a specific configuration for ML to use it with Certificate Security?
@gilberto The initial issue is not related to certificate authentication.
During the ML registration, OpenSearch couldnât verify the CA of the certificate.
OpenSearch was setting up a connection as a client against an HTTPS endpoint without authentication headers.
Does your office have any firewall?
I think the connection to download ONNX(or PyTorch) files will be blocked.