Error communicating with remote model - SSL handshake exception

[pablo]

This issue was created in the OpenSearch Security Slack channel.

Slack

Hi, I am trying to connect to a privately hosted llm from OpenSearch by following the documentation Connecting to externally hosted models - OpenSearch Documentation . But getting SSLHandshakeException .
“reason”:“Error communicating with remote model: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”}.
How can I fix this, where I have to add the CA certs.

Received error from remote service: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500) ~[?:?]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[?:?]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[?:?]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
    at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:647) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:467) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:363) ~[?:?]
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1695) ~[?:?]
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1541) ~[?:?]
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377) ~[?:?]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428) ~[?:?]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
    ... 16 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388) ~[?:?]
    at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271) ~[?:?]
    at java.base/sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284) ~[?:?]
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:625) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:467) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:363) ~[?:?]
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1695) ~[?:?]
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1541) ~[?:?]
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377) ~[?:?]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428) ~[?:?]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
    ... 16 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148) ~[?:?]
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129) ~[?:?]
    at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
    at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383) ~[?:?]
    at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271) ~[?:?]
    at java.base/sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284) ~[?:?]
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:625) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:467) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:363) ~[?:?]
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1695) ~[?:?]
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1541) ~[?:?]
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377) ~[?:?]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428) ~[?:?]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?] (edited) 
10:46
[2025-10-24T14:37:56,279][WARN ][r.suppressed       ] [integTest-0] path: /_plugins/_ml/models/i6t3FZoBZtYIOI3JY-C7/_predict, params: {model_id=i6t3FZoBZtYIOI3JY-C7}
org.opensearch.OpenSearchStatusException: Error communicating with remote model: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at org.opensearch.ml.engine.algorithms.remote.MLSdkAsyncHttpResponseHandler.onError(MLSdkAsyncHttpResponseHandler.java:106) ~[?:?]
    at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.lambda$exceptionCaught$2(ResponseHandler.java:212) ~[?:?]
    at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:246) ~[?:?]
    at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.exceptionCaught(ResponseHandler.java:211) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[?:?]
    at io.netty.handler.logging.LoggingHandler.exceptionCaught(LoggingHandler.java:214) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[?:?]
    at software.amazon.awssdk.http.nio.netty.internal.UnusedChannelExceptionHandler.exceptionCaught(UnusedChannelExceptionHandler.java:52) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[?:?]
    at software.amazon.awssdk.http.nio.netty.internal.FutureCancelHandler.exceptionCaught(FutureCancelHandler.java:44) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[?:?]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424) ~[?:?]
    at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92) ~[?:?]
    at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145) ~[?:?]
    at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[?:?]
    at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[?:?]
    at io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:1221) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[?:?]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[?:?]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
    at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

[pablo]

Please provide your current cluster configuration GET _cluster/settings.

[muneer]

Output of GET _cluster/settings is

{
 "persistent" : {
  "plugins" : {
   "ml_commons" : {
    "memory_feature_enabled" : "true",
    "only_run_on_ml_node" : "false",
    "trusted_connector_endpoints_regex" : \[
     "^https://private-url\\.com/.\*$"
    \]
   }
  }
 },
 "transient" : { }
} 

I didn’t enable the security.

Looks like ssl verification is enabled by default in ML Commons and there is no way we can disable it.

[pablo]

@muneer I’ve done some testing, and I couldn’t get the Machine Learning plugin to work with a self-signed certificate.
I’ve concatenated the remote SSL certificate of my Ollama instance with root-ca.pem and tried using keystore instead.

I also couldn’t find any option in the plugin to turn off the certificate validation.
The workaround would be using the Let’s Encrypt certificate with your remote model.

Also you could consider raising a Feature Request in ml_commons GitHub repository.

opensearch-project/ml-commons

[muneer]

Yes I could make it work by a small change in MLCommons code. I created the enhancement request. [FEATURE] Add an option to turn on and off the certificate validation in ML Commons · Issue #4371 · opensearch-project/ml-commons · GitHub

I will work on this, thanks for testing and confirming the issue.

[muneer]

Still we have to identify how can we configure certificate of llm server in OpenSerach. Disabling certificate validation is a workaround.

[pablo]

@muneer The current machine learning plugin doesn’t have option to present a SSL certificate.

Also, as I mentioned before, it’s ignoring concatenation of the LLM certificate with the root-ca.pem (plugins.security.ssl.transport.pemtrustedcas_filepath) or using truststore (plugins.security.ssl.transport.truststore_filepath)

I think this will require another Feature Request.

[pablo]

@muneer This workaround was mentioned today by one of the users.

Add Custom CA for internal HTTPS endpoints

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.3.0 Describe the issue: I am trying to connect my local (docker) OpenSearch Cluster to an LLM that can only be accessed through our internal OpenRouter installation, that is secured by our internal CA. I am unable to provide the CA to OpenSearch for outgoing HTTPS connections. I end up having: Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException…

What CA is used by notifications channels?

SOLVED Solved the issue after a good deal of trial and error. The notification channels outgoing httpclient uses the Java Runtime trust store, and NOT the opensearch truststore for verifying connections. This is important, as it means you will have to trust your CA with Javas cacerts truststore as well. This can be done by: Access your container as root, in order to have sufficient privileges: docker exec -u root -it opensearch_container_name bash Read your existing truststore and veri…

I have followed the steps and successfully connected to my Ollama with HTTPS.
I’ve shared my steps in the other thread. I understand that this is a workaround, and the ML plugin should contain an option for self-signed certificates

Add Custom CA for internal HTTPS endpoints

@FalcoSuessgott I’ve tested solution provided in the shared link and I was able to connect to my remote Ollama with HTTPS and self-signed certificate. These were my steps. Copy cacerts keystore from the running OpenSearch Docker docker cp opensearch-node1_3.3.0:/usr/share/opensearch/jdk/lib/security/cacerts . Get certificate from Ollama instance openssl s_client -connect ollama.pablo.local:443 Save certificate into ollama.crt Add ollama.crt to the carets keystore keytool -import -nopro…

[muneer]

I was trying the same thing, not in docker environment, by directly running ML Commons code base locally and importing CA certificates to JDK keystore. But, it didn’t work. I will double check whether I am doing anything wrong here.

[pablo]

@muneer Did you use jdk that is deployed with OpenSearch?