Error while executing securityadmin.sh in OS 1.2.4

[manoj]

Versions: OpenSearch 1.2.4

Configuration:

• 5 Nodes with Security Plugin Installed on each running in Kubernetes
• Internode and Rest TLS are enabled

Describe the issue:

Hi,
I’m trying to execute securityadmin.sh to update the security configurations.
But, it throwing NoNodeAvailableException[None of the configured nodes are available error.

Command used:

~/opensearch-security/tools/securityadmin.sh \
  -cd ~/opensearch-security/securityconfig/ -icl -nhnv \
  -cacert ~/ca.crt \
  -cert ~/tls.crt \
  -key ~/tls.key

Here, ca.crt is root CA, tls.crt is admin certificate and tls.key is it’s private key.
CN of admin certificate is added to plugins.security.authcz.admin_dn.

Error

12:37:59.290 [opensearch[_client_][transport_worker][T#1]] ERROR org.opensearch.security.ssl.transport.SecuritySSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:360) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:298) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1076) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1063) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1010) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1547) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1393) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.72.Final.jar:4.1.72.Final]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
        ... 30 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146) ~[?:?]
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127) ~[?:?]
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
        ... 30 more
ERR: Cannot connect to OpenSearch. Please refer to opensearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{F6KjGJwMRg6bQh343lCIuQ}{localhost}{127.0.0.1:9300}]]
        at org.opensearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:381)
        at org.opensearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:272)
        at org.opensearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:79)
        at org.opensearch.client.transport.TransportClient.doExecute(TransportClient.java:484)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:419)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:524)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:157)

[Mantas]

Hi manoj,

Try providing a full path to your certs:

-key "/usr/share/opensearch/config/tls.key" 
-cert "/usr/share/opensearch/config/tls.crt" 
-cacert "/usr/share/opensearch/config/ca.crt"

You could, as well, add -h [hostname] to specify your target host.

Let me know if that works.

Best,
Mantas

[manoj]

Tried using full path along with -h [hostname]. Still the same.

This is the opensearch.yml in all 5 nodes (2 data, 3 master)

opensearch.yml

path:
  data: "${OPENSEARCH_HOME}/data"
  logs: "${OPENSEARCH_HOME}/logs"
  repo: ["${OPENSEARCH_HOME}/repository"]

bootstra.memory_lock: false

compatibility.override_main_response_version: true

discovery.seed_hosts:
    - "opensearch-discovery"

network.tcp.keep_idle: 300

cluster:
  initial_master_nodes:
    - "opensearch-master-0"
    - "opensearch-master-1"
    - "opensearch-master-2"

http:
  compression: true
  host: 0.0.0.0

transport.host: 0.0.0.0

plugins.security.ssl.transport.pemcert_filepath: "/usr/os-certs/transport-cert/tls.crt"
plugins.security.ssl.transport.pemkey_filepath: "/usr/os-certs/transport-cert/tls.key"
plugins.security.ssl.transport.pemtrustedcas_filepath: "/usr/os-certs/transport-ca/ca.crt"
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: "/usr/os-certs/http-cert/tls.crt"
plugins.security.ssl.http.pemkey_filepath: "/usr/os-certs/http-cert/tls.key"
plugins.security.ssl.http.pemtrustedcas_filepath: "/usr/os-certs/http-client-ca/ca.crt"

plugins.security.ssl.http.clientauth_mode: OPTIONAL

plugins.security.allow_default_init_securityindex: true

plugins.security.authcz.admin_dn:
  - 'CN=opensearch-internal-client'

plugins.security.nodes_dn:
  - 'CN=opensearch-discovery'

plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

Command used:

/opt/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
  -cd /opt/opensearch/plugins/opensearch-security/securityconfig/ -icl -nhnv \
  -cacert /usr/os-certs/http-server-ca/ca.crt \
  -cert /usr/os-certs/http-client-cert/tls.crt \
  -key /usr/os-certs/http-client-cert/tls.key \
  -h localhost

[Mantas]

Please check the below:

TLS certificates

“The certificates must be under the config directory or its sub-directory”

[manoj]

I’ve moved the certificates under config directory. Still the issue occurs.

NOTE: Here, I have 3 CAs.
http-server-ca - Signs HTTP Server Certificate
http-client-ca - Signs HTTP Client Certificates (including Admin Certificate)
transport-ca - Signs Transport Certificate

So, in securityadmin.sh, I used admin-client-certificate with http-server-ca.

This same setup is working fine with OS 2.7.0. But it’s failing for OS 1.2.4.

While executing securityadmin.sh in OS 1.2.4, it’s trying to connect with localhost:9300 (I’m not sure how admin-client-certificate with http-server-ca will work here)
But in OS 2.7.0, it’s connecting with localhost:9200.

[Mantas]

Are you seeing the same error after changes?

@manoj

unable to find valid certification path to requested target

Could you please share your updated opensearch.yml file?

[manoj]

Here is the updated opensearch.yml.

opensearch.yml

path:
  data: "${OPENSEARCH_HOME}/data"
  logs: "${OPENSEARCH_HOME}/logs"
  repo: ["${OPENSEARCH_HOME}/repository"]

bootstra.memory_lock: false

compatibility.override_main_response_version: true

discovery.seed_hosts:
    - "opensearch-discovery"

network.tcp.keep_idle: 300

cluster:
  initial_master_nodes:
    - "opensearch-master-0"
    - "opensearch-master-1"
    - "opensearch-master-2"

http:
  compression: true
  host: 0.0.0.0

transport.host: 0.0.0.0

plugins.security.ssl.transport.pemcert_filepath: "/usr/share/opensearch/config/transport-cert/tls.crt"
plugins.security.ssl.transport.pemkey_filepath: "/usr/share/opensearch/config/transport-cert/tls.key"
plugins.security.ssl.transport.pemtrustedcas_filepath: "/usr/share/opensearch/config/transport-ca/ca.crt"
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: "/usr/share/opensearch/config/http-cert/tls.crt"
plugins.security.ssl.http.pemkey_filepath: "/usr/share/opensearch/config/http-cert/tls.key"
plugins.security.ssl.http.pemtrustedcas_filepath: "/usr/share/opensearch/config/http-client-ca/ca.crt"

plugins.security.ssl.http.clientauth_mode: OPTIONAL

plugins.security.allow_default_init_securityindex: true

plugins.security.authcz.admin_dn:
  - 'CN=opensearch-internal-client'

plugins.security.nodes_dn:
  - 'CN=opensearch-discovery'

plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

Command used:

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
  -cd /usr/share/opensearch/plugins/opensearch-security/securityconfig/ \
  -icl -nhnv \
  -cacert /usr/share/opensearch/config/http-server-ca/ca.crt \
  -cert /usr/share/opensearch/config/http-client-cert/tls.crt \
  -key /usr/share/opensearch/config/http-client-cert/tls.key \
  -h localhost

This same setup is working fine with OS 2.7.0

[Mantas]

can you test your command with -p 9200 flag added?

@manoj

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
  -cd /usr/share/opensearch/plugins/opensearch-security/securityconfig/ \
  -icl \
  -p 9200  \
  -h localhost \
  -cacert /usr/share/opensearch/config/http-server-ca/ca.crt \
  -cert /usr/share/opensearch/config/http-client-cert/tls.crt \
  -key /usr/share/opensearch/config/http-client-cert/tls.key \
  -nhnv 

[manoj]

Error:

WARNING: JAVA_HOME not set, will use /usr/bin/java
Security Admin v7
WARNING: Seems you want connect to the OpenSearch HTTP port.
         securityadmin connects on the transport port which is normally 9300.
Will connect to localhost:9200 ... done
ERR: Cannot connect to OpenSearch. Please refer to opensearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{GNI4ShfcQyextGgGAZ1XEg}{localhost}{127.0.0.1:9200}]]
        at org.opensearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:381)
        at org.opensearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:272)
        at org.opensearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:79)
        at org.opensearch.client.transport.TransportClient.doExecute(TransportClient.java:484)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:419)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:524)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:157)

There is no log in the console.

[Mantas]

Could you please share the error when you run with -p 9300 ?

[manoj]

Error with 9300:

Security Admin v7
Will connect to localhost:9300 ... done
15:49:02.846 [opensearch[_client_][transport_worker][T#1]] ERROR org.opensearch.security.ssl.transport.SecuritySSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:1.8.0_382]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:331) ~[?:1.8.0_382]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:274) ~[?:1.8.0_382]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:269) ~[?:1.8.0_382]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1356) ~[?:1.8.0_382]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1231) ~[?:1.8.0_382]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1174) ~[?:1.8.0_382]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) ~[?:1.8.0_382]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:1.8.0_382]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:981) ~[?:1.8.0_382]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:968) ~[?:1.8.0_382]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_382]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:915) ~[?:1.8.0_382]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1547) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1393) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.72.Final.jar:4.1.72.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.72.Final.jar:4.1.72.Final]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_382]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456) ~[?:1.8.0_382]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323) ~[?:1.8.0_382]
        at sun.security.validator.Validator.validate(Validator.java:271) ~[?:1.8.0_382]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315) ~[?:1.8.0_382]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:278) ~[?:1.8.0_382]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) ~[?:1.8.0_382]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1334) ~[?:1.8.0_382]
        ... 30 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146) ~[?:1.8.0_382]
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127) ~[?:1.8.0_382]
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_382]
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451) ~[?:1.8.0_382]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323) ~[?:1.8.0_382]
        at sun.security.validator.Validator.validate(Validator.java:271) ~[?:1.8.0_382]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315) ~[?:1.8.0_382]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:278) ~[?:1.8.0_382]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) ~[?:1.8.0_382]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1334) ~[?:1.8.0_382]
        ... 30 more
ERR: Cannot connect to OpenSearch. Please refer to opensearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{_oGbgaOyQtmCeQVHWW16kg}{localhost}{127.0.0.1:9300}]]
        at org.opensearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:381)
        at org.opensearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:272)
        at org.opensearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:79)
        at org.opensearch.client.transport.TransportClient.doExecute(TransportClient.java:484)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:419)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:524)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:157)

[Mantas]

Can you run the below and share the output?

ls -l /usr/share/opensearch/config/http-server-ca/
ls -l /usr/share/opensearch/config/http-client-cert/

[manoj]

$ ls -l /usr/share/opensearch/config/http-server-ca
total 4
-rw-r--r-- 1 manoj manoj 1403 Oct 25 15:56 ca.crt

$ ls -l /usr/share/opensearch/config/http-client-cert
total 8
-rw-r--r-- 1 manoj manoj 1387 Oct 25 15:56 tls.crt
-rw------- 1 manoj manoj 1708 Oct 25 15:57 tls.key

[Mantas]

please change owner to opensearch:opensearch see below (you will need root permissions):

chown opensearch:opensearch /usr/share/opensearch/config/http-server-ca/ca.crt
chown opensearch:opensearch /usr/share/opensearch/config/http-client-cert/tls.crt
chown opensearch:opensearch /usr/share/opensearch/config/http-client-cert/tls.key

[manoj]

Changed the owner and group.

$ ls -l /usr/share/opensearch/config/http-server-ca
total 4
-rw-r--r-- 1 opensearch opensearch 1403 Oct 25 16:00 ca.crt

$ ls -l /usr/share/opensearch/config/http-client-cert
total 8
-rw-r--r-- 1 opensearch opensearch 1387 Oct 25 16:01 tls.crt
-rw------- 1 opensearch opensearch 1708 Oct 25 16:01 tls.key

OpenSearch is also running as user opensearch

$ ps -ef | grep opensearch
opensearch 16 2 1 16:03 pts/2    00:00:37 /usr/lib/jvm/jdk-11.0.19/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/tmp/opensearch-4814320699987820682 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data...

Still same error.

[Mantas]

could you please run openssl x509 -in <*yourCert*> -noout -subject for your certs and share the output.

Please make sure to hide any sensitive information.

Thanks,
Mantas

[manoj]

Here is the list of certificates

$ openssl x509 -in usr/share/opensearch/config/http-server-ca/ca.crt -noout -subject
CN = opensearch-server-ca

# Signed by opensearch-client-ca
$ openssl x509 -in usr/share/opensearch/config/http-client-cert/tls.crt -noout -subject
CN = opensearch-internal-client

$ openssl x509 -in usr/share/opensearch/config/http-client-ca/ca.crt -noout -subject
CN = opensearch-client-ca

# Signed by opensearch-server-ca
$ openssl x509 -in usr/share/opensearch/config/http-cert/tls.crt -noout -subject
CN = opensearch-server

$ openssl x509 -in usr/share/opensearch/config/transport-ca/ca.crt -noout -subject
CN = opensearch-transport-ca

# Signed by opensearch-transport-ca
$ openssl x509 -in usr/share/opensearch/config/transport-cert/tls.crt -noout -subject
CN = opensearch-discovery

[Mantas]

As 1.2.4 is using TransportClient auth/auth you will have to concatenate your root ca(s) ( …/transport-ca/ca.crt and …/http-client-ca/ca.crt) and set it as:

opensersh.ym
plugins.security.ssl.transport.pemtrustedcas_filepath:

command
-cacert

I did some digging and found out that it works in 2.x.x because the TransportClient auth/auth was removed and the client authentication is available only at HTTP endpoint 9200, for more details please refer to the following: Remove TransportClient auth/auth by jochenkressin · Pull Request #1701 · opensearch-project/security · GitHub

[manoj]

Hi Mantas,

Yes. You’re correct.
After concatenating http-client-ca and transport-ca and using it in plugins.security.ssl.transport.pemtrustedcas_filepath, securityadmin.sh is working as expected.

Tried another way around.
Since this script is using TransportClient auth/auth, I’ve created another admin client certificate signed by transport-ca instead of http-client-ca, added CN to plugins.security.authcz.admin_dn and used this new client certificate while executing the script.

Thanks for your help and support.

[Mantas]

Glad to hear it solved it!