Unable to use securityadmin.sh script - CA certificate not a CA certificate error

thozook · December 20, 2023, 9:05pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OS: Ubuntu 20.04.6 LTS
Opensearch: 1.3.14
Java version: openjdk 11.0.21 2023-10-17

Describe the issue:
When attempting to use the securityadmin.sh script, I get a "TrustAnchor with subject is not a CA certificate error. CA certificate is specified in the command line invocation(along with the admin cert and key). This is the same CA Certificate that is used in the Opensearch configuration.

I followed Generating self-signed certificates - OpenSearch documentation to generate the self-signed CA and certificates.

#/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd /usr/share/opensearch/plugins/opensearch-security/securityconfig -cl -nhnv -cacert /etc/opensearch/certs/root-ca.pem -cert /etc/opensearch/certs/admin.pem -key /etc/opensearch/certs/admin-key.pem
WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use /usr/bin/java
Security Admin v7
Will connect to localhost:9300 ... done
20:55:56.387 [opensearch[_client_][transport_worker][T#1]] ERROR org.opensearch.security.ssl.transport.SecuritySSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: sun.security.validator.ValidatorException: TrustAnchor with subject "##Redacted Subject##" is not a CA certificate
javax.net.ssl.SSLHandshakeException: PKIX path validation failed: sun.security.validator.ValidatorException: TrustAnchor with subject "##Redacted Subject##" is not a CA certificate
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:360) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:298) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1076) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1063) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1010) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1651) ~[netty-handler-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1497) ~[netty-handler-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[netty-handler-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: sun.security.validator.ValidatorException: TrustAnchor with subject "##Redacted Subject##" is not a CA certificate
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
        ... 30 more
Caused by: sun.security.validator.ValidatorException: TrustAnchor with subject "CN=root.dns.a-record, OU=WSS, O=UHAUL, L=PHOENIX, ST=ARIZONA, C=US" is not a CA certificate
        at sun.security.validator.PKIXValidator.verifyTrustAnchor(PKIXValidator.java:393) ~[?:?]
        at sun.security.validator.PKIXValidator.toArray(PKIXValidator.java:333) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:366) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
        ... 30 more
ERR: Cannot connect to OpenSearch. Please refer to opensearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{TrgH3aItQtWfaL57I56kwA}{localhost}{127.0.0.1:9300}]]
        at org.opensearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:381)
        at org.opensearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:272)
        at org.opensearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:79)
        at org.opensearch.client.transport.TransportClient.doExecute(TransportClient.java:484)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:419)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:526)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)

Mantas · December 21, 2023, 11:27am

Hi @thozook,

Could you please run ls -l /etc/opensearch/certs and share the output?

Thanks,
Mantas

thozook · December 21, 2023, 2:47pm

Hello @Mantas,
Here’s the output:

#ls -l /etc/opensearch/certs/
total 20
-rw------- 1 opensearch opensearch 1704 Dec 19 18:21 admin-key.pem
-rw------- 1 opensearch opensearch 1224 Dec 15 16:58 admin.pem
-rw------- 1 opensearch opensearch 1375 Dec 15 16:58 root-ca.pem
-rw------- 1 opensearch opensearch 1704 Dec 15 16:58 wssdbsospls0001-key.pem
-rw------- 1 opensearch opensearch 1322 Dec 15 16:58 wssdbsospls0001.pem

Mantas · December 21, 2023, 5:16pm

Could you confirm a few things:

openssl verify -verbose -CAfile /etc/opensearch/certs/root-ca.pem /etc/opensearch/certs/admin.pem : the output of the command;
openssl x509 -text -noout -in /etc/opensearch/certs/root-ca.pem - could you please confirm if the Issuer:*** and the Subject:*** is matching, as well as if X509v3 Basic Constraints: critical CA: TRUE or FALSE

Thanks,
Mantas

thozook · December 21, 2023, 5:38pm

Output of the first openssl command:

#openssl verify -verbose -CAfile /etc/opensearch/certs/root-ca.pem /etc/opensearch/certs/admin.pem
C = US, ST = ARIZONA, L = PHOENIX, O = UHAUL, OU = WSS, CN = OpensearchAdmin
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/opensearch/certs/admin.pem: verification failed

For the second command, the issuer and subject do match but there are not any lines containing X509v3 Basic Constraints in the output. That did prompt me to go look at the default openssl.cnf file used by the machine that the certs and ca were generated on and every instance of “basicConstraints=CA:” was set to False.

I’m guessing that in order to be recognized as a CA certificate, that value needs to be set to true?

Mantas · December 22, 2023, 2:09pm

That’s correct.

Best,
mj

Topic		Replies	Views
Securityadmin.sh not working due to admin certificate issue Security	4	44	November 5, 2024
Securityadmin.sh unable to find valid certification path to requested target error Security configure , security-issue	2	737	November 1, 2023
Self-signed certificat : securityadmin.sh returns an error OpenSearch troubleshoot	3	238	March 5, 2024
Error while executing securityadmin.sh in OS 1.2.4 Security security-issue	19	801	October 26, 2023
Error securityadmin.sh execution Security	2	28	July 23, 2024

Unable to use securityadmin.sh script - CA certificate not a CA certificate error

Related topics