Self-signed certificat : securityadmin.sh returns an error

Raphy · March 1, 2024, 1:12pm

Hello

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch

Describe the issue:
I wanted to generate self-signed certificates for OpenSearch on a node of my OpenSearch server. I have 5 nodes in cluster : node1.rct.local, node2.rct.local to node5.rct.local.

So, I followed this link: https://opensearch.org/docs/latest/security/configuration/generate-certificates/

I generated everything using a script provided following the link :

# Root CA
openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/C=FR/O=rct.local/OU=Ops/DC=rct.local/CN=ROOT" -out root-ca.pem -days 1825
# Admin cert
openssl genrsa -out admin-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
openssl req -new -key admin-key.pem -subj "/C=FR/O=rct.local/OU=Ops/DC=rct.local/CN=ADMIN" -out admin.csr
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 1825
# Node cert 1
openssl genrsa -out node1-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in node1-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node1-key.pem
openssl req -new -key node1-key.pem -subj "/C=FR/O=rct.local/OU=Ops/DC=rct.local/CN=node1.rct.local" -out node1.csr
echo 'subjectAltName=DNS:node1.rct.local' > node1.ext
# Node cert2 to5 - same config but changed node1 to node2..5
# Client cert
openssl genrsa -out client-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in client-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out client-key.pem
openssl req -new -key client-key.pem -subj "/C=FR/O=rct.local/OU=Ops/DC=rct.local/CN=dashboard.rct.local" -out client.csr
echo 'subjectAltName=DNS:dashboard.rct.local' > client.ext
openssl x509 -req -in client.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out client.pem -days 1825

Then, I modified opensearch.yml (Configuration below) to add the DNs.

But when I try to run securityadmin.sh:

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh  -h node1.rct.local --diagnose --accept-red-cluster -cn  elk-cluster  \
-cacert /usr/share/opensearch/config/cert/root-ca.pem \
-cert /usr/share/opensearch/config/cert/admin.pem  \
-key /usr/share/opensearch/config/cert/admin-key.pem \
-cd /usr/share/opensearch/config/opensearch-security/

I get:

Security Admin v7
Will connect to node1.rct.local:9200 ... done
ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
        at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:948)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:333)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:573)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:163)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:289)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:357)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:545)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
        ... 19 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 24 more

Configuration:

openssl x509 -subject -nameopt RFC2253 -noout -in cert/node1.pem
subject=CN=node1.rct.local,DC=rct.local,OU=Ops,O=rct.local,C=FR

Opensearch.yml

plugins.security.ssl.transport.pemcert_filepath: cert/node1.pem
plugins.security.ssl.transport.pemkey_filepath: cert/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: cert/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: cert/node1.pem
plugins.security.ssl.http.pemkey_filepath: cert/node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: cert/root-ca.pem

plugins.security.nodes_dn:
- CN=node1.rct.local,DC=rct.local,OU=Ops,O=rct.local,C=FR
- CN=node2.rct.local,DC=rct.local,OU=Ops,O=rct.local,C=FR
- CN=node3.rct.local,DC=rct.local,OU=Ops,O=rct.local,C=FR
- CN=node4.rct.local,DC=rct.local,OU=Ops,O=rct.local,C=FR
- CN=node5.rct.local,DC=rct.local,OU=Ops,O=rct.local,C=FR
plugins.security.authcz.admin_dn:
- CN=ADMIN,DC=rct.local,OU=Ops,O=rct.local,C=FR
## END opensearch Security Node & Admin certificates configuration ##

Relevant Logs or Screenshots:

pablo · March 1, 2024, 2:09pm

@Raphy This might be caused by different root-ca.pem certificates in the plugins.security.ssl.http.pemtrustedcas_filepath and -cacert /usr/share/opensearch/config/cert/root-ca.pem \

Did you replace admin certificates in each OpenSearch node and retarded the service?

Raphy · March 1, 2024, 2:55pm

No it is the same certificate in plugins.security.ssl.http.pemtrustedcas_filepath : cert/root-ca.pem and in the command line : -cacert /usr/share/opensearch/config/cert/root-ca.pem

but I didn’t replace the admin certificate in each Opensearch node neither restarted the service

You mean I have to

replace the admin certificate in each Opensearch node (in /usr/share/opensearch/config or /usr/share/opensearch/config/cert because there is an existing admin certicat in /usr/share/opensearch/config)
restart it Opensearch
then execute securityadmin.sh

?

What I tried is on localhost where the admin certificate is properly present in cert/admin.pem
Thanks

pablo · March 5, 2024, 10:26am

@Raphy Sorry, no need to restart if you just replacing the admin certificate.
It is not mandatory to replace it. It depends on where you’re running your securityadmin.sh script. It is not mandatory to execute it from the node.

However, in your first post, you shared the commands for generating root-ca.pem and signing admin and node certificates with that cert.

Did you use all these certs in the OpenSearch nodes? If yes, then for node and rootCA certificates it is mandatory to restart the OpenSearch service.
OpenSearch service will load the certificates only at the startup. Replacement of certificates in the filesystem doesn’t change anything in the service.

Try running the below command against the OpenSearch service and port 9200 and compare the generated certificate with the one in the output.

openssl s_client -connect <OpenSearch_node>:9200

Topic		Replies	Views
"Seems you use a node certificate which is also an admin certificate" - Error when running security script Security	1	446	March 6, 2023
Securityadmin errors with certificates Security security-issue	13	1338	May 15, 2023
Unable to use securityadmin.sh script - CA certificate not a CA certificate error Security	5	277	December 22, 2023
Admin Certificate Generation in K8s Security configure	2	530	April 7, 2023
Self-signed admin certificate Security	1	305	August 5, 2022

Self-signed certificat : securityadmin.sh returns an error

Related Topics