Opensearch cant complete setup; Invalid CertificateVerify signature

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Opensearch 2.11

Describe the issue:

I have configured Opensearch with TLS. I have generate certificates for the admin, my two nodes, and the signing authority. The signing authority is an Active Directory Intermediate CA (Enterprise). Unfortunately, everytime I run

sudo OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk ./securityadmin.sh -h node-1 -cacert /etc/opensearch/EntCA.pem -cert /etc/opensearch/admin.pem -key /etc/opensearch/admin-key.pem -cd /usr/share/opensearch/config/opensearch-security/

I get:

Security Admin v7
Will connect to graynode-1:9200 ... done
ERR: An unexpected SSLHandshakeException occured: Invalid CertificateVerify signature
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: Invalid CertificateVerify signature
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
	at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:948)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:333)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:321)
	at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:573)
	at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:163)
Caused by: javax.net.ssl.SSLHandshakeException: Invalid CertificateVerify signature
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:365)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:312)
	at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:1012)
	at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyConsumer.consume(CertificateVerify.java:1163)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
	at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:289)
	at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:357)
	at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:545)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:833)

Thus preventing me from completing certification.

Configuration:

My openearch.yml:

cluster.name: cluster
node.name: node-1
node.roles: [ cluster_manager, data ]
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 10.100.0.228
http.port: 9200

discovery.seed_hosts: ["node-0.foo.bar", "node-1.foo.bar"]
cluster.initial_cluster_manager_nodes: ["node-0.foo.bar", "node-1.foo.bar"]

plugins.security.ssl.transport.pemcert_filepath: node-0.pem
plugins.security.ssl.transport.pemkey_filepath: node-0-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: EntCA.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node-0.pem
plugins.security.ssl.http.pemkey_filepath: node-0-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: EntCA.pem
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=admin,OU=sumthin,O=sumthin,L=Aplace,ST=Astate,C=US
plugins.security.nodes_dn:
  - CN=node-0.foo.bar,OU=sumthin,O=sumthin,L=Aplace,ST=Astate,C=US
  - CN=node-1.foo.bar,OU=sumthin,O=sumthin,L=Aplace,ST=Astate,C=US
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-config", ".plugins-ml-connector", ".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models"]
node.max_local_storage_nodes: 3

For what it is worth:

 sudo openssl verify -verbose -CAfile /etc/opensearch/EntCA.pem /etc/opensearch/admin.pem
/etc/opensearch/admin.pem: OK

Any help is greatly appreciated. Thank you!

2

Hi @04_996_C2,

How did you deploy your OpenSearch cluster?

According to the documentation " Path to the root CAs (PEM format), which must be under the config directory…" Configuring TLS certificates - OpenSearch Documentation

Could you try placing your CA under the config directory make sure the correct permissions are assigned and test it again?

Best,
mj

1 Like
3

I don’t even have a config directory (plus I’ve configured multiple multi-node opensearch clusters before in the exact way I am doing it now).

I may be barking up the wrong tree, here, but the root CA (EntCA) utilizes a 4096 pub key whereas the nodes and admin use a 2048 pub key. Think thats the issue? Every other cluster I’ve set up utilizes a root CA with a 2048 key.

4

Hi @04_996_C2 ,

I have generated a self-signed CA cert with Keysize 4096 bit and an admin cert with Keysize 2048 in my lab and had no issues executing ./securityadmin.sh.

Certs have been generated with OpenSSL as per Generating self-signed certificates - OpenSearch Documentation .

I am not greatly familiar with Active Directory signed certificates, but you could generate certificates using the instructions above and compare your certificates to help you to troubleshoot it, or you can share your cert details, and I can check it for you (please make sure to mask/remove/exclude ANY sensitive details).

best,
mj

1 Like
5

I really appreciate your assistance. I do have it working, now. In the end I just operated as if the AD certification process was the problem. So, I copied the AD Intermediate CA to a debian machine and create certificates via openssl using the Intermediate CA to sign. That did the trick!

Thank you!

1 Like
6

That’s good to know, thanks for sharing your solution!

Best,
mj