Exception when reloading self signed certs via REST API

reneradoi · June 21, 2024, 9:29am

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.14.0

Describe the issue:
As part of the TLS cert renewal process I want to use the REST API for hot reload of transport and http certificates. When reloading the already used certificates, everything works fine (response code 200). But when I want to apply newly generated, self signed certificates, I always run into the same exception telling me that New Certs do not have valid Issuer DN, Subject DN or SAN.

I’ve checked my configuration and the validity of the certs, everything seems to be fine here. If the application is restarted, the new certs are also picked up without any issue. Only reloading via the API produces this error.

Configuration:
Relevant part of opensearch.yml:

plugins.security.authcz.admin_dn:
- O=opensearch-x7ga,CN=admin
plugins.security.ssl.transport.pemcert_filepath: certificates/unit-transport.cert
plugins.security.ssl.transport.pemkey_filepath: certificates/unit-transport.key
plugins.security.ssl.transport.pemtrustedcas_filepath: certificates/root-ca.cert
plugins.security.ssl.http.pemcert_filepath: certificates/unit-http.cert
plugins.security.ssl.http.pemkey_filepath: certificates/unit-http.key
plugins.security.ssl.http.pemtrustedcas_filepath: certificates/root-ca.cert
plugins.security.ssl.http.clientauth_mode: OPTIONAL
cluster.name: opensearch-x7ga
node.name: opensearch-30
node.roles:
- data
- ingest
- ml
- coordinating_only
- cluster_manager
discovery.seed_providers: file
plugins.security.disabled: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.restapi.roles_enabled:
- all_access
- security_rest_api_access
plugins.security.unsupported.restapi.allow_securityconfig_modification: true
plugins.security.ssl_cert_reload_enabled: true

Cert files:

ls -la certificates/
total 40
drwxrwx---  2 snap_daemon root 4096 Jun 21 09:01 .
drwxrwx--- 11 snap_daemon root 4096 Jun 21 09:01 ..
-rw-r--r--  1 snap_daemon root 1179 Jun 21 09:04 app-admin.cert
-rw-r--r--  1 snap_daemon root 1704 Jun 21 09:04 app-admin.key
-rw-r--r--  1 snap_daemon root 2343 Jun 21 09:04 chain.pem
-rw-r--r--  1 snap_daemon root 1163 Jun 21 09:01 root-ca.cert
-rw-r--r--  1 snap_daemon root 1349 Jun 21 09:04 unit-http.cert
-rw-r--r--  1 snap_daemon root 1704 Jun 21 09:04 unit-http.key
-rw-r--r--  1 snap_daemon root 1268 Jun 21 09:04 unit-transport.cert
-rw-r--r--  1 snap_daemon root 1704 Jun 21 09:04 unit-transport.key

Subject Alternative Name in the certs include Registered ID:1.2.3.4.5.5 to make use of OIDs.

Relevant Logs or Screenshots:
API response:

{"error":{"root_cause":[{"type":"i_o_exception","reason":"OpenSearchSecurityException[Error while initializing http SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.]; nested: Exception[New Certs do not have valid Issuer DN, Subject DN or SAN.];"}],"type":"i_o_exception","reason":"OpenSearchSecurityException[Error while initializing http SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.]; nested: Exception[New Certs do not have valid Issuer DN, Subject DN or SAN.];","caused_by":{"type":"security_exception","reason":"Error while initializing http SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.","caused_by":{"type":"exception","reason":"New Certs do not have valid Issuer DN, Subject DN or SAN."}}},"status":500}

Server logfile:

[2024-06-21T09:04:26,877][WARN ][r.suppressed             ] [opensearch-30] path: /_plugins/_security/api/ssl/transport/reloadcerts, params: {certType=transport}
java.io.IOException: OpenSearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.]; nested: Exception[New Certs do not have valid Issuer DN, Subject DN or SAN.];
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.reloadCertificates(SecuritySSLCertsApiAction.java:203) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.lambda$securitySSLCertsRequestHandlers$3(SecuritySSLCertsApiAction.java:117) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.validation.ValidationResult.valid(ValidationResult.java:87) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.lambda$securitySSLCertsRequestHandlers$5(SecuritySSLCertsApiAction.java:105) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.RequestHandler$RequestHandlersBuilder.lambda$add$2(RequestHandler.java:97) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.AbstractApiAction.lambda$prepareRequest$23(AbstractApiAction.java:614) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.14.0.jar:2.14.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:484) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.reloadCertificates(SecuritySSLCertsApiAction.java:190) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        ... 11 more
Caused by: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.validateNewCerts(DefaultSecurityKeyStore.java:644) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:462) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.reloadCertificates(SecuritySSLCertsApiAction.java:190) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        ... 11 more

Any help would be highly appreciated!

pablo · June 21, 2024, 9:36am

@reneradoi Try adding plugins.security.cert.oid: '1.2.3.4.5.5' into opensearch.yml file in every OS node.

github.com

opensearch-project/security/blob/435856c919e4e92cf030fbdadc11bf87dd991543/config/opensearch.yml.example#L188


      
          
          ############## Expert settings ##############
          # WARNING: Expert settings, do only use if you know what you are doing
          # If you set wrong values here this this could be a security risk
          # or make OpenSearch Security stop working
          
          # Name of the index where .opendistro_security stores its configuration.
          
          #plugins.security.config_index_name: .opendistro_security
          
          # This defines the OID of server node certificates
          #plugins.security.cert.oid: '1.2.3.4.5.5'
          
          # This specifies the implementation of org.opensearch.security.transport.InterClusterRequestEvaluator
          # that is used to determine inter-cluster request.
          # Instances of org.opensearch.security.transport.InterClusterRequestEvaluator must implement a single argument
          # constructor that takes an org.opensearch.common.settings.Settings
          #plugins.security.cert.intercluster_request_evaluator_class: org.opensearch.security.transport.DefaultInterClusterRequestEvaluator
          
          # By default, normal users can restore snapshots if they have the priviliges 'cluster:admin/snapshot/restore', 
          # 'indices:admin/create', and 'indices:data/write/index' for the indices to be restored.

reneradoi · June 21, 2024, 10:48am

Thank you @pablo for your assistance and quick response! I’ve added the config, but unfortunately it still fails.

I’ve done some debugging, and I found that it breaks when comparing the sorted san lists of the current and new cert here.

Here’s the debugger output (relevant part):

opensearch[opensearch-31][generic][T#11][1] print currentCertDNList
 currentCertDNList = "[CN=CN_CA,C=US/O=opensearch-w4du,CN=10.27.170.243/[[2, juju-1e4384-35], [2, juju-1e4384-35.lxd], [2, opensearch-31], [7, 10.27.170.243], [8, 1.2.3.4.5.5]]]"
opensearch[opensearch-31][generic][T#11][1] print newCertDNList
 newCertDNList = "[CN=CN_CA,C=US/O=opensearch-w4du,CN=10.27.170.243/[[2, opensearch-31], [2, juju-1e4384-35], [2, juju-1e4384-35.lxd], [7, 10.27.170.243], [8, 1.2.3.4.5.5]]]"

Even though the san content is completely identical on the certs, the sorting of the maps is breaking it.

Should I file a bug for that on github?

Topic		Replies	Views
Can't reload SSL certs via API Security	5	702	August 1, 2023
Failed to initialize the server-side SSL context / Certificate chain is not valid Security	1	308	February 23, 2024
SSLconfigException Error OpenSearch troubleshoot	26	915	January 11, 2024
Fails on SSL when using REST API on Getting Started local docker compose cluster General Feedback troubleshoot	2	2667	January 19, 2022
Certificate reloading? Security	3	1793	September 14, 2022

Exception when reloading self signed certs via REST API

Related Topics