Keystore unable to work in version 2.18.0

OpenSearch
1

I was going to upgrade our OpenSearch cluster from 2.17.1 to 2.18.0 (tarball rolling upgrade), and copied the same keystore, truststore and opensearch.yml from version 2.17.1 which works properly. But 2.18.0 version got below exceptions during failed starting,

java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
…
Caused by: java.lang.reflect.InvocationTargetException
…
Caused by: org.opensearch.OpenSearchException: Failed to load key store form xxxxxx_keystore.jks
…
Caused by: java.security.UnrecoverableKeyException: Cannot recover key
…

We use the same keystore, struststore and opensearch.yml started from version 2.8.0, and there was no exception for every version upgrade except 2.18.0. The new version changed the logic of loading keystore file? How can I resolve it? Thanks.

2

@min, c ould you please share your opensearch.yml SSL configuration settings?

3

Here is my SSL setting,

plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.keystore_filepath: xxxxx_keystore.jks
plugins.security.ssl.transport.keystore_password: ${keystore_pwd}
plugins.security.ssl.transport.keystore_alias: xxxxx
plugins.security.ssl.transport.truststore_filepath: xxxxx_truststore.jks
plugins.security.ssl.transport.truststore_password: ${truststore_pwd}

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.keystore_filepath: xxxxx_keystore.jks
plugins.security.ssl.http.keystore_password: ${keystore_pwd}
plugins.security.ssl.http.keystore_alias: xxxxx
plugins.security.ssl.http.truststore_filepath: xxxxx_truststore.jks
plugins.security.ssl.http.truststore_password: ${truststore_pwd}

4

@min thank you.
I have a couple of questions:

  • Do you use a password for the private key?
  • If yes, which one: the default for the keystore (changeit) or an empty password?
5

@willyborankin
Thanks for help,

  1. Yes
  2. Neither default nor empty password for the keystore.

Moreover, in the keystore, we had below tries.

  1. Stored both domain cert and CA cert, it worked in previous versions but failed on 2.18.0
  2. Removed CA cert and only keep domain cert, it also failed on 2.18.0
6

@min, could you please create an issue in the security repository and include the full stack trace? Typically, an UnrecoverableKeyException is thrown when the JDK fails to decrypt the private key. Additionally, the default keystore and truststore types are now the JDK-provided defaults. Before version 2.18, the default was JKS.

7

@willyborankin sure, thanks.