Opensearch truststore issue

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 2.8.0
Deployment: docker-compose

Describe the issue:
To setup an Email notification channel I have to use an opensearch keystore to store the SMTP server credentials. In case I use a keystore, I also have to use a truststore and here’s the problem - I can’t find any information on how to create an opensearch compatible truststore. If I create a truststore using a java keytool, I’m getting the following error:
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer: java.io.IOException: Invalid keystore format

If I use the opensearch keystore as a truststore or if I use Java’s cacerts truststore I’m still getting the same error. That’s so confusing…

Configuration:
opensearch.yml:

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false

plugins.security.ssl.transport.keystore_filepath: custom.keystore
plugins.security.ssl.transport.keystore_password: password

plugins.security.ssl.transport.truststore_filepath: cacerts
plugins.security.ssl.transport.truststore_password: changeit

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem

plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de
plugins.security.nodes_dn:
  - 'CN=*'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models"]
node.max_local_storage_nodes: 3

Relevant Logs or Screenshots:

[2023-06-20T11:59:44,033][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node1] JVM supports TLSv1.3
[2023-06-20T11:59:44,034][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node1] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2023-06-20T11:59:44,034][DEBUG][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node1] Value for plugins.security.ssl.transport.keystore_filepath is custom.keystore
[2023-06-20T11:59:44,034][DEBUG][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node1] Resolved custom.keystore to /usr/share/opensearch/config/custom.keystore against /usr/share/opensearch/config
[2023-06-20T11:59:44,034][DEBUG][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node1] Value for plugins.security.ssl.transport.truststore_filepath is cacerts
[2023-06-20T11:59:44,035][DEBUG][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node1] Resolved cacerts to /usr/share/opensearch/config/cacerts against /usr/share/opensearch/config
[2023-06-20T11:59:44,083][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-node1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.8.0.jar:2.8.0]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.8.0.jar:2.8.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:791) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.node.Node.<init>(Node.java:454) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.node.Node.<init>(Node.java:381) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.8.0.jar:2.8.0]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.node.Node.<init>(Node.java:454) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.node.Node.<init>(Node.java:381) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.8.0.jar:2.8.0]
	... 6 more
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer: java.io.IOException: Invalid keystore format
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:370) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:256) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:177) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:266) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.node.Node.<init>(Node.java:454) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.node.Node.<init>(Node.java:381) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.8.0.jar:2.8.0]
	... 6 more
2

Hi @al3x

Can you see the contents of your keystore and truststore using keytool?

In order to check it, you can run the following command:

keytool -list -v -keystore custom.keystore
3

What is the reason for using both keystore and PEM certs at the same time?
Do the keystore and truststore match the format as per documentation?

4

Hi Eugene7,
Thank you for your reply!
Initially I went with PEM certs only and everything worked perfectly fine, however to setup email notifications I need to store SMTP server credentials and keystore seems to be the only option here. I cannot see the content of my opensearch keystore content with keytool:

#/usr/share/opensearch/jdk/bin/keytool -list -v -keystore custom.keystore
keytool error: java.security.KeyStoreException: Unrecognized keystore format. Please load it with a specified type
java.security.KeyStoreException: Unrecognized keystore format. Please load it with a specified type
	at java.base/java.security.KeyStore.getInstance(KeyStore.java:1808)
	at java.base/java.security.KeyStore.getInstance(KeyStore.java:1681)
	at java.base/sun.security.tools.keytool.Main.doCommands(Main.java:949)
	at java.base/sun.security.tools.keytool.Main.run(Main.java:423)
	at java.base/sun.security.tools.keytool.Main.main(Main.java:416)

# /usr/share/opensearch/jdk/bin/keytool -list -v -storetype JKS -keystore custom.keystore
keytool error: java.io.IOException: Invalid keystore format
java.io.IOException: Invalid keystore format
	at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:688)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
	at java.base/java.security.KeyStore.load(KeyStore.java:1473)
	at java.base/sun.security.tools.keytool.Main.doCommands(Main.java:954)
	at java.base/sun.security.tools.keytool.Main.run(Main.java:423)
	at java.base/sun.security.tools.keytool.Main.main(Main.java:416)

As per logs I provided I don’t have any problem with the keystore (created with opensearch-keystore tool), opensearch accepts it fine, but it fails to read the truststore (tried to create it using both opensearch-keystore and keytool with no avail).

5

Found the cause - it was actually not the truststore, but the keystore, which I had created using opensearch-keystore tool. After I re-created the keystore with the keytool opensearch has accepted it! The log is a bit confusing actually as it doesn’t tell which exactly file has invalid format.
Anyway, the issue is resolved.

1 Like