Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch Version: 2.18.0
Describe the issue:
I am currently trying to implement SSL certificates from our own PKI. Unfortunately I don’t have access to it so I can’t export anything by myself.
I was given a pfx file for every node and the dashboard node. To use it with opensearch I have created a truststore as following:
keytool -importcert -keystore my-truststore.p12 -file my-root-ca-1.crt -alias my-root-ca -trustcacerts -deststoretype pkcs12
keytool -importcert -keystore my-truststore.p12 -file my-intermediate-ca-1.crt -alias my-intermediate-ca -trustcacerts -deststoretype pkcs12
And configured opensearch as following:
plugins:
security:
ssl:
transport:
keystore_type: PKCS12
keystore_filepath: certs/opse1.my.domain.pfx
keystore_password: thisisastupidpassword
truststore_type: PKCS12
truststore_filepath: certs/my-truststore.p12
truststore_password: thisisanotherstupidpassword
When I try to start opensearch I get the following error:
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.18.0.jar:2.18.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.18.0.jar:2.18.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
Caused by: org.opensearch.OpenSearchException: Failed to create KeyManagerFactory
at org.opensearch.security.ssl.config.KeyStoreConfiguration.buildKeyManagerFactory(KeyStoreConfiguration.java:50) ~[?:?]
at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:41) ~[?:?]
at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$2(SslSettingsManager.java:100) ~[?:?]
at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:99) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
Caused by: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:454) ~[?:?]
at java.base/sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:93) ~[?:?]
at java.base/java.security.KeyStore.getKey(KeyStore.java:1075) ~[?:?]
at java.base/sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:141) ~[?:?]
at java.base/sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:64) ~[?:?]
at java.base/javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:270) ~[?:?]
at org.opensearch.security.ssl.config.KeyStoreConfiguration.buildKeyManagerFactory(KeyStoreConfiguration.java:47) ~[?:?]
at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:41) ~[?:?]
at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$2(SslSettingsManager.java:100) ~[?:?]
at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:99) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:861) ~[?:?]
at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:941) ~[?:?]
at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:734) ~[?:?]
at java.base/com.sun.crypto.provider.DESedeCipher.engineDoFinal(DESedeCipher.java:296) ~[?:?]
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2244) ~[?:?]
at java.base/sun.security.pkcs12.PKCS12KeyStore.lambda$engineGetKey$0(PKCS12KeyStore.java:370) ~[?:?]
at java.base/sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:257) ~[?:?]
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:361) ~[?:?]
at java.base/sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:93) ~[?:?]
at java.base/java.security.KeyStore.getKey(KeyStore.java:1075) ~[?:?]
at java.base/sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:141) ~[?:?]
at java.base/sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:64) ~[?:?]
at java.base/javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:270) ~[?:?]
at org.opensearch.security.ssl.config.KeyStoreConfiguration.buildKeyManagerFactory(KeyStoreConfiguration.java:47) ~[?:?]
at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:41) ~[?:?]
at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$2(SslSettingsManager.java:100) ~[?:?]
at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:99) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
When i verify the pfx file with openssl pkcs12 -info -in opse1.my.domain.pfx
everything looks fine.
What I am doing wrong here?