BadPaddingException with self singed certificates

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch Version: 2.18.0

Describe the issue:
I am currently trying to implement SSL certificates from our own PKI. Unfortunately I don’t have access to it so I can’t export anything by myself.

I was given a pfx file for every node and the dashboard node. To use it with opensearch I have created a truststore as following:

keytool -importcert -keystore my-truststore.p12 -file my-root-ca-1.crt -alias my-root-ca -trustcacerts -deststoretype pkcs12
keytool -importcert -keystore my-truststore.p12 -file my-intermediate-ca-1.crt -alias my-intermediate-ca -trustcacerts -deststoretype pkcs12

And configured opensearch as following:

plugins:
  security:
    ssl:
      transport:
        keystore_type: PKCS12
        keystore_filepath: certs/opse1.my.domain.pfx
        keystore_password: thisisastupidpassword
        truststore_type: PKCS12
        truststore_filepath: certs/my-truststore.p12
        truststore_password: thisisanotherstupidpassword

When I try to start opensearch I get the following error:

org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.18.0.jar:2.18.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.18.0.jar:2.18.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
        ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
        ... 6 more
Caused by: org.opensearch.OpenSearchException: Failed to create KeyManagerFactory
        at org.opensearch.security.ssl.config.KeyStoreConfiguration.buildKeyManagerFactory(KeyStoreConfiguration.java:50) ~[?:?]
        at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:41) ~[?:?]
        at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
        at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$2(SslSettingsManager.java:100) ~[?:?]
        at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:99) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
        ... 6 more
Caused by: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:454) ~[?:?]
        at java.base/sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:93) ~[?:?]
        at java.base/java.security.KeyStore.getKey(KeyStore.java:1075) ~[?:?]
        at java.base/sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:141) ~[?:?]
        at java.base/sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:64) ~[?:?]
        at java.base/javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:270) ~[?:?]
        at org.opensearch.security.ssl.config.KeyStoreConfiguration.buildKeyManagerFactory(KeyStoreConfiguration.java:47) ~[?:?]
        at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:41) ~[?:?]
        at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
        at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$2(SslSettingsManager.java:100) ~[?:?]
        at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:99) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
        ... 6 more
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:861) ~[?:?]
        at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:941) ~[?:?]
        at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:734) ~[?:?]
        at java.base/com.sun.crypto.provider.DESedeCipher.engineDoFinal(DESedeCipher.java:296) ~[?:?]
        at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2244) ~[?:?]
        at java.base/sun.security.pkcs12.PKCS12KeyStore.lambda$engineGetKey$0(PKCS12KeyStore.java:370) ~[?:?]
        at java.base/sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:257) ~[?:?]
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:361) ~[?:?]
        at java.base/sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:93) ~[?:?]
        at java.base/java.security.KeyStore.getKey(KeyStore.java:1075) ~[?:?]
        at java.base/sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:141) ~[?:?]
        at java.base/sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:64) ~[?:?]
        at java.base/javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:270) ~[?:?]
        at org.opensearch.security.ssl.config.KeyStoreConfiguration.buildKeyManagerFactory(KeyStoreConfiguration.java:47) ~[?:?]
        at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:41) ~[?:?]
        at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
        at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$2(SslSettingsManager.java:100) ~[?:?]
        at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:99) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
        ... 6 more

When i verify the pfx file with openssl pkcs12 -info -in opse1.my.domain.pfx everything looks fine.

What I am doing wrong here?

@audacity363 Can you list the content of the opse1.my.domain.pfx file?

keytool -list -v -keystore opse1.my.domain.pfx -storepass <password> -storetype PKCS12

@audacity363 I’ve just tested 2.17.1 and 2.18.0. There seems to be a bug or a change that wasn’t documented. At least I can’t find anything in the changelog of 2.18.0.

I’m getting exactly the same error as you. The same keystore and truststore works in 2.17.1 and lower. I have this issue only in 2.18.0

In my case I’ve created certificates from scratch.
This is my test script.

#!/bin/bash
#

#Step 1 - Generate CA

echo "Generating RootCA certificate\n"

keytool -genkeypair -alias root-ca \
    -keyalg RSA -keysize 2048 -validity 3650 \
    -keystore root-ca.jks \
    -dname "CN=RootCA,OU=OpenSearch,O=YourOrganization,L=City,ST=State,C=Country" \
    -ext bc:c \
    -storepass rootcapassword \
    -keypass rootcapassword

#Step 2 - Export CA to sign other certificates

echo "Export RootCA certificate\n"

keytool -exportcert -alias root-ca \
    -file root-ca.crt \
    -keystore root-ca.jks \
    -storepass rootcapassword \
    -rfc

#Step 3 - Generate a node certificate

echo "Generating node certificate\n"

keytool -genkeypair -alias opensearch-node \
    -keyalg RSA -keysize 2048 -validity 365 \
    -keystore opensearch-node.jks \
    -dname "CN=node1.example.com, OU=OpenSearch, O=YourOrganization, L=City, ST=State, C=Country" \
    -storepass nodepassword \
    -keypass nodepassword

#Step 4 - Create a certificate signing request (CSR)

echo "Creating node CSR"

keytool -certreq -alias opensearch-node \
    -file opensearch-node.csr \
    -keystore opensearch-node.jks \
    -storepass nodepassword

#Step 5 - Sign the node certificate with RootCA

echo "Signing node certificate with RootCA cert"

keytool -gencert -alias root-ca \
    -infile opensearch-node.csr \
    -outfile opensearch-node-cert.crt \
    -keystore root-ca.jks \
    -storepass rootcapassword \
    -validity 365 \
    -ext SAN=dns:node1.example.com \
    -ext ku:c=dig,keyEncipherment \
    -ext eku=sa,cl


#Step 6 - Import the signed certificate back into node keystore

echo "Importing signed node certificate to opensearch-node.jks keystore"

keytool -importcert -trustcacerts -alias root-ca \
    -file root-ca.crt \
    -keystore opensearch-node.jks \
    -storepass nodepassword

keytool -importcert -trustcacerts -alias opensearch-node \
    -file opensearch-node-cert.crt \
    -keystore opensearch-node.jks \
    -storepass nodepassword


#Step 7 - Create the Truststore

echo "Creating the truststore opensearch-truststore.jks"

keytool -importcert -alias root-ca \
    -file root-ca.crt \
    -keystore opensearch-truststore.jks \
    -storepass truststorepassword

Would you mind creating a bug in OpenSearch GitHub and sharing the link in this thread?

@pablo
It seems that I have found the issue. There seems to be a new configuration key “keystore_keypassword” which is undocumented.
It’s counterpart “keystore_keypassword_secure” is mentioned in the section " Separate client and server keystore and truststore files" (Configuring TLS certificates - OpenSearch Documentation).

The following configuration is working:

plugins:
  security:
    ssl:
      transport:
        keystore_type: PKCS12
        keystore_filepath: certs/opse1.my.domain.pfx
        keystore_password: thisisastupidpassword
        keystore_keypassword: thisisastupidpassword
        truststore_type: PKCS12
        truststore_filepath: certs/my-truststore.p12
        truststore_password: thisisanotherstupidpassword
1 Like

@audacity363 Thanks for sharing this information. It worked at my end too. It looks like an undocumented option.

@audacity363 This issue has been reported as a bug in OpenSearch GitHub.

I am gettingd same error PLUS. the new setting did not help in my case.
Is the same needed for hte ss.http settings?

Caused by: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

got it!! There are TWO undocumented settings

plugins.security.ssl.http.keystore_keypassword: