File does not contain valid private key

Hello,

we want to use our own certificates, but OpenSearch doesn’t load the private key.

Error:

[2022-06-24T12:19:32,940][ERROR][o.o.b.Bootstrap          ] [log-node-test-1] Exception
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.node.Node.<init>(Node.java:413) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.node.Node.<init>(Node.java:336) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:244) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:244) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:414) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.0.1.jar:2.0.1]
	at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) [opensearch-2.0.1.jar:2.0.1]
Caused by: java.lang.reflect.InvocationTargetException
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /opt/opensearch-2.0.1/config/log-node-test-1.key
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:255) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /opt/opensearch-2.0.1/config/log-node-test-1.key
	at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:386) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:255) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more
Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked
	at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1155) ~[?:?]
	at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1123) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:255) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
	at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:170) ~[?:?]
	at java.security.KeyFactory.generatePrivate(KeyFactory.java:389) ~[?:?]
	at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1153) ~[?:?]
	at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1123) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:255) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more
Caused by: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
	at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:135) ~[?:?]
	at sun.security.pkcs.PKCS8Key.<init>(PKCS8Key.java:95) ~[?:?]
	at sun.security.ec.ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:75) ~[?:?]
	at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:245) ~[?:?]
	at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:166) ~[?:?]
	at java.security.KeyFactory.generatePrivate(KeyFactory.java:389) ~[?:?]
	at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1153) ~[?:?]
	at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1123) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:255) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more

Changes in the opensearch configuration:

#plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemcert_filepath: log-node-test-1.crt
#plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemkey_filepath: log-node-test-1.key
#plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: sectigo-chain.crt
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
#plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemcert_filepath: log-node-test-1.crt
#plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemkey_filepath: log-node-test-1.key
#plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.http.pemtrustedcas_filepath: sectigo-chain.crt

If we configure Apache to use the same certificate, everything works.

Any ideas?

@d_raulf How did you generate the certificates?

As per documentation, OpenSearch uses PKCS#8 format.