File does not contain valid private key

Security
1

Hello,

we want to use our own certificates, but OpenSearch doesn’t load the private key.

Error:

[2022-06-24T12:19:32,940][ERROR][o.o.b.Bootstrap          ] [log-node-test-1] Exception
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.node.Node.<init>(Node.java:413) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.node.Node.<init>(Node.java:336) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:244) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:244) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:414) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.0.1.jar:2.0.1]
	at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) [opensearch-2.0.1.jar:2.0.1]
Caused by: java.lang.reflect.InvocationTargetException
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /opt/opensearch-2.0.1/config/log-node-test-1.key
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:255) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /opt/opensearch-2.0.1/config/log-node-test-1.key
	at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:386) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:255) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more
Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked
	at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1155) ~[?:?]
	at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1123) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:255) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
	at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:170) ~[?:?]
	at java.security.KeyFactory.generatePrivate(KeyFactory.java:389) ~[?:?]
	at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1153) ~[?:?]
	at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1123) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:255) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more
Caused by: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
	at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:135) ~[?:?]
	at sun.security.pkcs.PKCS8Key.<init>(PKCS8Key.java:95) ~[?:?]
	at sun.security.ec.ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:75) ~[?:?]
	at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:245) ~[?:?]
	at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:166) ~[?:?]
	at java.security.KeyFactory.generatePrivate(KeyFactory.java:389) ~[?:?]
	at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1153) ~[?:?]
	at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1123) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:255) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.0.1.jar:2.0.1]
	... 15 more

Changes in the opensearch configuration:

#plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemcert_filepath: log-node-test-1.crt
#plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemkey_filepath: log-node-test-1.key
#plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: sectigo-chain.crt
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
#plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemcert_filepath: log-node-test-1.crt
#plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemkey_filepath: log-node-test-1.key
#plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.http.pemtrustedcas_filepath: sectigo-chain.crt

If we configure Apache to use the same certificate, everything works.

Any ideas?

2

@d_raulf How did you generate the certificates?

As per documentation, OpenSearch uses PKCS#8 format.

3

I ended up here after searching for “File does not contain valid private key”. While this post did not address my issue, I thought I share my solution to my issue, just in case someone else also searches and ends up here. :slight_smile:

I am deploying Opensearch 2.11.1 on Azure Kubernetes Service (ASK) and I had the error “File does not contain valid private key” after following the instructions in the official Opensearch documentation to the letter (although I might be blind and might have overlooked something). I also tried using a password (optional AES256 flag).

After a lot of back and forth, this is what worked for me:

  1. I generated the keys using the instructions from Opensearch, without a password
  2. I connected to my AKS cluster and stored the keys in a Kubernetes Secret
  3. I added a volume and a volumeMount in the Kubernetes deployment manifest
  4. I created environment variables for each key in the deployment manifest
  5. Finally, I added the variable to the opensearch.yml file

Note:
I found it easiest to create a separate Kubernetes Secret for each node and each client, meaning repeat steps 2-5 for each node and client, even though Kubernetes Secrets are per namespace and accessible by all pods in that namespace. I repeated the mentioned steps for Opensearch Node1, Opensearch Node2, and Opensearch Dashboards.

Let us take node1.pem as an example. I am simplifying the instructions, assuming you know where to place the sections in the deployment and other files. If not, you should find templates online.

Store key in Kubernetes Secret:
kubectl create secret generic opensearch-node1-certificates
–from-file=node1.pem=./node1.pem
-n “your Kubernetes namespace”

Create volume and volumeMount in Kubernetes deployment manifest
volumeMounts:

  • name: opensearch-certificates-mount
    mountPath: “/usr/share/opensearch/config/certificates”
    readOnly: true

volumes:

  • name: opensearch-certificates-mount
    secret:
    secretName: opensearch-node1-certificates

Add env variable in Kubernetes deployment manifest

  • name: NODE_CERTIFICATE_PEM
    value: “certificates/node1.pem”

Add variable to opensearch.yml
plugins.security.ssl.transport.pemcert_filepath: ${NODE_CERTIFICATE_PEM}
plugins.security.ssl.http.pemcert_filepath: ${NODE_CERTIFICATE_PEM}

Note:
Remember to add all other security settings, like the corresponding pemkey_filepath (transport as well as http) and plugins.security.nodes_dn.

Most importantly:
Cross your fingers!
:smiling_face:

And if this still does not work, open a beer and talk to a rubber duck. Because the Ballmer Peak together with old-school debugging is still the best way forward (move over, lousy AI Copilot!).
:beer: :duck:

I hope this helps someone!
:beers: