Demo certificates found?

[kwslavens]

I’m working on switching over to internal Certs and I’ve run into this start error. Demo certificates found. Do I need to remove all the demo certificates from the Elasticsearch servers?

The service says its up and running but it is not listening on any ports.

java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:156) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.node.Node.(Node.java:338) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.node.Node.(Node.java:265) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.5.4.jar:6.5.4]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.5.4.jar:6.5.4]
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.5.4.jar:6.5.4]
… 15 more
Caused by: java.lang.RuntimeException: Demo certificates found [d14aefe70a592d7a29e14f3ff89c3d0070c99e87d21776aa07d333ee877e758f, 54a70016e0837a2b0c5658d1032d7ca32e432c62c55f01a2bf5adcb69a0a7ba9, bdc141ab2272c779d0f242b79063152c49e1b06a2af05e0fd90d505f2b44d5f5, 3e839e2b059036a99ee4f742814995f2fb0ced7e9d68a47851f43a3c630b5324, 9b13661c073d864c28ad7b13eda67dcb6cbc2f04d116adc7c817c20b4c7ed361]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.(OpenDistroSecurityPlugin.java:368) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.5.4.jar:6.5.4]
… 15 more
[2019-04-03T13:09:18,709][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-idsvm01] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.5.4.jar:6.5.4]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.5.4.jar:6.5.4]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:156) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.node.Node.(Node.java:338) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.node.Node.(Node.java:265) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.4.jar:6.5.4]
… 6 more
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:156) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.node.Node.(Node.java:338) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.node.Node.(Node.java:265) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.4.jar:6.5.4]
… 6 more
Caused by: java.lang.RuntimeException: Demo certificates found [d14aefe70a592d7a29e14f3ff89c3d0070c99e87d21776aa07d333ee877e758f, 54a70016e0837a2b0c5658d1032d7ca32e432c62c55f01a2bf5adcb69a0a7ba9, bdc141ab2272c779d0f242b79063152c49e1b06a2af05e0fd90d505f2b44d5f5, 3e839e2b059036a99ee4f742814995f2fb0ced7e9d68a47851f43a3c630b5324, 9b13661c073d864c28ad7b13eda67dcb6cbc2f04d116adc7c817c20b4c7ed361]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.(OpenDistroSecurityPlugin.java:368) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:156) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.node.Node.(Node.java:338) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.node.Node.(Node.java:265) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.4.jar:6.5.4]

[kwslavens]

This appears to be to be caused by this line the Elasticsearch.yml

opendistro_security.allow_unsafe_democertificates: false

Even though I am not using any “demo” certificates if I set this to false Elasticsearch security plugin fails and will not allow elasticsearch to start correctly. Setting it to true clears the error.

I found this reference under Search Guard
https://github.com/floragunncom/search-guard/issues/432

This pointed me to the source of the issue.

Can anyone explain why this occurs when there are no demo Certificates being used?

[nprashar]

How did you create these “internal certs” ? I’m guessing something isn’t correct in either the signing, or possibly the root-ca, etc … We just put out a blog posts detailing how to create your own certificate ?

Add your own SSL certificates to Open Distro for Elasticsearch | Amazon Web...

Can you try what they are suggesting ?

[kwslavens]

I originally used the certutil that came from Elastic.co. I’ve used it for another Elasticsearch prod cluster. I have not switched to using the search guard tool that includes the DN and OID into the certs. This has worked much better but I still have the same issue with the demo cert error unless i keep this config in the elasticsearch.yml.

opendistro_security.allow_unsafe_democertificates: true

[hanozelavia]

I used the rpm version and followed the steps specified in the link above. However, I solved this issue by actually deleting the demo certs from the /etc/elasticsearch folder. Please note that moving the certs in to a new / temp sub-folder under /etc/elasticsearch does not work.

[kwslavens]

Thanks I’ll give that a try and see if it resolves the issue.

[j0hnth0m]

Ha, I moved the original certs to a temp folder under config. Thanks to your comment I deleted the folder and am able to use my custom certs with enabling the ‘demo cert’ setting.