When installing opensearch, am getting error like failed to load plugin class for securtiy plugin

info2kool · April 12, 2023, 10:33am

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 1.2.3

Describe the issue:

[ERROR][o.o.b.Bootstrap ] [devqa-logs-2] Exception
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.(PluginsService.java:194) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.node.Node.(Node.java:396) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.node.Node.(Node.java:319) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) [opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) [opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) [opensearch-1.2.3.jar:1.2.3]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) [opensearch-1.2.3.jar:1.2.3]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-1.2.3.jar:1.2.3]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) [opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) [opensearch-1.2.3.jar:1.2.3]
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-1.2.3.jar:1.2.3]
… 15 more
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: java.security.cert.CertificateParsingException: signed fields invalid
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:179) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]

Configuration:
cluster.name: devqa-loges
node.name: devqa-logs-2
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: [“10.10.20.123”]
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes: devqa-logs-1
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3
plugins.security.allow_default_init_securityindex: true
compatibility.override_main_response_version: true
plugins.security.ssl.transport.pemcert_filepath: xxx.pem
plugins.security.ssl.transport.pemkey_filepath: xx-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: entrustroot.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.http.pemcert_filepath: xxx.pem
plugins.security.ssl.http.pemkey_filepath: xx-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: entrustroot.pem

Relevant Logs or Screenshots:

pablo · April 13, 2023, 12:00am

@info2kool Did you use demo certificates or custom in opensearch.yml?

info2kool · April 13, 2023, 12:16am

Hey @pablo I used custom certs which is signed by Entrust Root CA cert. We dont have seperate certs for each node in our cluster as such but we use one general SSL cert for entire organization. And the CN would like this - CN = *.organization.com

pablo · April 13, 2023, 12:25am

@info2kool Are those certs in the supported format as per docs?

info2kool · April 13, 2023, 9:48am

@pablo Yes all certs are in supported format. That above error is resolved now. But I am getting new exceptions like certificate_unknown and unable to find valid certification path to the requested target.

We have 2 nodes opensearch cluster. Both nodes have the same configuration as below.

cluster.name: devqa-loges
node.name: devqa-logs-2
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: [“10.10.20.244”]
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes: devqa-logs-1
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3
plugins.security.allow_default_init_securityindex: true
compatibility.override_main_response_version: true
plugins.security.ssl.transport.pemcert_filepath: xxx.pem
plugins.security.ssl.transport.pemkey_filepath: xx-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: entrustroot.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.http.pemcert_filepath: xxx.pem
plugins.security.ssl.http.pemkey_filepath: xx-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: entrustroot.pem
#plugins.security.authcz.admin_dn:

- ‘CN=Entrust,OU=Entrust OU,O=Entrust,L=Toronto,ST=Ontario,C=CA’

plugins.security.nodes_dn:

‘CN=*.organization.com,O=ORG NAME,L=Toronto,S=Ontario,C=CA’

For Pemtrustedcas, I used Entrust Root CA cert but commented out that section in the config file, do you think i need to enable it ?

Exceptions:
[ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [devqa-logs-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]

Exceptions:

[ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [devqa-logs-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandsha
keException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:202) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:171) ~[?:?]

Please help me in resolving this issue.

pablo · April 13, 2023, 3:32pm

@info2kool The pemtrustedcas must be defined for transport and http in the opensearch.yml.

The plugins.security.ssl.transport.enabled: true shouldn’t be used as SSL in transport layer must be always enabled.

info2kool · April 13, 2023, 4:30pm

@pablo The pemtrustedcas is already defined for transport and http in the opensearch.yml.

plugins.security.ssl.transport.pemtrustedcas_filepath: entrustroot.pem
plugins.security.ssl.http.pemtrustedcas_filepath: entrustroot.pem

I didn’t understand the second line… u meant plugins.security.ssl.transport.enabled: true should not be true?

pablo · April 13, 2023, 8:04pm

I was referring to this comment.

@info2kool You don’t need this line. It can’t be disabled and is true by default.

pablo · April 13, 2023, 8:05pm

@info2kool How many master nodes do you have?

info2kool · April 14, 2023, 5:45am

@pablo I have only one master node.

By disabling the security, everything is working fine. When I am security plugin and adding these certs, I am having these issues.

Ironically my master node is working fine when using the security plugin but the other node is not working. when I checked that node url in browser it is saying security index not initialized message.

pablo · April 14, 2023, 9:52am

@info2kool The other node is trying to connect to the existing cluster. It is failing to connect due to SLL handshake failure on the transport layer. That also produce the error OpenSearch Security not initialized as the new node can’t access the .opendistro_security index that holds the security plugin configuration.
The security plugin must be initialized once. That creates .opendistro_security index. All nodes that join the cluster can access the same security configuration. You don’t have to initialize the security plugin with every new node in the cluster.

Did you use the same root CA certificate for both nodes?
You can check that with the below command.

openssl x509 -in <node_cert> -text -noout

info2kool · April 14, 2023, 10:08am

@pablo Yes, I am using the same cert on both nodes. Please find the below cert details.

openssl x509 -in entrustroot.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1246989352 (0x4a538c28)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = “Entrust, Inc.”, OU = See Legal and Compliance, OU = “(c) 2009 Entrust, Inc. - for authorized use only”, CN = Entrust Root Certification Authority - G2
Validity
Not Before: Jul 7 17:25:54 2009 GMT
Not After : Dec 7 17:55:54 2030 GMT
Subject: C = US, O = “Entrust, Inc.”, OU = See Legal and Compliance, OU = “(c) 2009 Entrust, Inc. - for authorized use only”, CN = Entrust Root Certification Authority - G2
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:ba:84:b6:72:db:9e:0c:6b:e2:99:e9:30:01:a7:
76:ea:32:b8:95:41:1a:c9:da:61:4e:58:72:cf:fe:
f6:82:79:bf:73:61:06:0a:a5:27:d8:b3:5f:d3:45:
4e:1c:72:d6:4e:32:f2:72:8a:0f:f7:83:19:d0:6a:
80:80:00:45:1e:b0:c7:e7:9a:bf:12:57:27:1c:a3:
68:2f:0a:87:bd:6a:6b:0e:5e:65:f3:1c:77:d5:d4:
85:8d:70:21:b4:b3:32:e7:8b:a2:d5:86:39:02:b1:
b8:d2:47:ce:e4:c9:49:c4:3b:a7:de:fb:54:7d:57:
be:f0:e8:6e:c2:79:b2:3a:0b:55:e2:50:98:16:32:
13:5c:2f:78:56:c1:c2:94:b3:f2:5a:e4:27:9a:9f:
24:d7:c6:ec:d0:9b:25:82:e3:cc:c2:c4:45:c5:8c:
97:7a:06:6b:2a:11:9f:a9:0a:6e:48:3b:6f:db:d4:
11:19:42:f7:8f:07:bf:f5:53:5f:9c:3e:f4:17:2c:
e6:69:ac:4e:32:4c:62:77:ea:b7:e8:e5:bb:34:bc:

pablo · April 14, 2023, 10:26am

@info2kool This is your root CA. What about the cert defined in plugins.security.ssl.transport.pemcert_filepath in both nodes?

Are they signed by the same root CA (the same X509v3 Authority Key Identifier)?

Could you check the cert nodes with the same command and confirm that X509v3 Authority Key Identifier: is the same for all node certs and root CA?

Is there any intermediate root CA in the trust chain?

How do you deploy your cluster? Is it a service or a container?

info2kool · April 14, 2023, 12:47pm

@pablo

What about the cert defined in plugins.security.ssl.transport.pemcert_filepath in both nodes? - Yes it is the same cert which has CN value as CN=*.organizationname.ca

Are they signed by the same root CA (the same X509v3 Authority Key Identifier )? - Yes

Could you check the cert nodes with the same command and confirm that X509v3 Authority Key Identifier: is the same for all node certs and root CA? - Yes

Is there any intermediate root CA in the trust chain? - Yes

I have deployed it as a service. Below is the screenshot of the cert with root ca and intermediate ca.

info2kool · April 17, 2023, 9:15am

Hi @pablo I resolved the issues and am able to use my certs to login to opensearch with default credentials. Here is my opensearch.yml looks like in both the nodes.

cluster.name: devqa-loges
node.name: devqa-logs-1
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: [“10.10.20.243”,“10.10.20.244”]
cluster.initial_master_nodes: [“devqa-logs-1”, “devqa-logs-2”]
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3
compatibility.override_main_response_version: true
plugins.security.allow_default_init_securityindex: true
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/config/node.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/config/node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/config/entrustl1k.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: false
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/config/node.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/config/node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/config/entrustl1k.pem
plugins.security.authcz.admin_dn:

‘CN=Entrust Certification Authority - L1K,OU=“(c) 2012 Entrust, Inc. - for authorized use only”,OU=See www.entrust.net/legal-terms,O=“Entrust, Inc.”,C=US’
plugins.security.nodes_dn:
‘CN=*.organization.ca,O=OrganizationName,L=Toronto,ST=Ontario,C=CA’

However, when I am trying to run securityadmin.sh to use my basic HTTP authentication internal users , I am getting error message like below.

Error MSG:
…/plugins/opensearch-security/tools/./securityadmin.sh -cd …/securityconfig/ -icl -nhnv \

-cacert /etc/opensearch/config/entrustl1k.pem
-cert /etc/opensearch/config/node.pem
-key /etc/opensearch/config/node-key.pem
WARNING: JAVA_HOME not set, will use /usr/bin/java
Security Admin v7
Will connect to localhost:9300 … done
Connected as CN=.organization.ca,O=ORGNAME,L=Toronto,ST=Ontario,C=CA
ERR: CN=.organization.ca,O=ORGNAME=Toronto,ST=Ontario,C=CA is not an admin user
Seems you use a node certificate. This is not permitted, you have to use a client certificate and register it as admin_dn in opensearch.yml

I have entrust root and intermediate certs and node cert signed by the entrust. What cert and key I need to use in the configuration for -cert and -key value?

pablo · April 17, 2023, 1:43pm

@info2kool You’re trying to use a node certificate to authenticate as an admin.
The admin_dn doesn’t contain the CN that was in the cert used with securityadmin.sh. You shouldn’t use node certificates as admin certs.
Also, as per the documentation, you can’t use wildcards or regular expressions with admin_dn, therefore CN of the admin certificate can’t be a wildcard.

How many certificates does your entrustl1k.pem contain? Is it just root CA?

info2kool · April 17, 2023, 2:17pm

@pablo entrustl1k.pem contains only Entrust Intermediate cert. So in my scenario, what should be the admin cert and key to use ?

pablo · April 17, 2023, 5:08pm

@info2kool The certificate file in the plugins.security.ssl.http.pemtrustedcas_filepath: should contain RootCA and plugins.security.ssl.transport.pemcert_filepath
Intermediate RootCA and node certificates.

node cert content example

-----BEGIN CERTIFICATE-----
<node certificate>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<Intermediate/Signing certificate>
-----END CERTIFICATE-----

As per documentation, admin certificates are client certificates.

info2kool · April 17, 2023, 5:23pm

@pablo can u provide any admin cert content example as well? I am not finding it anywhere on how this admin certificate is generated?

pablo · April 17, 2023, 5:49pm

@info2kool Please check this procedure.

Topic		Replies	Views
Unable to start opensearch from command line (Error: plugins.security.ssl.transport.keystore_filepath) OpenSearch configure , security-issue	3	209	February 26, 2024
OpenSearch ssl exception (tar) OpenDistro	2	7093	December 9, 2021
Using Helm chart, demo install with default certs fails to start OpenSearch install	8	226	March 26, 2024
File does not contain valid private key Security troubleshoot	2	1079	January 29, 2024
Need Support to use open search in windows OpenSearch troubleshoot , install	4	310	October 7, 2023

When installing opensearch, am getting error like failed to load plugin class for securtiy plugin

- ‘CN=Entrust,OU=Entrust OU,O=Entrust,L=Toronto,ST=Ontario,C=CA’

Related Topics