Securityadmin errors with certificates

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.6.0
Server OS: Fedora Server 37

Describe the issue:

From Documentation:

plugins.security.ssl.transport.pemcert_filepath - Path to the X.509 node certificate chain (PEM format)
plugins.security.ssl.transport.pemtrustedcas_filepath - Path to the root CAs (PEM format)

plugins.security.ssl.http.pemcert_filepath - Path to the X.509 node certificate chain (PEM format)
plugins.security.ssl.http.pemtrustedcas_filepath - Path to the root CAs (PEM format)

securityadmin.sh

-cert - The location of the PEM file containing the admin certificate and all intermediate certificates, if any.

-cacert - The location of the PEM file containing the root certificate.

I’ve got such certificate for admin:
1

jet.pem - node certificate
jet-key.pem - node key
jet-ca.pem - as documentation says chain of CA + INT-CA
CA.pem - Root CA

admin.pem - as documentation says admin certificate + INT-CA

CA and INT-CA jas been added to /etc/pki/ca-trust/source/anchors/

update-ca-trust

securityadmin.sh -cd /opt/opensearch/config/opensearch-security/ -cacert CA.pem -cert admin.pem -key admin-key.pem -icl -nhnv

ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Another try:
securityadmin.sh -cd /opt/opensearch/config/opensearch-security/ -cacert jet-ca.pem -cert admin.pem -key admin-key.pem -icl -nhnv

ERR: An unexpected SSLHandshakeException occured: Received fatal alert: certificate_unknown

Any suggestions?

Configuration:

network.host: 0.0.0.0
discovery.type: single-node
plugins.security.disabled: false

plugins.security.ssl.transport.pemcert_filepath: /opt/opensearch/config/jet.pem
plugins.security.ssl.transport.pemkey_filepath: /opt/opensearch/config/jet-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /opt/opensearch/config/jet-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /opt/opensearch/config/jet.pem
plugins.security.ssl.http.pemkey_filepath: /opt/opensearch/config/jet-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /opt/opensearch/config/jet-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • CN=admin.apps.okd.cvbs.jet.msk.su,OU=IT,O=JET,L=Moscow,ST=Moscow,C=RU

plugins.security.nodes_dn:

  • CN=*.apps.okd.cvbs.jet.msk.su

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]

Relevant Logs or Screenshots:

I added admin certificate to keystore (full chain cert + CA + INT), and CA + INT to trusted keystore.

Now i’ve got another error:

OPENSEARCH_JAVA_HOME=/opt/opensearch/jdk /opt/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd /opt/opensearch/config/opensearch-security/ -ks /opt/opensearch/jdk/lib/security/cacerts -kspass changeit -kst JKS -ksalias JETADMIN -ts /opt/opensearch/jdk/lib/security/trusted.jks -tspass Qwerty_1 -tst JKS -tsalias JETCA -icl -nhnv

ERR: null is not an admin user
Seems you use a client certificate but this one is not registered as admin_dn
Make sure opensearch.yml on all nodes contains:
plugins.security.authcz.admin_dn:

  • “null”

openssl x509 -in admin.pem -noout -subject
subject=C = RU, ST = Moscow, L = Moscow, O = JET, OU = IT, CN = admin.apps.okd.cvbs.jet.msk.su

Could it be because of my certificate got Extensions?

Hey @anteus

Example, your setup maybe different.

What I did was find which java version

java -version

Check for full path.

sudo update-alternatives --config java

Set Java home.

export OPENSEARCH_JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

as for this error

Your cert must match with the opensearch.yml file.

hope that helps

Hi, thanks for the reply.

  1. I use opensearch.tar so java already builtin.

openjdk version “17.0.6”

  1. I decided to start from begining cause i can’t fix errors…
    I don’t use self-signed certificates. I tried and it works. I use internal CA to sign.

So now i have:

  • admin.pem (certificate + CA)
  • admin-key.pem (private key)
  • jet.pem (node certificate + CA)
  • jet-key.pem (private key)
  • jet-ca.pem (ROOT CA)

I’ve copied jet-ca.pem to /etc/pki/ca-trust/source/anchors/ and done command update-ca-trust

Started opensearch

After:
securityadmin.sh -cd /opt/opensearch/config/opensearch-security/ -cacert jet-ca.pem -cert admin.pem -key admin-key.pem -icl -nhnv

I’ve got:

Security Admin v7
Will connect to localhost:9200 … done
ERR: An unexpected SSLHandshakeException occured: Received fatal alert: certificate_unknown
See Java high-level REST client - OpenSearch documentation for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
See Java high-level REST client - OpenSearch documentation for troubleshooting.
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:947)
at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:462)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

I installed Nginx to check certificate and it works fine…

1

I don’t get it why there is certificate_unknown error in opensearch…

Also flags -dg and -w for command securityadmin.sh don’t show any additional information.

Hey @anteus

Is you Root CA in the trusted store? Did you make your own keystore or are you using JAVA default “cacerts”?

I’ve added jet-ca.pem to java default cacerts.

/opt/opensearch/jdk/bin/keytool -importcert -file jet-ca.pem -cacerts -alias DPFS

Hey @anteus

Understood… What do you get when executing this?

echo $JAVA_HOME

If the above command is correct then I think the issue you’re having is with the certificates.

It’s strange cause i did everything like in instructions:

# Create a private key for the admin certificate.
sudo openssl genrsa -out admin-key-temp.pem 2048

# Convert the private key to PKCS#8.
sudo openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
   
# Create the certficiate signing request (CSR). A common name (CN) of "A" is acceptable because this certificate is
# used for authenticating elevated access and is not tied to a host.
sudo openssl req -new -key admin-key.pem -subj "/CN=Admin" -out admin.csr

Except last step. I signed csr with my AD Certificate Service.
I checked certificates with Nginx and they are fine. I think i tried everything for last 5 days but opensearch doesn’t want to apply changes with my certs…

Difference is only with certificate extensions. Self-signed admin certificate has no extensions.

hey,

For troubleshooting have you tried to just use the self-signed certs? If that results is the same issue then we could look at other configurations made. Just an idea.

Some else i did different then you , was I used my java keystore.

keytool -importcert -keystore  /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts -storepass changeit  -alias opensearch.domain.com -file /etc/opensearch/root-ca.pem

Also in my notes, I had to set java path, mine was.

OPENSEARCH_JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/

by chance did you create your certificates with localhost? I adjusted my opensearch.yml file.

network.host: opensearch.domain.com (i.e.192.168.1.100)

Once completed i executed the following.

./securityadmin.sh -h opensearch.domain.com   -cd /etc/opensearch/opensearch-security/ -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/admin.pem -key /etc/opensearch/admin-key.pem -icl -nhnv

What I found was if you do not use the -h opensearch.domain.com then I believe it defaults to localhost.

Hope that helps

1 Like

hi.
There is my identity keystore:

There is my trust keystore:

OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk ./securityadmin.sh -cd /etc/opensearch/opensearch-security/ -h opensearch.dpfs3197.int -ks /etc/opensearch/identity.jks -ksalias admin -kspass changeit -kst JKS -ts /etc/opensearch/trust.jks -tspass changeit -tsalias jetca -tst JKS -icl -nhnv

Will connect to opensearch.dpfs3197.int:9200 … done
ERR: An unexpected SSLHandshakeException occured: Received fatal alert: certificate_unknown
See Java high-level REST client - OpenSearch documentation for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

Hey @anteus

I think you missing the part about the keystore. This would be where you store you certificate so Openseach can access them.

You may want to read back over the documentation here…

EDIT: I just seen you have a keystore called trust.jks by chance did you configured it so Opensearch can find it?

As shown from that link above , in this section marked in red.

1 Like

Hi. Looks like i found the solution.
Node certificates need to have both serverAuth and clientAuth set in the extended key usage field.

I added new template to Active Directory Certificate Services with this extensions.
1

After this i recreated new certificates with this template and opensearch command has worked without errors.

OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk ./securityadmin.sh -cd /etc/opensearch/opensearch-security/ -h opensearch.dpfs3197.int -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/admin.pem -key /etc/opensearch/admin-key.pem -icl -nhnv

Will connect to opensearch.dpfs3197.int:9200 … done
Connected as “CN=Admin”
OpenSearch Version: 2.7.0
Contacting opensearch cluster ‘opensearch’ and wait for YELLOW clusterstate …
Clustername: opensearch
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/opensearch/opensearch-security/
Will update ‘/config’ with /etc/opensearch/opensearch-security/config.yml
SUCC: Configuration for ‘config’ created or updated
Will update ‘/roles’ with /etc/opensearch/opensearch-security/roles.yml
SUCC: Configuration for ‘roles’ created or updated
Will update ‘/rolesmapping’ with /etc/opensearch/opensearch-security/roles_mapping.yml
SUCC: Configuration for ‘rolesmapping’ created or updated
Will update ‘/internalusers’ with /etc/opensearch/opensearch-security/internal_users.yml
SUCC: Configuration for ‘internalusers’ created or updated
Will update ‘/actiongroups’ with /etc/opensearch/opensearch-security/action_groups.yml
SUCC: Configuration for ‘actiongroups’ created or updated
Will update ‘/tenants’ with /etc/opensearch/opensearch-security/tenants.yml
SUCC: Configuration for ‘tenants’ created or updated
Will update ‘/nodesdn’ with /etc/opensearch/opensearch-security/nodes_dn.yml
SUCC: Configuration for ‘nodesdn’ created or updated
Will update ‘/whitelist’ with /etc/opensearch/opensearch-security/whitelist.yml
SUCC: Configuration for ‘whitelist’ created or updated
Will update ‘/audit’ with /etc/opensearch/opensearch-security/audit.yml
SUCC: Configuration for ‘audit’ created or updated
Will update ‘/allowlist’ with /etc/opensearch/opensearch-security/allowlist.yml
SUCC: Configuration for ‘allowlist’ created or updated
SUCC: Expected 10 config types for node {“updated_config_types”:[“allowlist”,“tenants”,“rolesmapping”,“nodesdn”,“audit”,“roles”,“whitelist”,“internalusers”,“actiongroups”,“config”],“updated_config_size”:10,“message”:null} is 10 ([“allowlist”,“tenants”,“rolesmapping”,“nodesdn”,“audit”,“roles”,“whitelist”,“internalusers”,“actiongroups”,“config”]) due to: null
Done with success

1 Like

Hey @anteus

Thanks for the update :+1:

1 Like

thanks for coming back to share the solution with the community @anteus!!

1 Like