Securityadmin errors with certificates

[anteus]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.6.0
Server OS: Fedora Server 37

Describe the issue:

From Documentation:

plugins.security.ssl.transport.pemcert_filepath - Path to the X.509 node certificate chain (PEM format)
plugins.security.ssl.transport.pemtrustedcas_filepath - Path to the root CAs (PEM format)

plugins.security.ssl.http.pemcert_filepath - Path to the X.509 node certificate chain (PEM format)
plugins.security.ssl.http.pemtrustedcas_filepath - Path to the root CAs (PEM format)

securityadmin.sh

-cert - The location of the PEM file containing the admin certificate and all intermediate certificates, if any.

-cacert - The location of the PEM file containing the root certificate.

I’ve got such certificate for admin:

jet.pem - node certificate
jet-key.pem - node key
jet-ca.pem - as documentation says chain of CA + INT-CA
CA.pem - Root CA

admin.pem - as documentation says admin certificate + INT-CA

CA and INT-CA jas been added to /etc/pki/ca-trust/source/anchors/

update-ca-trust

securityadmin.sh -cd /opt/opensearch/config/opensearch-security/ -cacert CA.pem -cert admin.pem -key admin-key.pem -icl -nhnv

ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Another try:
securityadmin.sh -cd /opt/opensearch/config/opensearch-security/ -cacert jet-ca.pem -cert admin.pem -key admin-key.pem -icl -nhnv

ERR: An unexpected SSLHandshakeException occured: Received fatal alert: certificate_unknown

Any suggestions?

Configuration:

network.host: 0.0.0.0
discovery.type: single-node
plugins.security.disabled: false

plugins.security.ssl.transport.pemcert_filepath: /opt/opensearch/config/jet.pem
plugins.security.ssl.transport.pemkey_filepath: /opt/opensearch/config/jet-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /opt/opensearch/config/jet-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /opt/opensearch/config/jet.pem
plugins.security.ssl.http.pemkey_filepath: /opt/opensearch/config/jet-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /opt/opensearch/config/jet-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

• CN=admin.apps.okd.cvbs.jet.msk.su,OU=IT,O=JET,L=Moscow,ST=Moscow,C=RU

plugins.security.nodes_dn:

• CN=*.apps.okd.cvbs.jet.msk.su

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]

Relevant Logs or Screenshots:

[anteus]

I added admin certificate to keystore (full chain cert + CA + INT), and CA + INT to trusted keystore.

Now i’ve got another error:

OPENSEARCH_JAVA_HOME=/opt/opensearch/jdk /opt/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd /opt/opensearch/config/opensearch-security/ -ks /opt/opensearch/jdk/lib/security/cacerts -kspass changeit -kst JKS -ksalias JETADMIN -ts /opt/opensearch/jdk/lib/security/trusted.jks -tspass Qwerty_1 -tst JKS -tsalias JETCA -icl -nhnv

ERR: null is not an admin user
Seems you use a client certificate but this one is not registered as admin_dn
Make sure opensearch.yml on all nodes contains:
plugins.security.authcz.admin_dn:

• “null”

openssl x509 -in admin.pem -noout -subject
subject=C = RU, ST = Moscow, L = Moscow, O = JET, OU = IT, CN = admin.apps.okd.cvbs.jet.msk.su

Could it be because of my certificate got Extensions?

[Gsmitt]

Hey @anteus

@anteus

OPENSEARCH_JAVA_HOME=/opt/opensearch/jdk /opt/opensearch/plugins/opensearch-

Example, your setup maybe different.

What I did was find which java version

java -version

Check for full path.

sudo update-alternatives --config java

Set Java home.

export OPENSEARCH_JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

as for this error

@anteus

ERR: null is not an admin user
Seems you use a client certificate but this one is not registered as admin_dn
Make sure opensearch.yml on all nodes contains:
plugins.security.authcz.admin_dn:

Your cert must match with the opensearch.yml file.

Error on Apply changes using securityadmin.sh

Hey @luke_devon Check your admin.pem certificate. Example: openssl x509 -in admin.pem -text -noout Make sure it matches this section. plugins.security.authcz.admin_dn: - 'CN=opensearch.domain.com,OU=admin,O=my_job,L=cedar,ST=iowa,C=us' Looks like you have double quotes, is this correct? https://opensearch.org/docs/latest/security/configuration/generate-certificates/#add-certificate-files-to-opensearchyml

hope that helps

[anteus]

Hi, thanks for the reply.

1. I use opensearch.tar so java already builtin.

openjdk version “17.0.6”

2. I decided to start from begining cause i can’t fix errors…
   I don’t use self-signed certificates. I tried and it works. I use internal CA to sign.

So now i have:

• admin.pem (certificate + CA)
• admin-key.pem (private key)
• jet.pem (node certificate + CA)
• jet-key.pem (private key)
• jet-ca.pem (ROOT CA)

I’ve copied jet-ca.pem to /etc/pki/ca-trust/source/anchors/ and done command update-ca-trust

Started opensearch

After:
securityadmin.sh -cd /opt/opensearch/config/opensearch-security/ -cacert jet-ca.pem -cert admin.pem -key admin-key.pem -icl -nhnv

I’ve got:

Security Admin v7
Will connect to localhost:9200 … done
ERR: An unexpected SSLHandshakeException occured: Received fatal alert: certificate_unknown
See Java high-level REST client - OpenSearch documentation for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
See Java high-level REST client - OpenSearch documentation for troubleshooting.
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:947)
at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:462)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

I installed Nginx to check certificate and it works fine…

I don’t get it why there is certificate_unknown error in opensearch…

Also flags -dg and -w for command securityadmin.sh don’t show any additional information.

[Gsmitt]

Hey @anteus

Is you Root CA in the trusted store? Did you make your own keystore or are you using JAVA default “cacerts”?

[anteus]

I’ve added jet-ca.pem to java default cacerts.

/opt/opensearch/jdk/bin/keytool -importcert -file jet-ca.pem -cacerts -alias DPFS

[Gsmitt]

Hey @anteus

@anteus

I’ve added jet-ca.pem to java default cacerts.

Understood… What do you get when executing this?

echo $JAVA_HOME

@anteus

ERR: An unexpected SSLHandshakeException occured: Received fatal alert: certificate_unknown

If the above command is correct then I think the issue you’re having is with the certificates.

[anteus]

It’s strange cause i did everything like in instructions:

# Create a private key for the admin certificate.
sudo openssl genrsa -out admin-key-temp.pem 2048

# Convert the private key to PKCS#8.
sudo openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
   
# Create the certficiate signing request (CSR). A common name (CN) of "A" is acceptable because this certificate is
# used for authenticating elevated access and is not tied to a host.
sudo openssl req -new -key admin-key.pem -subj "/CN=Admin" -out admin.csr

Except last step. I signed csr with my AD Certificate Service.
I checked certificates with Nginx and they are fine. I think i tried everything for last 5 days but opensearch doesn’t want to apply changes with my certs…

Difference is only with certificate extensions. Self-signed admin certificate has no extensions.

[Gsmitt]

hey,

For troubleshooting have you tried to just use the self-signed certs? If that results is the same issue then we could look at other configurations made. Just an idea.

Some else i did different then you , was I used my java keystore.

keytool -importcert -keystore  /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts -storepass changeit  -alias opensearch.domain.com -file /etc/opensearch/root-ca.pem

Also in my notes, I had to set java path, mine was.

OPENSEARCH_JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/

@anteus

Will connect to localhost:9200 … done

by chance did you create your certificates with localhost? I adjusted my opensearch.yml file.

network.host: opensearch.domain.com (i.e.192.168.1.100)

Once completed i executed the following.

./securityadmin.sh -h opensearch.domain.com   -cd /etc/opensearch/opensearch-security/ -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/admin.pem -key /etc/opensearch/admin-key.pem -icl -nhnv

What I found was if you do not use the -h opensearch.domain.com then I believe it defaults to localhost.

Hope that helps

[anteus]

hi.
There is my identity keystore:

There is my trust keystore:

OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk ./securityadmin.sh -cd /etc/opensearch/opensearch-security/ -h opensearch.dpfs3197.int -ks /etc/opensearch/identity.jks -ksalias admin -kspass changeit -kst JKS -ts /etc/opensearch/trust.jks -tspass changeit -tsalias jetca -tst JKS -icl -nhnv

Will connect to opensearch.dpfs3197.int:9200 … done
ERR: An unexpected SSLHandshakeException occured: Received fatal alert: certificate_unknown
See Java high-level REST client - OpenSearch documentation for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

[Gsmitt]

Hey @anteus

@anteus

Received fatal alert: certificate_unknown

I think you missing the part about the keystore. This would be where you store you certificate so Openseach can access them.

@anteus

See Java high-level REST client - OpenSearch documentation for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

You may want to read back over the documentation here…

• Configuring TLS certificates - OpenSearch documentation

EDIT: I just seen you have a keystore called trust.jks by chance did you configured it so Opensearch can find it?

As shown from that link above , in this section marked in red.

[anteus]

Hi. Looks like i found the solution.
Node certificates need to have both serverAuth and clientAuth set in the extended key usage field.

I added new template to Active Directory Certificate Services with this extensions.

After this i recreated new certificates with this template and opensearch command has worked without errors.

OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk ./securityadmin.sh -cd /etc/opensearch/opensearch-security/ -h opensearch.dpfs3197.int -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/admin.pem -key /etc/opensearch/admin-key.pem -icl -nhnv

Will connect to opensearch.dpfs3197.int:9200 … done
Connected as “CN=Admin”
OpenSearch Version: 2.7.0
Contacting opensearch cluster ‘opensearch’ and wait for YELLOW clusterstate …
Clustername: opensearch
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/opensearch/opensearch-security/
Will update ‘/config’ with /etc/opensearch/opensearch-security/config.yml
SUCC: Configuration for ‘config’ created or updated
Will update ‘/roles’ with /etc/opensearch/opensearch-security/roles.yml
SUCC: Configuration for ‘roles’ created or updated
Will update ‘/rolesmapping’ with /etc/opensearch/opensearch-security/roles_mapping.yml
SUCC: Configuration for ‘rolesmapping’ created or updated
Will update ‘/internalusers’ with /etc/opensearch/opensearch-security/internal_users.yml
SUCC: Configuration for ‘internalusers’ created or updated
Will update ‘/actiongroups’ with /etc/opensearch/opensearch-security/action_groups.yml
SUCC: Configuration for ‘actiongroups’ created or updated
Will update ‘/tenants’ with /etc/opensearch/opensearch-security/tenants.yml
SUCC: Configuration for ‘tenants’ created or updated
Will update ‘/nodesdn’ with /etc/opensearch/opensearch-security/nodes_dn.yml
SUCC: Configuration for ‘nodesdn’ created or updated
Will update ‘/whitelist’ with /etc/opensearch/opensearch-security/whitelist.yml
SUCC: Configuration for ‘whitelist’ created or updated
Will update ‘/audit’ with /etc/opensearch/opensearch-security/audit.yml
SUCC: Configuration for ‘audit’ created or updated
Will update ‘/allowlist’ with /etc/opensearch/opensearch-security/allowlist.yml
SUCC: Configuration for ‘allowlist’ created or updated
SUCC: Expected 10 config types for node {“updated_config_types”:[“allowlist”,“tenants”,“rolesmapping”,“nodesdn”,“audit”,“roles”,“whitelist”,“internalusers”,“actiongroups”,“config”],“updated_config_size”:10,“message”:null} is 10 ([“allowlist”,“tenants”,“rolesmapping”,“nodesdn”,“audit”,“roles”,“whitelist”,“internalusers”,“actiongroups”,“config”]) due to: null
Done with success

[Gsmitt]

Hey @anteus

Thanks for the update

[kris]

thanks for coming back to share the solution with the community @anteus!!