Opensearch security plugin enable error with securityadmin.sh

idan17 · January 24, 2024, 11:50am

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:
Im installing opensearch and opensearch dashboard using the corresponding helm charts. when trying to configure the opensearch security plugin,
after changing the configuration for opensearch.yml and the config.yml of the opensearch security config, im trying to apply the changes using the securityadmin.sh script as mentioned in the docs.
but when trying to, i get the error :
"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target "

i supplied the certificates (key, certificate, and root-ca) in the relevant places in the config files, and in PEM format, and this error keeps coming,
any help?

Mantas · January 24, 2024, 12:08pm

Hi @idan17,

Could you please share the command you are running?

Thanks,
mn

idan17 · January 24, 2024, 12:21pm

./securityadmin.sh
-f …/…/…/config/opensearch-security/config.yml
-icl -nhnv -cert /usr/share/opensearch/config/cert/cert.pem
-cacert /usr/share/opensearch/config/root-ca/root-ca.pem
-key /usr/share/opensearch/config/key/key.pem
-t config

and i will add that those certificates i mounted using the :
secretMounts option in the helm

idan17 · January 24, 2024, 12:50pm

ok now when i put all the CA chain in the certificate file i moved to another error, now the error is "
certificate_unknown" when running the admin script

Mantas · January 24, 2024, 1:03pm

Have you configured it to be your admin certificate?

For more details, see here: Configuring TLS certificates - OpenSearch Documentation

best,
mj

idan17 · January 24, 2024, 1:06pm

yes, i set the value of admin_dn to be as the value in the “subject” field of my certificate

idan17 · January 24, 2024, 6:02pm

im still getting the “certificate_unknown” error from the securityadmin script, is there a variable in the values of the helm chart where i need to specify my self signed certificates other than the ssl config in the opensearch.yml file ?,
and if there is or isn’t, is there a way i can confirm my certificates are suitable for the opensearch security plugin?,
i would appreciate any help because this is a bug that is holding my team for too long on an important project

GSmith · January 24, 2024, 8:38pm

It can be very confusing because there are 3 sets of certs:

transport - defined by the keys starting with plugins.security.ssl.transport in the opensearch.yaml file
REST/HTTP - defined by the keys starting with plugins.security.ssl.http in the opensearch.yaml file
admin - which are NOT defined the opensearch.yaml file at all but still need to be accessible.

When you shared the command you are trying to run, you included these items:

-cert /usr/share/opensearch/config/cert/cert.pem
-cacert /usr/share/opensearch/config/root-ca/root-ca.pem
-key /usr/share/opensearch/config/key/key.pem

Are those paths pointing to the admin certificate or to one of the other sets of certs (e.g. the transport or rest/http certs)?

In our deployments, we have the following extraVolumes defined and we point to the last set (admin-certs) when running the securityAdmin.sh command.

extraVolumes:

name: transport-certs
secret:
secretName: es-transport-tls-secret
name: rest-certs
secret:
secretName: es-rest-tls-secret
name: admin-certs
secret:
secretName: es-admin-tls-secret

idan17 · January 25, 2024, 7:20am

Hi @GSmith !,
first of all thank you for replying,
well yes, im using the same certificates for the 3 sets you specified ,
thats also because im working in a closed (air gapped) network and have access to those signed certificates (cert, key, and root-ca).
i also used the admin DN as the DN in those certs.
i mounted the certificates using the secretMounts option , and i know they are accessible in the pod because i tried to change the path when executing the command and got an error of not found. So i know i mounted the certificates, the thing is how can i know they are in the correct format and structure (meaning the correct ca chain) for the script to work.

Mantas · January 25, 2024, 10:25am

Have you checked the Generate an admin certificate here: Generating self-signed certificates - OpenSearch Documentation ?

idan17 · January 25, 2024, 10:53am

hi, i verified the certificates,
they work on my other deployments, and when changing it to .crt i can open it and see it is valid chain.
but still getting the error “certificate_unknown” when running the script,
i saw in the internet that it has to do with adding the certs to the truststore using keytool.
i tried to add to the default keystore of java, and it didn’t help, and when trying to add that to the opensearch.keystore i get an error of unrecognized type. how can i add my certs to the opensearch truststore and do i even need to?

idan17 · January 25, 2024, 11:49am

just updating:
i tried to do the same but now with my own keystore (which i know that works in another system), and with the option of -ks and -ts for the script, and im getting the exact same error of certificate_unknown.
does anyone have any ideas ?

Mantas · January 25, 2024, 12:27pm

Where do you execute the securityadmin.sh command (pod or externally)?

Could you also run the below and share the output:
openssl s_client -connect <OpenSearch_service>:<9200_or_node_port>

idan17 · January 25, 2024, 1:06pm

im doint kubectl exec to open bash terminal in my pod (im deploying using helm)
and then running the securityadmin.sh script from within.

upon running the openssl command, when pointing to my ingress i configured to this deployment,
im getting a long output containing the certificate, another data like subject, issuer and more about this certificate.
sorry i cannot pass the whole output since its in a closed confidential network

Mantas · January 25, 2024, 4:34pm

Could you try cert authentication without the securityadmin.sh and share if you are getting any errors, you can try with something as per below:

curl --cacert /usr/share/opensearch/config/root-ca/root-ca.pem --key /usr/share/opensearch/config/key/key.pem --cert /usr/share/opensearch/config/cert/cert.pem -XGET -k https://<OpenSearch:9200>/_plugins/_security/authinfo?pretty

thanks,
mj

idan17 · January 28, 2024, 7:07am

im getting bad gateway 502,
and in the logs of my pod im getting:

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: …

idan17 · January 28, 2024, 8:07am

i tried to uninstall (with PVC delete) both opensearch and opensearch dashboards, and now i have the same SSL record exception in my opensearch pod logs,
and my opensearch-dashboards is failing to run with error :
Startup probe failed: dial tcp …:5601 connect: connection refused

Mantas · January 29, 2024, 11:16am

Hi @idan17,

At this point, I would say we need to backtrack a bit and understand your setup a bit better.
Could you please share your opensearch.yml and opensearch_dashboards.yml the ones that were picked by the pods?

best,
mj

idan17 · January 31, 2024, 7:40am

ok, supplying my configuration files i add to the relevant values.yml for my helm charts:

opensearch.yml:

cluster.name: opensearch-cluster

network.host: 0.0.0.0

plugins:
  security:
    ssl:
      transport:
        pemcert_filepath: cert/<my-full-ca-chain>.pem
        pemkey_filepath: key/key.pem
        pemtrustedcas_filepath: root-ca/root-ca.pem # my root certificate of CA chain
        enforce_hostname_verification: false
      http:
        pemcert_filepath: cert/<my-full-ca-chain>.pem
        pemkey_filepath: key/key.pem
        pemtrustedcas_filepath: root-ca/root-ca.pem # my root certificate of CA chain
    allow_unsafe_democertificates: true
    allow_default_init_securityindex: true
    authcz:
      admin_dn:
        - CN=XX,OU=XX,O=XX,C=XX
        - CN=kirk,OU=client,O=client,L=test,C=de
    nodes_dn:
      - CN=XX,OU=XX,O=XX,C=XX
    audit.type: internal_opensearch
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
    restapi:
      roles_enabled: ["all_access", "security_rest_api_access"]
    system_indices:
      enabled: true
      indices:
        [
          ".opendistro-alerting-config",
          ".opendistro-alerting-alert*",
          ".opendistro-anomaly-results*",
          ".opendistro-anomaly-detector*",
          ".opendistro-anomaly-checkpoints",
          ".opendistro-anomaly-detection-state",
          ".opendistro-reports-*",
          ".opendistro-notifications-*",
          ".opendistro-notebooks",
          ".opendistro-asynchronous-search-response*",
        ]

config.yml for security plugin:

config:
  openid_connect_idp:
    enable_ssl: true
    pemtrustedcas_filepath: /usr/share/opensearch/config/root-ca/root-ca.pem
    pemcert_filepath: /usr/share/opensearch/config/cert/<my-full-ca-chain>.pem
    pemkey_filepath: /usr/share/opensearch/config/key/key.pem
  dynamic:
    basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
    openid_auth_domain:
        http_enabled: true
        enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
            type: "openid"
            challenge: false
            config:
                enable_ssl: true
                subject_key: preferred_username
                roles_key: roles
                openid_connect_url: "https://<keycloak_address>/realms/master/.well-known/openid-configuration"
                openid_connect_idp:
                    enable_ssl: true
                    pemtrustedcas_filepath: /usr/share/opensearch/config/root-ca/root-ca.pem
        authentication_backend:
            type: noop

opensearch_dashboards.yml:

opensearch:
    hosts: ["https://opensearch-cluster.master.opensearch.svc:9200"]
    ssl:
        verificationMode: none
        certificateAuthorities: /usr/share/opensearch-dashboards/config/certs/cert/<my-full-ca-chain>.pem
opensearch_security:
    auth:
        type: ["basicauth","openid"]
        multiple_auth_enabled: true
    openid:
        connect_url: "https://<keycloak_address>/realms/master/.well-known/openid-configuration"
        client_id: <id>
        client_secret: <secret>
        scope: "openid profile email"
server:
  ssl:
    enabled: true
    certificate: /usr/share/opensearch-dashboards/config/certs/cert/>my-full-ca-chain>.pem
    key: /usr/share/opensearch-dashboards/config/certs/key/key.pem

just adding that im using singleNode , and the dashboards and the opensearch are residing on different namespaces in my k3s cluster.

Mantas · January 31, 2024, 1:44pm

Could you please add plugins.security.ssl.http.enabled: true in your opensearch.yml, test it, and let me know?

Best,
mj

Topic		Replies	Views
Securityadmin.sh unable to find valid certification path to requested target error Security configure , security-issue	2	731	November 1, 2023
Error while executing securityadmin.sh in OS 1.2.4 Security security-issue	19	794	October 26, 2023
Securityadmin.sh not working due to admin certificate issue Security	4	36	November 5, 2024
Opensearch security plugin with certificate chain Security	10	1616	June 12, 2024
Securityadmin errors with certificates Security security-issue	13	2007	May 15, 2023

Opensearch security plugin enable error with securityadmin.sh

Related topics