Not able to setup client authentication with certificates

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.11

Describe the issue:

I’m trying to setup client authentication with certificates, but I cannot get it to work.
E.g the command

curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_opendistro/_security/api/ssl/certs?pretty'

fails with 403. Therefore I assume I have to use the admin_dn and I try with the admin user certificates

curl -XGET -k 'https://localhost:9200/_opendistro/_security/api/ssl/certs?pretty' --cert opensearch-admin.pem --key opensearch-admin-key.pem --cacert root-ca.pem

But I get 401.
The admin certificates are ok, I can use them locally in the node with the securityadmin.sh scripts. But using the same certificates with curl does not work.

How can I troubleshoot with the certificate authentication fails?

Configuration:

This is a deployment in K8s, and the nginx ingress is configured to pass the client certificate to the nodes.

I configured the client authentication to include

              clientcert_auth_domain:
                description: "Authenticate via SSL client certificates"
                http_enabled: true
                transport_enabled: true
                order: 1
                http_authenticator:
                  type: clientcert
                  challenge: false
                  config:
                    username_attribute: cn #optional, if omitted DN becomes username
                authentication_backend:
                  type: noop

Relevant Logs or Screenshots:

I enabled audit debug and I see the failed login, with user ‘none’, but no more reason.

[2025-01-27T16:13:44,387][INFO ][o.o.s.a.s.DebugSink      ] [opensearch-master-2] AUDIT_LOG: {
  "audit_cluster_name" : "opensearch",
  "audit_rest_request_params" : {
    "pretty" : ""
  },
  "audit_node_name" : "opensearch-master-2",
  "audit_rest_request_method" : "GET",
  "audit_category" : "FAILED_LOGIN",
  "audit_request_origin" : "REST",
  "audit_node_id" : "NjeCF8q4S-OnIuYh",
  "audit_request_layer" : "REST",
  "audit_rest_request_path" : "/_plugins/_security/api/securityconfig",
  "@timestamp" : "2025-01-27T16:13:44.387+00:00",
  "audit_request_effective_user_is_admin" : false,
  "audit_format_version" : 4,
  "audit_request_remote_address" : "10.42.1.219",
  "audit_node_host_address" : "10.42.2.120",
  "audit_rest_request_headers" : {
    "X-Request-ID" : [
      "97c94e314b031a25aa7d33c5bb374d23"
    ],
    "X-Forwarded-Host" : [
      "opensearch-dev.k8s.lab.se"
    ],
    "X-Forwarded-Proto" : [
      "https"
    ],
    "X-Forwarded-For" : [
      "10.56.235.100"
    ],
    "Host" : [
      "opensearch-dev.k8s.lab.se"
    ],
    "X-Forwarded-Port" : [
      "443"
    ],
    "X-Forwarded-Scheme" : [
      "https"
    ],
    "X-Real-IP" : [
      "10.56.235.100"
    ],
    "accept" : [
      "*/*"
    ],
    "user-agent" : [
      "curl/7.76.1"
    ],
    "X-Scheme" : [
      "https"
    ]
  },
  "audit_request_effective_user" : "<NONE>",
  "audit_node_host_name" : "10.42.2.120"
2

Hi @Julio,

For admin certs to work you need them to be set in opensearch.yml plugins.security.authcz.admin_dn:

i.e:


plugins.security.authcz.admin_dn: ['CN=kirk,OU=client,O=client,L=test,C=de']

more here: Configuring TLS certificates - OpenSearch Documentation

You might as well be interested in checking out:

best,
mj

3

Hi,

Yes, I had configured
plugins.security.authcz.admin_dn:

with the value of the subject in the certificate, and it works because I can run securityadmin.sh script with the same certificate.

However, I cannot succeed with the curl command. I suspect it could be that the Ingress is replacing the client certificate, though I have configured the Ingress endpoint to not do so.

I’m missing debug information about why the authentication failed, e.g. more details about why the certificate authentication failed, what is the subject in the certificate and why it could not be matched.

I was expecting audit debug logs to provide such information.

4

Have you set the below to true?

plugins.security.ssl_cert_reload_enabled: true

Do you mind sharing your opensearch.yml?

best,
mj

5

Hi,

I have set plugins.security.ssl_cert_reload_enabled: true but it did not help.
I have also set plugins.security.restapi.admin.enabled: true but it did not work either.

My opensearch.yml is

    cluster.name: opensearch-master

    # Bind to all interfaces because we don't know what IP address Docker will assign to us.
    network.host: 0.0.0.0

    # Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
    # Implicitly done if ".singleNode" is set to "true".
    # discovery.type: single-node

    # Start OpenSearch Security Demo Configuration
    # WARNING: revise all the lines below before you go into production
        
    plugins:
      security:
        disabled: false
        ssl:
          transport:
            enabled: true
            pemcert_filepath: node-certs/node.crt
            pemkey_filepath: node-certs/node.key
            pemtrustedcas_filepath: root-ca-cert/ca.crt
            truststore_filepath: /usr/share/opensearch/config/ca-certs/cacerts.jks
            truststore_password: ${TRUSTSTORE_PASSWORD}
            enforce_hostname_verification: false
          http:
            enabled: true
            pemcert_filepath: node-certs/node.crt
            pemkey_filepath: node-certs/node.key
            pemtrustedcas_filepath: root-ca-cert/ca.crt
            truststore_filepath: /usr/share/opensearch/config/ca-certs/cacerts.jks
            truststore_password: ${TRUSTSTORE_PASSWORD}
            clientauth_mode: OPTIONAL
        ssl_cert_reload_enabled: true
        allow_unsafe_democertificates: false
        allow_default_init_securityindex: true
        
        authcz:
          admin_dn:
            - 'CN=kirk,OU=DevOps,O=Example,C=SE'

        nodes_dn:
          - 'CN=opensearch-dev-ccr.k8s.lab.se,OU=DevOps,O=Example'
          - 'CN=opensearch-master-0'
          - 'CN=opensearch-master-1'
          - 'CN=opensearch-master-2'
          - 'CN=opensearch-dev3-ccr.nms-sto-blue.k8s.lab.se,OU=DevOps,O=Example'

        audit.type: debug
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true
        restapi:
          roles_enabled: ["all_access", "security_rest_api_access"]
          admin:
            enabled: true
        system_indices:
          enabled: true
          indices:
            [
              ".opendistro-alerting-config",
              ".opendistro-alerting-alert*",
              ".opendistro-anomaly-results*",
              ".opendistro-anomaly-detector*",
              ".opendistro-anomaly-checkpoints",
              ".opendistro-anomaly-detection-state",
              ".opendistro-reports-*",
              ".opendistro-notifications-*",
              ".opendistro-notebooks",
              ".opendistro-asynchronous-search-response*",
            ]
6

Have you rebooted your cluster after updating your config files?

Best,
Mantas

7

Sure, I did after every update.

Anyway, I got it work. The issue I had is that the K8s Ingress is doing something with the certificates I specify in the curl command, thus not using the certificates for authentication.
I created a load balancer service that exposes an FQDN, and with that load balancer I am able to use the admin cert and execute REST API actions restricted to the admin user.
I do not know why it does not work with Nginx or Traefik ingress, so far I will bypass them.

1 Like