Client certificate authentication - between Dashboard and Elastic nodes failing

Running OpenSeach 1.2.4 and OpenSearch Dashboards1.2.0 on separate hosts -
3 x OpenSearch and 1 x Dashboard.

For security, I want to make enable client certificate authentication mandatory on the OpenSearch nodes.
Following Client certificate authentication - OpenSearch documentation
I set:

plugins.security.ssl.http.clientauth_mode: "REQUIRE"

The OpenSearch nodes are able to communicate, and I can make API calls using client certs and the admin username/password.
The Dashboard can’t connect.
Has anyone got this working?

These are the errors in the OpenSearch log (10.19.192.2 is the IP of the Dashboard):

caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.19.192.4:9200, remoteAddress=/10.19.192.2:59484}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
...
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain

Error on Dashboard host:

{"type":"log","@timestamp":"2022-01-26T10:39:31Z","tags":["error","opensearch","opendistro_security"],"pid":11115,"message":"Request error, retrying\nGET https://elastic-03.opensearch.example.com:9200/_plugins/_security/authinfo => write EPROTO 140278061963136:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 42\n"}

@michaelk Could you share your opensearch_dashboards.yml file?

@pablo Contents of /usr/share/opensearch-dashboards/config/opensearch_dashboards.yml

server.port: 5601
server.host: "0.0.0.0"
opensearch.hosts: ["https://elastic-01.ourdomain.com:9200","https://elastic-03.ourdomain.com:9200","https://elastic-02.ourdomain.com:9200"]
opensearch.ssl.verificationMode: full
opensearch.username: "kibanaserver"
opensearch.password: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]
server.ssl.enabled: true
server.ssl.certificate: "/usr/share/opensearch-dashboards/config/node.pem"
server.ssl.key: "/usr/share/opensearch-dashboards/config/node.key"
opensearch.ssl.certificateAuthorities: [ "/usr/share/opensearch-dashboards/config/root-ca.pem", "/usr/share/opensearch-dashboards/config/intermediate-ca.pem" ]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: false
opensearch_security.multitenancy.tenants.preferred: ["Global"]
opensearch_security.multitenancy.enable_filter: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
opensearch_security.cookie.secure: true
opensearch_security.basicauth.forbidden_usernames: ["kibanaserver", "filebeat"]

logging.dest: "/var/log/dashboard/dashboard.log"
logging.quiet: true```

@michaelk What is the order in config.yml?

@michaelk Does it work with opensearch.ssl.verificationMode: none ?

@pablo below is my config.yml
Note, I also tried setting http_enabled and transport_enabled to true in clientcert_auth_domain:, but didn’t make any difference

---

# This is the main OpenSearch Security configuration file where authentication
# and authorization is defined.
#
# You need to configure at least one authentication domain in the authc of this file.
# An authentication domain is responsible for extracting the user credentials from
# the request and for validating them against an authentication backend like Active Directory for example.
#
# If more than one authentication domain is configured the first one which succeeds wins.
# If all authentication domains fail then the request is unauthenticated.
# In this case an exception is thrown and/or the HTTP status is set to 401.
#
# After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect
# the roles from a given backend for the authenticated user.
#
# Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.
#        http_enabled: true
#        transport_enabled: true
#
# For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to
# find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.
# If none can be found the user will be authenticated as an "anonymous" user. This user has always the username "anonymous"
# and one role named "anonymous_backendrole".
# If you enable anonymous authentication all HTTP authenticators will not challenge.
#
#
# Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like "proxy" or "clientcert"
# first and the challenging one last.
# Because it's not possible to challenge a client with two different authentication methods (for example
# Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation
# by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.
#
# Default value of the challenge flag is true.
#
#
# HTTP
#   basic (challenging)
#   proxy (not challenging, needs xff)
#   kerberos (challenging)
#   clientcert (not challenging, needs https)
#   jwt (not challenging)
#   host (not challenging) #DEPRECATED, will be removed in a future version.
#                          host based authentication is configurable in roles_mapping

# Authc
#   internal
#   noop
#   ldap

# Authz
#   ldap
#   noop



_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
    # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
    # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
    #filtered_alias_mode: warn
    #do_not_fail_on_forbidden: false
    kibana:
    # Kibana multitenancy
      multitenancy_enabled: false
      server_username: kibanaserver
      index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        #internalProxies: '.*' # trust all internal proxies, regex pattern
        #remoteIpHeader:  'x-forwarded-for'
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:
      kerberos_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 6
        http_authenticator:
          type: kerberos
          challenge: true
          config:
            # If true a lot of kerberos/security related debugging output will be logged to standard out
            krb_debug: false
            # If true then the realm will be stripped from the user name
            strip_realm_from_principal: true
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        description: "Authenticate via proxy"
        http_enabled: false
        transport_enabled: false
        order: 3
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      jwt_auth_domain:
        description: "Authenticate via Json Web Token"
        http_enabled: false
        transport_enabled: false
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            roles_key: null
            subject_key: null
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(sAMAccountName={0})'
            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: null
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
            rolesearch: '(member={0})'
            # Specify the name of the attribute which value should be substituted with {2} above
            userroleattribute: null
            # Roles as an attribute of the user entry
            userrolename: disabled
            #userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(uid={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            #skip_users:
            #  - 'cn=Michael Jackson,ou*people,o=TEST'
            #  - '/\S*/'
      roles_from_another_ldap:
        description: "Authorize via another Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap
          #config goes here ...
  #    auth_failure_listeners:
  #      ip_rate_limiting:
  #        type: ip
  #        allowed_tries: 10
  #        time_window_seconds: 3600
  #        block_expiry_seconds: 600
  #        max_blocked_clients: 100000
  #        max_tracked_clients: 100000
  #      internal_authentication_backend_limiting:
  #        type: username
  #        authentication_backend: intern
  #        allowed_tries: 10
  #        time_window_seconds: 3600
  #        block_expiry_seconds: 600
  #        max_blocked_clients: 100000
  #        max_tracked_clients: 100000

@pablo I tried setting opensearch.ssl.verificationMode: none, but it didn’t make any difference

I think I might have fixed it.
I copied admin.key and admin.pem client certs over from one of the Elastic nodes to the Dashboard, and added these lines to opensearch_dashboards.yml

opensearch.ssl.certificate: "/usr/share/opensearch-dashboards/config/admin.pem"
opensearch.ssl.key: "/usr/share/opensearch-dashboards/config/admin.key"

@michaelk When you cat node.pem and admin.pem, do both files contains two certs or one?

@pablo node.pem and admin.pem contain only one cert

I thought it was working, but is still gives the same SSL error!

It works when I configure using API calls to one of the Elastic nodes (e.g. using Terraform Elasticsearch provider add users etc).

Error in dashboard log is:

{"type":"log","@timestamp":"2022-01-28T10:29:14Z","tags":["error","opensearch","opendistro_security"],"pid":13606,"message":"Request error, retrying\nGET https://elastic-03.mydomain.com:9200/_plugins/_security/authinfo => write EPROTO 139995432486784:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 42\n"}
{"type":"log","@timestamp":"2022-01-28T10:29:15Z","tags":["error","plugins","securityDashboards"],"pid":13606,"message":"Failed authentication: Error: No Living connections"}

I have this in config.yml

      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop

This is my opensearch_dashboard.yml

server.port: 5601
server.host: "0.0.0.0"
opensearch.hosts: ["https://elastic-02.mydomain.com:9200","https://elastic-01mydomain.com:9200","https://elastic-03mydomain.com:9200"]
opensearch.ssl.verificationMode: full
opensearch.username: "kibanaserver"
opensearch.password: "p7Ak6eyq_vtMQsuw_CJ5w7E1pmbGcjnS4NrkbE7I"
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]
server.ssl.enabled: true
server.ssl.certificate: "/usr/share/opensearch-dashboards/config/node.pem"
server.ssl.key: "/usr/share/opensearch-dashboards/config/node.key"
opensearch.ssl.certificate: "/usr/share/opensearch-dashboards/config/admin.pem"
opensearch.ssl.key: "/usr/share/opensearch-dashboards/config/admin.key"
opensearch.ssl.certificateAuthorities: [ "/usr/share/opensearch-dashboards/config/root-ca.pem", "/usr/share/opensearch-dashboards/config/intermediate-ca.pem" ]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: false
opensearch_security.multitenancy.tenants.preferred: ["Global"]
opensearch_security.multitenancy.enable_filter: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
opensearch_security.cookie.secure: true
opensearch_security.basicauth.forbidden_usernames: ["kibanaserver", "filebeat"]

@michaelk Found this issue report in OpenSearch Github (https://github.com/opensearch-project/security/issues/1470)

It looks like not a supported option yet.