Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.4.0 (also tested with 2.9.0), running on GCP-backed Kubernetes (v1.26.5), installation done through Helm with ArgoCD.
Describe the issue:
Opensearch Cluster is deployed and configured properly to accept Client Authentication using SSL certificates. Testing with curl yields correct and expected results. However, Opensearch Dashboards, try as you might configuring the related opensearch.ssl settings, it will not authenticate using certificates to the cluster. This behavior is described as mTLS (Mutual TLS authentication between Kibana and Elasticsearch | Kibana Guide [8.10] | Elastic). This is NOT authenticating end-users to Dashboards using SSL Certificates. This is only getting Dashboards to authenticate using SSL Certificates instead of username/password (internal auth). As mentioned, authenticating using cert/keys via curl works correctly, it even picks up the username for Dashboards. However, Dashboards itself does not want to pay any mind.
Configuration:
Dashboards:
opensearch_dashboards.yml:
server:
ssl:
enabled: true
clientAuthentication: required
certificate: /usr/share/opensearch-dashboards/config/tls.crt
key: /usr/share/opensearch-dashboards/config/tls.key
certificateAuthorities: /usr/share/opensearch-dashboards/config/ca.crt
xsrf:
allowlist:
- "/_opendistro/_security/saml/acs"
opensearch_security:
cookie:
secure: true
allow_client_certificates: true
multitenancy:
enabled: false
tenants:
enable_global: true
enable_private: false
preferred:
- "Global"
auth:
multiple_auth_enabled : false
type:
- "saml"
saml:
extra_storage:
cookie_prefix: opensearch_security_authentication_saml
additional_cookies: 3
opensearch:
ssl:
alwaysPresentCertificate: true
certificate: /usr/share/opensearch-dashboards/config/tls.crt
key: /usr/share/opensearch-dashboards/config/tls.key
certificateAuthorities: /usr/share/opensearch-dashboards/config/ca.crt
verificationMode: "full"
Server/Custer:
{
"config" : {
"dynamic" : {
"filtered_alias_mode" : "warn",
"disable_rest_auth" : false,
"disable_intertransport_auth" : false,
"respect_request_indices_options" : false,
<trunc>
},
"authc" : {
"basic_internal_auth_domain" : {
"http_enabled" : true,
"transport_enabled" : true,
"order" : 0,
"http_authenticator" : {
"challenge" : false,
"type" : "basic",
"config" : { }
},
"authentication_backend" : {
"type" : "internal",
"config" : { }
}
},
"clientcert_auth_domain" : {
"http_enabled" : true,
"transport_enabled" : true,
"order" : 1,
"http_authenticator" : {
"challenge" : false,
"type" : "clientcert",
"config" : {
"username_attribute" : "cn"
}
},
"authentication_backend" : {
"type" : "noop",
"config" : { }
},
"description" : "Authenticate via SSL client certificates"
},
"saml_auth_domain" : {
"http_enabled" : true,
"transport_enabled" : true,
"order" : 2,
"http_authenticator" : {
"challenge" : true,
"type" : "saml",
"config" : {
"exchange_key" : "redacted",
"idp" : {
"entity_id" : "redacted",
"metadata_url" : "redacted"
},
"jwt" : {
"expiry" : "SESSION+7220"
},
"kibana_url" : "redacted",
"roles_key" : "Groups",
"sp" : {
"entity_id" : "redacted",
"forceAuthn" : false
}
}
},
"authentication_backend" : {
"type" : "noop",
"config" : { }
},
"description" : "SAML provider"
}
},
"authz" : { },
"auth_failure_listeners" : { },
"do_not_fail_on_forbidden" : false,
"multi_rolespan_enabled" : true,
"hosts_resolver_mode" : "ip-only",
"do_not_fail_on_forbidden_empty" : false
}
}
}
More cluster config:
plugins:
security:
nodes_dn:
[
"CN=opensearch-cluster",
]
authcz:
admin_dn:
[
"CN=opensearch-admin",
]
ssl:
transport:
enabled: true
pemcert_filepath: tls.crt
pemkey_filepath: tls.key
pemtrustedcas_filepath: ca.crt
enforce_hostname_verification: false
http:
enabled: true
pemcert_filepath: tls.crt
pemkey_filepath: tls.key
pemtrustedcas_filepath: ca.crt
clientauth_mode: REQUIRE
Relevant Logs or Screenshots:
Relevant output from Dashboards:
{"type":"log","@timestamp":"2023-09-12T16:06:49Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
{"type":"log","@timestamp":"2023-09-12T16:06:49Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2023-09-12T16:06:49Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}
{"type":"log","@timestamp":"2023-09-12T16:06:51Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
There are no audit logs available from the cluster, as if the request wasn’t coming cleanly through. This is not the case with curl, where the logs do not yield anything either, but the request is successful:
/certs $ curl --cacert dashboards/ca.crt --cert dashboards/tls.crt --key dashboards/tls.key https://opensearch-cluster:9200/_plugins/_security/authinfo?pretty
{
"user" : "User [name=opensearch-dashboards, backend_roles=[], requestedTenant=null]",
"user_name" : "opensearch-dashboards",
"user_requested_tenant" : null,
"remote_address" : "10.152.1.1:52414",
"backend_roles" : [ ],
"custom_attribute_names" : [ ],
"roles" : [
"readall_and_monitor",
"kibana_user",
"all_access"
],
"tenants" : {
"opensearch-dashboards" : true,
"global_tenant" : true
},
"principal" : "CN=opensearch-dashboards",
"peer_certificates" : "2",
"sso_logout_url" : null
}
/certs $