Openid auth not working - unable to get issuer certificate

Security
1

Hi,
Using openid_authentication for authentication with keycloak for Opensearch and Opensearch-dashboards -

SSL is enabled for Keycloak server. Its server certificate is signed by a chain of certs like this →
(where Root CA is usually a self-signed well-known cert signing authority)

Certificate[1]:
Owner: CN=keycloak.io, C=IN
Issuer: CN=Signing CA2, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com

Certificate[2]:
Owner: CN=Signing CA2, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
Issuer: CN=Signing CA1, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com

Certificate[3]:
Owner: CN=Signing CA1, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
Issuer: CN=Root CA

Certificate[4]:
Owner: CN=Root CA
Issuer: CN=Root CA

Under opensearch and opensearch-dashboards openid configs, rootca of keycloak server is configured at -
pemtrustedcas_filepath : under openid_auth_domain (for opensearch)
opensearch_security.openid.root_ca : (for opensearch-dashboards)

Issue:
When we configure the complete chain of certs i.e #2, # 3 and #4 for keycloak rootca in opensearch and opensearch-dashboards , opensearch-dashboards works fine.
But when we configure only certs #2 and #3, opensearch-dashboards fails with this error in logs -

{"log":{"message":"{ Error: unable to get issuer certificate\n    at TLSSocket.onConnectSecure (_tls_wrap.js:1088:34)\n    at TLSSocket.emit (events.js:198:13)\n    at TLSSocket._finishInit (_tls_wrap.js:666:8)\n  code: 'UNABLE_TO_GET_ISSUER_CERT',\n  trace:\n   [ { method: 'GET',\n       url:\n        'https://keycloak.io/access/realms/elk/.well-known/openid-configuration' } ],\n  isBoom: true,\n  isServer: true,\n  data: null,\n  output:\n   { statusCode: 502,\n     payload:\n      { message: 'Client request error: unable to get issuer certificate',\n        statusCode: 502,\n        error: 'Bad Gateway' },\n     headers: {} } }"},"extension":{"type":"log","tags":["error","plugins","opendistroSecurityKibana"],"pid":175},"type":"log","level":"info","timezone":"UTC","time":"2022-08-05T07:27:23Z"}
{"log":{"message":"Detected an unhandled Promise rejection.\nError: Failed when trying to obtain the endpoints from your IdP"},"extension":{"type":"log","tags":["warning","environment"],"pid":175},"type":"log","level":"info","timezone":"UTC","time":"2022-08-05T07:27:23Z"}

Whereas opensearch works fine i.e. if we issue a REST call to opensearch passing the authentication bearer token from keycloak, it works all fine.

#Fetch token from keycloak
curl --cacert /etc/keycloakRootCaPem  --data "client_id=<clientid>&client_secret=<client-secret>&username=<user>&password=<[pwd>&grant_type=password" https://keycloak.io/access/realms/elk/protocol/openid-connect/token
# Using token in REST call
curl -k https://elasticsearch:9200/_cat/indices -H "Authorization: Bearer <obtained access token> "

Why is the behaviour such that - opensearch-dashboards needs the entire chain of certs in its truststore while opensearch does not need it and works fine with just immediate signing authorities?

Thanks!

2

@shivani What’s your OpenSearch version?
Could you share your config.yml file?

3

Currently, I am facing the issue on elasticsearch and kibana with opendistro-security kibana plugin - (but similar behavior is seen with opensearch 1.3.2 as well)

Version details:
Elasticsearch, Kibana - 7.10.2
Opendistro security - 1.13.1.0

Elasticsearch security configfile:
/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml

_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    kibana:
       multitenancy_enabled: false
       server_username: kibanaserver
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: ".+"
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: "basic"
          challenge: false   
          config: {}
        authentication_backend:
          type: "intern"
          config: {}
      openid_auth_domain:
        http_enabled: true             
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: https://keycloak.io/access/realms/elk/.well-known/openid-configuration
            openid_connect_idp:
              enable_ssl: true
              verify_hostnames: false
              trust_all: false      
              pemtrustedcas_filepath: "/etc/keycloakRootCaPem"
        authentication_backend:
            type: noop

Kibana config file:
/usr/share/kibana/config/kibana.yml

server.name: kibana
server.customResponseHeaders: { "X-Frame-Options": "DENY" }
#opendistro_security.cookie.secure: true
opendistro_security.multitenancy.enabled: false
# Whitelist basic headers and multi tenancy header
##elasticsearch.requestHeadersWhitelist: [ "Authorization", "securitytenant", "x-forwarded-for", "x-proxy-user", "x-proxy-roles" ]
opendistro_security.openid.client_id: elk-kibana
opendistro_security.auth.type: "openid"
opendistro_security.openid.connect_url: "https://keycloak.io/access/realms/elk/.well-known/openid-configuration"
opendistro_security.openid.client_id: elk-kibana
opendistro_security.openid.client_secret: kQXCx0mxMEtzuEZaAh0XMnTXfE2BHchl
opendistro_security.openid.header: "Authorization"
opendistro_security.openid.base_redirect_url: "https://<ip>/kibana"   
opendistro_security.openid.root_ca: "/etc/keycloakRootCaPem"

In both ES and Kibana → /etc/keycloakRootCaPem is a pem file containing 2 certificates (certs #2 and #3 from above mentioned chain) → 

Certificate[2]:
Owner: CN=Signing CA2, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
Issuer: CN=Signing CA1, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com

Certificate[3]:
Owner: CN=Signing CA1, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
Issuer: CN=Root CA
4

@shivani I’ve tested your scenario and reproduced the issue. It is also present in version 2.2.0.
This worked like that since version 1.0.0.

I’d suggest reporting a feature request to implement full certificate chain verification in OpenSearch’s GitHub.

Please share the link to that feature request once is opened.

5

Hi @pablo
Thanks for reproducing. But do you mean that →

  • Opensearch-dashboards is working as expected and feature request is needed for Opensearch?

  • Like Opensearch-Dashboards, even Opensearch should have not allowed the interaction with Keycloak without having the entire certificate chain in its truststore?

6

@shivani I’d say that from a security point OpenSearch Dashboards plugin has a better solution.
I’d prefer to have it implemented in the OpenSearch security plugin.