OpenSearch Dashboards 401 Unauthorized using OIDC with Keycloak

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 2.2.1
OpenSearch Dashboards: 2.2.1
Keycloak: 20.0.0
Ubuntu-20.04

Describe the issue:
Set up OpenSearch, OpenSearch Dashboards and Keycloak inside of a kubernetes cluster. Opensearch, and Dashboards run in one cluster, whereas Keycloak run on a different cluster. When trying to access Opensearch Dashboards I get redirected to Keycloaks login page. After inputting the credentials I get redirected back to opensearch dashboards, but am meet with a white page saying:

{ā€œstatusCodeā€:401,ā€œerrorā€:ā€œUnauthorizedā€,ā€œmessageā€:ā€œUnauthorizedā€}

The url in the browser at this point looks like this:

http://dashboard:5601/auth/openid/login?state=9NwaJ_W4L3l3B9PA3_FOoR&session_state=0cf3549b-c0bd-4fc0-aacc-f3d13f69b507&code=e3d571b5-05e4-4ce7-b9fb-ccb458eed5aa.0cf3549b-c0bd-4fc0-aacc-f3d13f69b507.160a0333-c129-46ad-ba92-cbf5efe95bd0

Keycloak has no logs (which usually means there is no error). It seems I was Authenticated with Keycloak but not Authorized. OpenSearch Dashboards has a generic 401 exception in its logs. The only log in OpenSearch is the following:

No ā€˜Basic Authorizationā€™ header, send 401 and ā€˜WWW-Authenticate Basicā€™

Configuration:

The configurations are loaded in using Helm Charts. The following are the relevant configurations for OpenSearch and OpenSearch Dashboards:

image:
  repository: opensearch-dashboards
  pullPolicy: Never
  tag: latest

config:
  opensearch_dashboards.yml: |
    opensearch.hosts: [http://opensearch-master-cluster:9200]
    opensearch.ssl.verificationMode: none
    opensearch.username: admin
    opensearch.password: admin
    opensearch.requestHeadersWhitelist: [Authorization, securitytenant]
    
    server.ssl.enabled: false

    opensearch_security.openid.base_redirect_url: 'http://dashboards:5601/'
    opensearch_security.auth.type: "openid"
    opensearch_security.openid.connect_url: "https://keycloak.example/realms/opensearch/.well-known/openid-configuration"
    opensearch_security.openid.client_id: "opensearch"
    opensearch_security.openid.client_secret: REDACTED
    opensearch_security.openid.scope: openid profile email
    opensearch_security.openid.header: "Authorization"

    opensearch_security.multitenancy.enable_filter: false
    opensearch_security.multitenancy.enabled: true
    opensearch_security.multitenancy.tenants.enable_global: true
    opensearch_security.multitenancy.tenants.enable_private: true

    opensearch_security.multitenancy.tenants.preferred: [Private, Global]
    opensearch_security.readonly_mode.roles: [kibana_read_only]

    opensearch_security.cookie.secure: true

    server.host: 'http://dashboards:5601'

sysctlInit:
  enabled: true

sysctlVmMaxMapCount: 262144

securityConfig:
  enabled: true
  config:
    dataComplete: false
    data:
      config.yml: |
        _meta:
          type: "config"
          config_version: 2
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: 0
                http_authenticator:
                  type: basic
                  challenge: false
                authentication_backend:
                  type: internal
              openid_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: 1
                http_authenticator:
                  type: openid
                  challenge: false
                  config:
                    openid_connect_idp:
                      enable_ssl: true
                      verify_hostnames: false  
                    roles_key: roles
                    subject_key: preferred_username
                    openid_connect_url: "https://keycloak.example/realms/opensearch/.well-known/openid-configuration"
                authentication_backend:
                  type: noop
            authz: {}  
  
  opensearch.yml: |
    cluster.name: opensearch-cluster
    logger.level: debug
    network.host: 0.0.0.0
    plugins:
      security:
        ssl:
          transport:
            enabled: true
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
            enforce_hostname_verification: false
          http:
            enabled: true
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
        allow_unsafe_democertificates: true
        allow_default_init_securityindex: true
        authcz:
          admin_dn:
            - CN=kirk,OU=client,O=client,L=test,C=de
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true
        restapi:
          roles_enabled: ["all_access", "security_rest_api_access"]
        system_indices:
          enabled: true
          indices:
            [
              ".opendistro-alerting-config",
              ".opendistro-alerting-alert*",
              ".opendistro-anomaly-results*",
              ".opendistro-anomaly-detector*",
              ".opendistro-anomaly-checkpoints",
              ".opendistro-anomaly-detection-state",
              ".opendistro-reports-*",
              ".opendistro-notifications-*",
              ".opendistro-notebooks",
              ".opendistro-asynchronous-search-response*",
            ]
    ######## End OpenSearch Security Demo Configuration ########

Relevant Logs or Screenshots:

Keycloak Role Mappings:

@Ditlevsen In config.yml you have the following IDP URL.

openid_connect_url: "https://qa.idp.supplier-overview.kmd.dk/realms/opensearch/.well-known/openid-configuration"

However, in opensearch_dashboards.yml the below.

opensearch_security.openid.connect_url: "https://keycloak.example/realms/opensearch/.well-known/openid-configuration"

sorry this is a typo mistake, was trying to obfuscate my domain names with keycloak.example since the actual domain name was not relevant for my question, much in the same way as i REDACTED the client secret.

@Ditlevsen The config seems to be fine. Whatā€™s the test user? Did you assigned realm roles or groups to the keycloakā€™s user?

I have tried two things. On Realms ā†’ Opensearch ā†’ Clients ā†’ Opensearch ā†’ Client Scope ā†’ opensearch_dedicated I have created a User Realm Role Mapper as shown in the picture from my post. Under Realms ā†’ Opensearch ā†’ Clients ā†’ Opensearch ā†’ Roles I have created the admin, all_access and kibanaserver backend roles. From what I understand with the default roles.yml and role_mappings.yml file, if the user has the backend role admin it should have all access. Finally, under Realms ā†’ Opensearch ā†’ Users I have created the dit user, for which I have added the realm roles mentioned before.

Edit forgot to tag you @pablo

@Ditlevsen Try this procedure. Run these steps in the OpenSearch node.

1. RESULT=$(curl -k --noproxy '*' -d 'client_id=<client_id>' -d 'username=admin' -d 'password=<password>' -d 'grant_type=password' -d 'client_secret=<client_secret>' -d 'scope=openid' 'https://<keycloak_IP_or_FQDN>:8443/realms/<realm>/protocol/openid-connect/token')

2. TOKEN=$(echo $RESULT | sed 's/.*access_token":"\([^"]*\).*/\1/')

3. curl --insecure -H "Authorization: Bearer $TOKEN" https://localhost:9200

In the logs of the OpenSearch you should see the JWT.

You can check with jwt.io if the roles are assigned.

@pablo This I have already done! In my realm Opensearch I went to users, created a user, went to role mapping and added the admin, all_access and kibanaserver roles.

@Ditlevsen Iā€™ve noticed that, just edited my answer.

@pablo I tried running your commands, I am assuming you meant run these commands from inside the OpenSearch Dashboards container. I did receive an access_token from Keycloak with the following data decoded from jwt.io:

{
  "exp": 1674587633,
  "iat": 1674587333,
  "jti": "REDACTED",
  "iss": "REDACTED",
  "aud": "account",
  "sub": "REDACTED",
  "typ": "Bearer",
  "azp": "opensearch",
  "session_state": "89e1c750-6cb6-4ecb-a59f-4e2f5d1073a1",
  "acr": "1",
  "realm_access": {
    "roles": [
      "default-roles-opensearch",
      "offline_access",
      "admin",
      "uma_authorization",
      "all_access",
      "kibanauser"
    ]
  },
  "resource_access": {
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "openid profile email",
  "sid": "REDACTED",
  "email_verified": false,
  "roles": [
    "default-roles-opensearch",
    "offline_access",
    "admin",
    "uma_authorization",
    "all_access",
    "kibanauser",
    "manage-account",
    "manage-account-links",
    "view-profile"
  ],
  "preferred_username": "dit",
  "given_name": "",
  "family_name": ""
}

I do not see the token anywhere in the logs of either OpenSearch Dashboards or Opensearch. Instead I got the token from Echoing the TOKEN variable after performing the operations.

The only log I get from Opensearch is the following:

The line ā€œNo ā€˜Basic Authorizationā€™ header, send 401 and ā€˜WWW-Authenticate Basicā€™ā€ is the only thing that happens when I run the third curl command from your list. It seems to be the same behavior as when I use the browser. I assume this is because it first attempts basic auth, which fails before it then moves on to OIDC as per the configuration ordering.

Inside the Opensearch Dashboards container the only result I get from running the third curl command is an ā€œUnauthorizedā€ message.

When looking at the actual logs of Opensearch Dashboards, this is what comes up:

{
    "type": "log",
    "@timestamp": "2023-01-25T10:05:28Z",
    "tags": [
        "error",
        "plugins",
        "securityDashboards"
    ],
    "pid": 1,
    "message": "OpenId authentication failed: Error: Authentication Exception"
}

@pablo Another thing I have noticed, not sure if it is helpful. When I exec into an OpenSearch container I can see that config.yml and opensearch.yml are both configured correctly, i.e. the values they have are the ones I set in the helm chart. However, if I remove these files completely from the helm chart and redeploy, thereby using the default config.yml and opensearch.yml that comes with the OpenSearch image, I still get the exact same error as before. Unauthorized after logging in with my keycloak user. Only log in OpenSearch being ā€œNo ā€˜Basic Authorizationā€™ header, send 401 and ā€˜WWW-Authenticate Basicā€™ā€.

Hello, I am also facing the same issue. Iā€™ve also noticed that the changes I have made in config.yml, mainly the order sequence of the authentication methods arenā€™t reflected correctly in Opensearch Dashboards Authentication Sequences section. The openid config, openid_auth_domain is also not shown on the page.

Could this be the reason why it is not authenticating correctly?

Opensearch Dashboards Authentication Sequences & config.yml:

@pablo @OkComputer I found the solution, tagging you OkComputer, in case you have the same problem as I had.

I managed to find a way to enable more verbose logs in my OpenSearch container by adding the following yaml key-value pair to my OpenSearch .values file at the top level.

rootLogger.level: Debug

This was kinda hard to figure out, since the helm chart templates that you can find for OpenSearch has changed a lot since the guides were made. After enabling this debugging I was able to find out that the error was due to OpenSearch not being able to verify the CA-chain of Keycloak. I fixed this issue by adding the CA chain to my OpenSearch container with the following addition to the yaml:

securityConfig:
  enabled: true
  config:
    dataComplete: false
    data:
      root-ca.pem: |
        -----BEGIN CERTIFICATE-----
        xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        -----END CERTIFICATE-----
        -----BEGIN CERTIFICATE-----
        xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        -----END CERTIFICATE-----
        -----BEGIN CERTIFICATE-----
        xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        -----END CERTIFICATE-----

as well as the following addition to my config.yml:

              openid_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: 1
                http_authenticator:
                  type: openid
                  challenge: false
                  config:
                    openid_connect_idp:
                      enable_ssl: true
                      verify_hostnames: false  
                      pemtrustedcas_filepath: /usr/share/opensearch/config/opensearch-security/root-ca.pem
                    roles_key: roles
                    subject_key: preferred_username
                    openid_connect_url: "https://qa.idp.supplier-overview.kmd.dk/realms/opensearch/.well-known/openid-configuration"
                authentication_backend:
                  type: noop

The important change here is the pemtrustedcas_filepath which is now pointing to the root-ca.pem which I defined earlier. The way to get your CA chain is to go the URL of your IDP in the browser (in my case it was keycloak). Click on the lock next to the left of your URL, click the arrow to the right of ā€œconnection is secureā€, then click on ā€œCertificate is validā€ and go to details. Here you will see the Certificate Hierachy. Export all the certificates in your hierachy and concatenate them into a single file like I have doon in my root-ca.pem. These are all public certificates, and do not need to be hidden away as a secret.

1 Like

Thanks @Ditlevsen for tagging me in your post. It seemed like mine was not an SSL issue but a problem with the configuration not being loaded as I suspected initially. Iā€™ve solved it by running:

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/kirk.pem -key /etc/opensearch/kirk-key.pem -cd /etc/opensearch/opensearch-security/

After the configuration has been applied, the order sequence of the authentication methods in Opensearch Dashboards UI appeared to be correct. I am also able to authenticate with OIDC with no issue now.