OpenID(keycloak) 401 Unauthorized

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch - 2.4.0 / Dashboard - 2.4.0 /Server OS - Centos 7.9 / Browser - Chrome

Describe the issue:

I’m testing opensearch 2.4.0 cluster and Opensearch on VM server and installed opensearch from a tarball.

I’m trying to integrate with OpenID Connect by following the guide here ([OpenID Connect - Documentation]). But never allows me to visit login page, just straight error:
{"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}

Configuration:

config.yml authc:

    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: "internal"
    openid_auth_domain:
      http_enabled: false
      transport_enabled: true
      order: 1
      http_authenticator:
        type: openid
        challenge: false
        config:
          subject_key: preferred_username
#          subject_key: given_name
          roles_key: roles
          openid_connect_url: https://dev-sso.cloud.hyundai-autoever.com/auth/realms/HCP/.well-known/openid-configuration
      authentication_backend:
        type: noop

opensearch_dashboard.yml

opensearch.hosts: [https://10.5.186.236:9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersWhitelist: ["Authorization", "securitytenant"]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: false


opensearch_security.auth.type: ["basicauth","openid"]
opensearch_security.auth.multiple_auth_enabled: true
server.xsrf.allowlist: ["/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/logout"]



opensearch_security.openid.connect_url: "https://my-keycloak/ auth/realms/HCP/.well-known/openid-configuration"
opensearch_security.openid.client_id: "opensearch"
opensearch_security.openid.client_secret: "NWqUEsgkRGvrggUhD6EPGEt5NfCjg2wV"
opensearch_security.openid.base_redirect_url: "http://10.5.186.236:5601/api/security/v1/master"
opensearch_security.openid.scope: "openid profile email"
opensearch_security.openid.header: "Authorization"

Keyclaok Settings


Relevant Logs or Screenshots:
Dashboard logs

  log   [07:32:02.293] [debug][server][OpenSearchDashboards][cookie-session-storage][http] Error: Unauthorized
respons [07:32:02.293]  GET /favicon.ico 401 1ms - 9.0B
  log   [07:32:06.479] [debug][metrics] Refreshing metrics

when I excute the security_admin.sh, generated Error log as below.

/data/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
  -cacert /data/opensearch/config/root-ca.pem \
  -cert /data/opensearch/config/kirk.pem \
  -key /data/opensearch/config/kirk-key.pem \
  -cd /data/opensearch/config/opensearch-security/

log

Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=kirk,OU=client,O=client,L=test,C=de"
OpenSearch Version: 2.4.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: test
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /data/opensearch/config/opensearch-security/
ERR: Seems /data/opensearch/config/opensearch-security/config.yml is not in OpenSearch Security 7 format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "openid_auth_domain" (class org.opensearch.security.securityconf.impl.v7.ConfigV7$Dynamic), not marked as ignorable (15 known properties: "license", "disable_intertransport_auth", "http", "kibana", "authz", "auth_failure_listeners", "transport_userrname_attribute", "filtered_alias_mode", "authc", "disable_rest_auth", "respect_request_indices_options", "multi_rolespan_enabled", "do_not_fail_on_forbidden", "hosts_resolver_mode", "do_not_fail_on_forbidden_empty"])
 at [Source: (String)"{"_meta":{"type":"config","config_version":2},"config":{"dynamic":{"http":{"anonymous_auth_enabled":false,"xff":{"enabled":false,"internalProxies":"192\\.168\\.0\\.10|192\\.168\\.0\\.11"}},"authc":{"basic_internal_auth_domain":{"description":"Authenticate via HTTP Basic against internal users database","http_enabled":true,"transport_enabled":true,"order":0,"http_authenticator":{"type":"basic","challenge":false},"authentication_backend":{"type":"internal"}}},"openid_auth_domain":{"http_enabled":f"[truncated 1172 chars]; line: 1, column: 485] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["config"]->org.opensearch.security.securityconf.impl.v7.ConfigV7["dynamic"]->org.opensearch.security.securityconf.impl.v7.ConfigV7$Dynamic["openid_auth_domain"])
ERR: Seems /data/opensearch/config/opensearch-security/roles.yml is not in OpenSearch Security 7 format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "cluster" (class org.opensearch.security.securityconf.impl.v7.RoleV7), not marked as ignorable (7 known properties: "index_permissions", "reserved", "hidden", "description", "static", "cluster_permissions", "tenant_permissions"])
 at [Source: (String)"{"_meta":{"type":"roles","config_version":2},"kibana_read_only":{"reserved":true},"security_rest_api_access":{"reserved":true},"alerting_read_access":{"reserved":true,"cluster_permissions":["cluster:admin/opendistro/alerting/alerts/get","cluster:admin/opendistro/alerting/destination/get","cluster:admin/opendistro/alerting/monitor/get","cluster:admin/opendistro/alerting/monitor/search","cluster:admin/opensearch/alerting/findings/get"]},"alerting_ack_alerts":{"reserved":true,"cluster_permissions":"[truncated 7235 chars]; line: 1, column: 7659] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["sg_human_resources"]->org.opensearch.security.securityconf.impl.v7.RoleV7["cluster"])
ERR: Seems /data/opensearch/config/opensearch-security/roles_mapping.yml is not in OpenSearch Security 7 format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "backendroles" (class org.opensearch.security.securityconf.impl.v7.RoleMappingsV7), not marked as ignorable (7 known properties: "backend_roles", "users", "hidden", "hosts", "description", "reserved", "and_backend_roles"])
 at [Source: (String)"{"_meta":{"type":"rolesmapping","config_version":2},"all_access":{"reserved":false,"backend_roles":["admin"],"description":"Maps admin to all_access"},"own_index":{"reserved":false,"users":["*"],"description":"Allow full access to an index named like the username"},"logstash":{"reserved":false,"backend_roles":["logstash"]},"kibana_user":{"reserved":false,"backend_roles":["kibanauser"],"description":"Maps kibanauser to kibana_user"},"readall":{"reserved":false,"backend_roles":["readall"]},"manage"[truncated 171 chars]; line: 1, column: 665] (through reference chain: org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration["sg_human_resources"]->org.opensearch.security.securityconf.impl.v7.RoleMappingsV7["backendroles"])
Will update '/internalusers' with /data/opensearch/config/opensearch-security/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /data/opensearch/config/opensearch-security/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /data/opensearch/config/opensearch-security/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /data/opensearch/config/opensearch-security/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /data/opensearch/config/opensearch-security/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /data/opensearch/config/opensearch-security/audit.yml
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /data/opensearch/config/opensearch-security/allowlist.yml
   SUCC: Configuration for 'allowlist' created or updated
ERR: cannot upload configuration, see errors above

Please help what the problem…

@jun What is the Keycloak’s version?
According to your last output, the securityadmin.sh failed to update config.yml.
As far as I can see, openid_auth_domain has incorrect indent and is at the same level as authc.

@jun In your config.yml you have roles_key: roles but the Token Claim Name in the Mappers is all_access. These values have to match to pass the roles from the keycloak.

Also, the below is incorrect.

opensearch_security.openid.base_redirect_url: "http://10.5.186.236:5601/api/security/v1/master"

It should be.

opensearch_security.openid.base_redirect_url: "http://10.5.186.236:5601"

Hi, pablo.
My keycloak version is 18.0.0.
After modified indent in config.yml as your guide , securityadmin.sh was executed successfully
and changed opensearch_dashboard.yml

opensearch_security.openid.base_redirect_url: "http://10.5.186.236:5601"

But still having the same error.

  • opensearch.log
[2022-12-13T10:50:59,172][WARN ][o.o.s.h.HTTPBasicAuthenticator] [node-1] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
  • opensearch_dashboard
 [error][plugins][securityDashboards] OpenId authentication failed: Error: Response Error: 400 Bad Request
respons [01:54:26.167]  GET /auth/openid/login?state=O14wF3BZAI2ueU-GXIwv0G&session_state=624a14be-c91b-473d-9c09-fe2a7f360754&code=b6b9aca8-07c6-4dd6-a55c-62967580a29a.624a14be-c91b-473d-9c09-fe2a7f360754.69979055-ff4d-4f16-9b31-bf658bb9fb06 302 11ms - 9.0B

Could you guide how to match the roles_key and Mappers?

@jun Your role mapper is correct. However, you need to change the value of the Token Claim Name to roles so it will correspond with roles_key: roles in config.yml or change roles_key to roles_key: all_access.
Then you need to create roles and assign to the users in Keycloak. Finally assign those roles in the roles_mapping.yml as backend_roles.

You can reference my thread:

please check whether registering route handler for [/auth/openid/login] successful before http server running at https://0.0.0.0:5601.

As I test, the openid route handler is not register successful before the server start. So it will always 401.

Thank you for your reply.

But I don’t know how to assign the keycloak users and setting the roles_mapping.yml properly.

Any examples of how to configure it?

Thanks for you reply.
I checked the /auth/openid/login registed log line as below.

 log   [01:48:24.402] [debug][server][OpenSearchDashboards][http] registering route handler for [/auth/openid/login]
  log   [01:48:24.402] [debug][server][OpenSearchDashboards][http] registering route handler for [/auth/openid/logout]

But still having the same error.

opensearch.log

[2022-12-15T10:50:59,172][WARN ][o.o.s.h.HTTPBasicAuthenticator] [node-1] No 'Basic Authorizat

Anything else to check?

@jun Please follow this documentation.

@pablo
hi,

After I set the config.yml and keycloak as your guide, the error message “No ‘Basic Authorization’ header, send 401 and ‘WWW-Authenticate Basic’” was solved.
But when I try to connect from OIDC , generated Error log as below and back to log in page(http://10.5.186.236:5601/app/login?)

respons [05:53:48.195]  GET /app/login 200 9ms - 9.0B
  log   [05:53:48.275] [debug][server][OpenSearchDashboards][cookie-session-storage][http] Error: Unauthorized
respons [05:53:48.274] [api] GET /bootstrap.js 304 5ms - 9.0B
respons [05:53:49.123]  GET /translations/en.json 304 1ms - 9.0B
respons [05:53:49.126]  GET /ui/legacy_light_theme.css 200 4ms - 9.0B
respons [05:53:49.131]  GET /node_modules/@osd/ui-framework/dist/kui_light.css 200 2ms - 9.0B
  log   [05:53:49.415] [debug][server][OpenSearchDashboards][cookie-session-storage][http] Error: Unauthorized
respons [05:53:49.415]  GET /api/v1/restapiinfo 401 1ms - 9.0B
respons [05:53:49.417]  GET /ui/fonts/inter_ui/Inter-UI-Regular.woff2 200 1ms - 9.0B
  log   [05:53:49.445] [debug][server][OpenSearchDashboards][cookie-session-storage][http] Error: Unauthorized
respons [05:53:49.445]  GET /api/v1/configuration/account 401 1ms - 9.0B
respons [05:53:49.464]  POST /api/core/capabilities 200 3ms - 9.0B

JWT token samle was as bellow.

{
  "exp": 1673350065,
  "iat": 1673335665,
  "jti": "7c4b929f-44b5-4844-8ccd-ac02a8abbff3",
  "iss": "https://mycompany/auth/realms/TEST",
  "aud": [
    "test",
    "account"
  ],
  "sub": "43b2cf9f-92f6-496d-985c-84e7f1413ee7",
  "typ": "Bearer",
  "azp": "opensearch",
  "session_state": "ae1ac5c1-300a-4ecb-920e-6df802634a37",
  "acr": "1",
  "realm_access": {
    "roles": [
      "DEFAULT-MEMBER",
      "offline_access",
      "uma_authorization",
      "COMPANY-MEMBER"
    ]
  },
  "resource_access": {
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "openid displayName department app-audience groups profile company email",
  "sid": "ae1ac5c1-300a-4ecb-920e-6df802634a37",
  "email_verified": false,
  "displayName": "test",
  "roles": [
    "DEFAULT-MEMBER",
    "offline_access",
    "uma_authorization",
    "COMPANY-MEMBER"
  ],
  "name": "test",
  "groups": [
    "/test",
    "/DEFAULT-GROUP"
  ],
  "company": "test-company",
  "preferred_username": "test",
  "department": "test; testteam",
  "given_name": "test",
  "family_name": "test",
  "email": "kwonhc@gmail.com"
}