OpenID 401 Unauthorized with Keycloak

wdijkerman · July 25, 2022, 2:58pm

Hi,

I am trying to setup an Opensearch Cluster with OpenID with Keycloak. I am running locally (With Vagrant) to see if I can get it all to work, before I build my infra at AWS. So i generated some self signed certficates btw.

I have setup a Keycloak (18.0.2) with Nginx running on https://keycloak.example.com. I have created a realm monitoring and a (Access Type Public) client named dashboard. Created a mapper User Realm Role with a Token Claim Name of roles. In my realm, I have created a role named all_access.

I have a basic setup of 2 OpenSearch 2.1.0 instances in a cluster and a single dashoard 2.1.0 instance, running on https://opensearch.example.com.

I have the following opensearch-dashboard.yml file:

server.name: os_dashboards
server.host: "0.0.0.0"

opensearch.hosts: [https://opensearch-node1:9200, https://opensearch-node2:9200]
opensearch.username: kibanaserver
opensearch.password: kibanaserver

server.ssl.enabled: true
server.ssl.certificate: /usr/share/opensearch-dashboards/config/certificates/os-dashboards/os-dashboards.pem
server.ssl.key: /usr/share/opensearch-dashboards/config/certificates/os-dashboards/os-dashboards.key

opensearch.ssl.certificateAuthorities:
  [/usr/share/opensearch-dashboards/config/certificates/ca/ca.pem]
opensearch.ssl.verificationMode: full

opensearch.requestHeadersWhitelist: ["securitytenant", "authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: false
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
opensearch_security.multitenancy.enable_filter: false

opensearch_security.auth.type: "openid"
opensearch_security.openid.connect_url: https://keycloak.example.com/auth/realms/monitoring/.well-known/openid-configuration
opensearch_security.openid.base_redirect_url: https://opensearch.example.com:5601
opensearch_security.openid.client_id:  dashboard
opensearch_security.openid.header: "Authorization"
# opensearch_security.openid.client_secret: YitqxX5cRHgrvtTVulDC08qbnfwzleM2
opensearch_security.openid.root_ca: /usr/share/opensearch-dashboards/config/certificates/ca/ca.pem
opensearch_security.openid.verify_hostnames: false

I have the following config.yml for the security plugin:

_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    http:
      anonymous_auth_enabled: false

    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal

      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            openid_connect_idp:
              enable_ssl: true
              verify_hostnames: false
              pemtrustedcas_filepath: /usr/share/opensearch/config/certificates/ca/ca.pem
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: https://keycloak.example.com/auth/realms/monitoring/.well-known/openid-configuration
            skip_users:
              - kibanaserver
        authentication_backend:
          type: noop

So far, to various topic here on this forum this seems all correct. I have used a vanilla role_mappings.yml, but I changed the all_access mapping to:

all_access:
  reserved: false
  backend_roles:
    - "admin"
  users:
    - wdijkerman
  description: "Maps admin to all_access"

Started the cluster, executed the command:

sudo docker compose exec os01 bash -c "cd plugins/opensearch-security/tools/ && chmod +x securityadmin.sh && bash ./securityadmin.sh -cd ../../../config/opensearch-security/ -icl -nhnv -cacert ../../../config/certificates/ca/ca.pem -cert ../../../config/certificates/ca/admin.pem -key ../../../config/certificates/ca/admin.key -h localhost"

And when I open the dashboard on “https://opensearch.example.com:5601” I am redirected to the Keycloak login page in the monitoring realm, fill in my credentials of a user that I manually created in Keycloak and once I click login, I get a 401 Unauthorized. But when I use curl:

curl -ks \
-d "client_id=dashboard" \
-d "username=wdijkerman" \
-d "password=password" \
-d "grant_type=password" \
"https://keycloak.example.com/auth/realms/monitoring/protocol/openid-connect/token"

I get some output, like the access_token:

eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMd0RYamIzbEVwZHQwRlRTT3JEd0Q5N2ZZaDRobGhJS3RmeHlBeHdzUi1JIn0.eyJleHAiOjE2NTg3NjA1MjAsImlhdCI6MTY1ODc2MDIyMCwianRpIjoiMzdiYTRmOWUtMzNlOC00Y2YxLWIzZGYtOGVhN2NlM2E1NWEzIiwiaXNzIjoiaHR0cHM6Ly9rZXljbG9hay5leGFtcGxlLmNvbS9hdXRoL3JlYWxtcy9tb25pdG9yaW5nIiwic3ViIjoiMWMwZjkwM2UtOWE1ZC00MDEwLWEzYjgtMmMxZGJkNDliNjZhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiZGFzaGJvYXJkIiwic2Vzc2lvbl9zdGF0ZSI6ImY2MzY3MDQ4LTNjNTYtNGEyMy1hMTc2LTdlMzU3ZjYzM2IxOCIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiIl0sInNjb3BlIjoiZW1haWwgcHJvZmlsZSIsInNpZCI6ImY2MzY3MDQ4LTNjNTYtNGEyMy1hMTc2LTdlMzU3ZjYzM2IxOCIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiV2VybmVyIERpamtlcm1hbiIsInByZWZlcnJlZF91c2VybmFtZSI6IndkaWprZXJtYW4iLCJnaXZlbl9uYW1lIjoiV2VybmVyIiwiZmFtaWx5X25hbWUiOiJEaWprZXJtYW4iLCJlbWFpbCI6Im1lQGV4YW1wbGUuY29tIn0.glZUdPlFUnmMqLITQzrMksqTJOciuHNSULCmdXnqdCXvLiTmBJDgQHqGwN53ahZuUgZtZ2hPoQ5V6rVHqddn8d-WfuEPLkUvGaPAZn67sd9qhIH8fWchxIFl7KvLC_RN9xPBniD-6e1ZPj6uYo76a7nc-nmVIC2L_D3iaYDEyoibPsWbJknPmSE7hjkv0csRfNnuZQhUSxi6zGDh3u9jqUMGRgNnYi4VxaZjVDUXtZzI6QFef9Jxr72fXD1yw3N33bu1HxLZk6oQFk--jp7UkXHT88fx9mnpqs5f_68T7E5GCt0BLbyIFNG1hjLDnLXtGxRu4kZ3j6JV7YTfp3-U6g

The only error I see in Opensearch nodes is:

[2022-07-25T14:47:12,988][WARN ][o.o.s.h.HTTPBasicAuthenticator] [os01] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'

Then I created a mapper in my dashboard client, with User Client Role and also used the Token Claim Name of roles and same error message, but I then have something in my token:

eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMd0RYamIzbEVwZHQwRlRTT3JEd0Q5N2ZZaDRobGhJS3RmeHlBeHdzUi1JIn0.eyJleHAiOjE2NTg3NjA3MzgsImlhdCI6MTY1ODc2MDQzOCwianRpIjoiYmZkZjYyNGQtZDM1ZC00MTYyLTk4MGYtMjYwZDZjMGM3YTBhIiwiaXNzIjoiaHR0cHM6Ly9rZXljbG9hay5leGFtcGxlLmNvbS9hdXRoL3JlYWxtcy9tb25pdG9yaW5nIiwic3ViIjoiMWMwZjkwM2UtOWE1ZC00MDEwLWEzYjgtMmMxZGJkNDliNjZhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiZGFzaGJvYXJkIiwic2Vzc2lvbl9zdGF0ZSI6ImM5ZTQ0YmI5LWNmYTUtNGIwZC1iYzYyLTRjMTI5NGY2NTQ2OSIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiIl0sInJlc291cmNlX2FjY2VzcyI6eyJkYXNoYm9hcmQiOnsicm9sZXMiOlsiYWxsX2FjY2VzcyJdfX0sInNjb3BlIjoiZW1haWwgcHJvZmlsZSIsInNpZCI6ImM5ZTQ0YmI5LWNmYTUtNGIwZC1iYzYyLTRjMTI5NGY2NTQ2OSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJyb2xlcyI6WyJhbGxfYWNjZXNzIl0sIm5hbWUiOiJXZXJuZXIgRGlqa2VybWFuIiwicHJlZmVycmVkX3VzZXJuYW1lIjoid2Rpamtlcm1hbiIsImdpdmVuX25hbWUiOiJXZXJuZXIiLCJmYW1pbHlfbmFtZSI6IkRpamtlcm1hbiIsImVtYWlsIjoibWVAZXhhbXBsZS5jb20ifQ.Ht5AiL7OfC9ujuWhGTQQMJIaFKcFZh8X_livSGFWlG-NQWO1W7frEWPF0z-4u6kfWp5vbl1DWuVJvdRYqdpkVzjhdEYCOO7SonOZPvvaTPbfm1Qw02xSkWt-726CBquGV0R2vNDbxhPd4-6cBgMOiCDlKLl8nwsFpzrc780xRzpvb3qqjPOKr1qrnEXqdh3QoVHyCy6LDMDnwLYOSsmK79D2ivCRsgcthXid9VVxNCaf55yxekqWK_04ktO0dksfEhspmbdHrHLWK6b83WTHxQ9XN5VOL6IH_4VJeXjH8rdkyIWPoaO9sdO87GDTlmREYcKv0jipFYZEpRpPz9oAQg

So at the moment, I have really no idea what I am doing wrong. I probably overlook something, but searching on “opensearch openid” here on the forum I found lots of topics, but I tried to apply/verify my setup but no luck so far. I need help, I don’t see the problem right now.

This is all locally, so if you need additional info or screenshots please let me know. Thanks in advance!

Kind regards,
Werner

pablo · July 25, 2022, 4:49pm

@wdijkerman With vanilla roles.yml and roles_mapping.yml you must map your user to backend roles and not the role in keycloak.
For the reported user try changing all_access to admin as admin is a backend role for all_access role.
Also, add kibanauser to have read access to Kibana UI.

wdijkerman · July 25, 2022, 5:59pm

I don’t understand what you mean. So my user “wdijkerman” also needs to be configured in these yml files, so that means I would have a role_mappings.yml snippet:

all_access:
  reserved: false
  backend_roles:
    - "admin"
  users:
    - wdijkerman
  description: "Maps admin to all_access"
kibana_user:
  reserved: false
  hidden: false
  backend_roles:
  - "kibanauser"
  hosts: []
  users:
    - wdijkerman
  and_backend_roles: []
  description: "Maps kibanauser to kibana_user"
  ...

Run the securityadmin script and then it should work?

I added in Keycloak in the client configuration a role named kibanauser and admin, assigned my user to both roles, but same error. Once I log in, I still get a 401 Unauthorized.

I do see both kibanauser as admin in my access token:

eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJMd0RYamIzbEVwZHQwRlRTT3JEd0Q5N2ZZaDRobGhJS3RmeHlBeHdzUi1JIn0.eyJleHAiOjE2NTg3NzIxNTQsImlhdCI6MTY1ODc3MTg1NCwianRpIjoiM2ExOGI0M2EtYzZlMi00ZDJiLTg5ZWYtODBiNTM0M2ExOGJlIiwiaXNzIjoiaHR0cHM6Ly9rZXljbG9hay5leGFtcGxlLmNvbS9hdXRoL3JlYWxtcy9tb25pdG9yaW5nIiwic3ViIjoiMWMwZjkwM2UtOWE1ZC00MDEwLWEzYjgtMmMxZGJkNDliNjZhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiZGFzaGJvYXJkIiwic2Vzc2lvbl9zdGF0ZSI6ImM3ODgwMjRjLTFhOTMtNDNhMC1hZWQ0LTkwNDcwMjMzMTU4MCIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiIl0sInJlc291cmNlX2FjY2VzcyI6eyJkYXNoYm9hcmQiOnsicm9sZXMiOlsiYWRtaW4iLCJraWJhbmF1c2VyIl19fSwic2NvcGUiOiJlbWFpbCBwcm9maWxlIiwic2lkIjoiYzc4ODAyNGMtMWE5My00M2EwLWFlZDQtOTA0NzAyMzMxNTgwIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsInJvbGVzIjpbImFkbWluIiwia2liYW5hdXNlciJdLCJuYW1lIjoiV2VybmVyIERpamtlcm1hbiIsInByZWZlcnJlZF91c2VybmFtZSI6IndkaWprZXJtYW4iLCJnaXZlbl9uYW1lIjoiV2VybmVyIiwiZmFtaWx5X25hbWUiOiJEaWprZXJtYW4iLCJlbWFpbCI6Im1lQGV4YW1wbGUuY29tIn0.gACtjPjGAGjegdl8iuVISvcMGYKwao4zLqiXP_AVzMwlSyxTpQ8w7kdAou-u2r9gYHPxnhod31XaPAtk6Ftw1zA0Pgx-Uxgc7LR84nwGrZ7UwfVVZCC7u7p6WCX4U4KHIV6geTe4W2VOAfEid3HUqDS7BUwkesYt3A_Pg-dUj1ZRP0thZkw1q7nRLQY1TOi-8pphG4Ty1Zc-4VVsaRPXrg930qu8cWzfHsaGKmpt11qdJdYDsoTTZWbqQM8HVVwVIr42kynC3L9xv2hMQ9KrG-MdsZYlnrP-HjkqL59ePBXZICV9qA2mkq6yAIpKfiHNHUTOy2O6_qLCV2VDP5kn-

Thanks in advance.

pablo · July 25, 2022, 7:23pm

@wdijkerman No, you don’t need to map that user. kibanauser and admin backend roles are built-in. The security plugin will map those backend roles to correct roles.

pablo · July 25, 2022, 7:29pm

@wdijkerman Could you share your role mapping from your client in Keycloak?

wdijkerman · July 26, 2022, 5:40am

Hi, yes:

And I have

I have 2 right because the User Realm Role did not provide any info in the access_token, thats why I also added the User Client Role.

wdijkerman · July 28, 2022, 6:18am

Any one have an idea why it isn’t working, please?

pablo · July 29, 2022, 12:38am

@wdijkerman My previous repro was working with Keycloak 11.0.2 and I had no issues with OpenID authentication.

I’ve repro your error with Keycloak 18.0.2 and OpenSearch 2.1.0.
I’ve found the root cause was in the openid_connect_url.

In version 11.0.2 this URL is:

https://keycloak.example.com/auth/realms/monitoring/.well-known/openid-configuration

and in 18.0.2:

https://keycloak.example.com/realms/monitoring/.well-known/openid-configuration

You can confirm that by checking URL in Endpoints of any realm in Keycloack

I still don’t know how you got your curl command working against the keycloak as the URL is incorrect.

Your OpenSearch Dashboards and security config use incorrect URL.

wdijkerman · July 29, 2022, 12:19pm

Hi Pablo,

Thank you for your reply, but this doesn’t work for me. I have the following set in my docker compose for keycloak:

KEYCLOAK_FRONTEND_URL: https://keycloak.example.com/auth

When I fill in the credentials in the Login form, I see the following 2 requests appear in my nginx:

10.10.1.1 - - [29/Jul/2022:12:06:21 +0000] "POST /auth/realms/monitoring/login-actions/authenticate?session_code=_y2WqG5F5fsjJHTWTGrg1luol9Jm1ypDAd80xBpk9BY&execution=2c3c7540-1278-4ea1-a8ec-a3fa242681b6&client_id=dashboard&tab_id=4k5swvkjEIU HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0" "-"
10.10.1.12 - - [29/Jul/2022:12:06:21 +0000] "POST /auth/realms/monitoring/protocol/openid-connect/token HTTP/1.1" 200 3581 "-" "-" "-"

And this works:

So I am very curious what I am doing wrong.

Hopefully some one knows or has a pointer for me.

wdijkerman · August 2, 2022, 3:26pm

I noticed one typo in my docker-compose file with regards to the mount of config.yml, which I now corrected to /usr/share/opensearch/config/opensearch-security/config.yml

But now I have this as an error:

[2022-08-02T15:19:43,527][WARN ][o.o.s.h.HTTPBasicAuthenticator] [os01] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-08-02T15:19:43,528][WARN ][o.o.s.a.BackendRegistry  ] [os01] Authentication finally failed for null from 172.21.0.3:46212

And see this in the dashboards container:

{"type":"log","@timestamp":"2022-08-02T15:19:43Z","tags":["error","plugins","securityDashboards"],"pid":1,"message":"OpenId authentication failed: Error: Authentication Exception"}
{"type":"response","@timestamp":"2022-08-02T15:19:43Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/auth/openid/login?state=VwfdhE9PkHLHnSG295ukmk&session_state=b037630a-2732-4051-95d2-bb84db521d1c&code=ca223aa0-08c7-47ec-8bbb-fda9cdf5824a.b037630a-2732-4051-95d2-bb84db521d1c.a170f59f-48fa-4eb9-8507-41b0e8e79cae","method":"get","headers":{"host":"opensearch.example.com:5601","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","dnt":"1","connection":"keep-alive","upgrade-insecure-requests":"1","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1"},"remoteAddress":"10.10.1.1","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0"},"res":{"statusCode":401,"responseTime":347,"contentLength":9},"message":"GET /auth/openid/login?state=VwfdhE9PkHLHnSG295ukmk&session_state=b037630a-2732-4051-95d2-bb84db521d1c&code=ca223aa0-08c7-47ec-8bbb-fda9cdf5824a.b037630a-2732-4051-95d2-bb84db521d1c.a170f59f-48fa-4eb9-8507-41b0e8e79cae 401 347ms - 9.0B"}

So, one small step closer …

pablo · August 2, 2022, 8:37pm

@wdijkerman Does your OpenSearch Dashboards run behind the reverse proxy? Could you share your Nginx config?

wdijkerman · August 3, 2022, 5:42am

No Nginx is yet running with Dashboard. Only on keycloak (And I only did Keycloak because I have read somewhere that Opensearch could have issues when it is running on TLS when Keycloak didn’t have TLS certificates. But I don’t mind placing an Nginx in front of it. When this POC is working it will go to an environment where each dashboard has an Nginx in front of it.

wdijkerman · August 3, 2022, 7:30am

I have created a Github repo with my work: GitHub - dj-wasabi/opensearch-keycloak

It contains a Vagrantfile, with 2 nodes (keycloak and opensearch). Keycloak is still manually creating a realm, client etc. Once it works, I will automate it with Ansible. When both are started, go to /opt/resources and do a docker compose up -d and make sure on you device, you have the following in hosts file:

10.10.1.10 keycloak.example.com
10.10.1.11 grafana.example.com
10.10.1.12 opensearch.example.com

(Where grafana is optional btw as this one is also commented in the Vagrantfile, was part of my POC.)

wdijkerman · August 3, 2022, 12:02pm

I have it working now.

      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: "openid"
          challenge: false
          config:
            openid_connect_idp:
              enable_ssl: true
              verify_hostnames: false
              pemtrustedcas_filepath: /usr/share/opensearch/config/certificates/rootCa.crt
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: https://keycloak.example.com/auth/realms/monitoring/.well-known/openid-configuration
        authentication_backend:
          type: noop

I had to change the pemtrustedcas_filepath to the ca file of my keycloak and I had this one to my ca of opensearch. To get it deployed to AWS, I need to figure out which CA is used with the generated certificates used by my employer…

Thank you pablo!

pablo · August 3, 2022, 2:08pm

@wdijkerman I assumed that it was keycloak’s cert. Thanks for sharing the solution.

Topic		Replies	Views
Keycloak OpenID 401 Unauthorized Security	22	5595	September 21, 2022
OpenSearch cannot Authenticate Keycloak server that use Self Sign Certificate Security troubleshoot , configure	1	809	March 4, 2022
OpenID(keycloak) 401 Unauthorized Security	10	2227	January 10, 2023
OpenSearch Dashboard with Keycloak IdP is working, but with OpenSearch not Security troubleshoot , configure	2	497	February 23, 2023
Integrates with Keycloak OpenSearch Dashboards	1	250	May 9, 2024

OpenID 401 Unauthorized with Keycloak

Related topics