Problems with Keycloak OpenID Connect and Dashboards

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch/OpenSearch Dashboards 2.8.0 (RPM installation on CentOS 7.9.2009)
Keycloak 20.0.3 (On Kubernetes)

Describe the issue:
When I try to login with OIDC in OSD I receive an error 401

{"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}

In the logs I can see
Algorithm of JWT does not match algorithm of JWK (HS512 != RS256)

Trying since Monday to get it up and running, first on a new cluster, now for testing on one server to eliminate complexity.

Configuration:

config.yml

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
    authc:
      basic_internal_auth_domain:
        order: 0
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      saml_auth:
        order: 1
        http_enabled: true
        transport_enabled: false
        http_authenticator:
          type: saml
          challenge: true
          config:
            idp:
              metadata_url: https://keycloak.company/realms/infra/protocol/saml/descriptor
              entity_id: https://keycloak.company/realms/infra
            sp:
              entity_id: opensearch-saml
            kibana_url: http://lab51.company.lan:5601
            subject_key: NameID
            roles_key: Role
            exchange_key: 1a2a3a4a5a6a7a8a9a0a1b2b3b4b5b6b
        authentication_backend:
          type: noop
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 2
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: https://keycloak.company/realms/infra/.well-known/openid-configuration
            jwks_uri: https://keycloak.company/realms/infra/protocol/openid-connect/certs
        authentication_backend:
          type: noop

The keycloak server has a public certificate.

opensearch-dashboards.yml

opensearch.hosts: [https://localhost:9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
#opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch.requestHeadersWhitelist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false

opensearch_security.auth.type: ["basicauth","openid" ]
opensearch_security.auth.multiple_auth_enabled: true
opensearch_security.openid.connect_url: "https://keycloak.company/realms/infra/.well-known/openid-configuration"
opensearch_security.openid.client_id: "opensearch"
opensearch_security.openid.client_secret: "verysecurepassword"

Relevant Logs or Screenshots:

Logs from OpenSearch, JWT token inside

[2023-06-30T10:43:10,887][WARN ][o.o.s.h.HTTPBasicAuthenticator] [lab51.company.lan] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic' [2023-06-30T10:43:10,888][INFO ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [lab51.company.lan] Extracting JWT token from eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJxdS1BN3d1T1dvWngtSUE1Xy1NdFdlcDF2UGNlUERaRG90b0JTR0ZteEx3In0.eyJleHAiOjE2ODgxMTQ4OTAsImlhdCI6MTY4ODExNDU5MCwiYXV0aF90aW1lIjoxNjg4MTE0NTkwLCJqdGkiOiJiMDgwOTYyNi1kMmNjLTQ5YjgtYTY1Yi04NDNmZjZkYjVmMDIiLCJpc3MiOiJodHRwczovL2tleWNsb2FrLmhtbS50b29scy9yZWFsbXMvaW5mcmEiLCJhdWQiOiJvcGVuc2VhcmNoIiwic3ViIjoiOGFkN2E5MjktMDUxOS00ZGJmLTg0NzgtODBjYjcyYjgzNmRkIiwidHlwIjoiSUQiLCJhenAiOiJvcGVuc2VhcmNoIiwic2Vzc2lvbl9zdGF0ZSI6ImZmYjE4NDg0LTViYjYtNGExZC05ZjQyLTkyYTc4YjIzZDgwYSIsImF0X2hhc2giOiIyTTR3WjQ4MVlCcVMzd3FIa01aQVB3IiwiYWNyIjoiMSIsInNpZCI6ImZmYjE4NDg0LTViYjYtNGExZC05ZjQyLTkyYTc4YjIzZDgwYSIsImFkZHJlc3MiOnt9LCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInJvbGVzIjpbIk9QRU5TRUFSQ0gtREFTSEJPQVJELUFETUlOIiwiSE1NLVBST0QiLCJvZmZsaW5lX2FjY2VzcyIsIkhNTS1ERVYtUk8iLCJITU0tSU5GUkEiLCJITU0tREVWIiwiU1lTVEVNLUFETUlOSVNUUkFUT1IiLCJ1bWFfYXV0aG9yaXphdGlvbiIsImRlZmF1bHQtcm9sZXMtaW5mcmEiXSwibmFtZSI6IlNhc2NoYSBIZW5rZSIsInByZWZlcnJlZF91c2VybmFtZSI6InNoZW5rZSIsImdpdmVuX25hbWUiOiJTYXNjaGEiLCJmYW1pbHlfbmFtZSI6IkhlbmtlIiwiZW1haWwiOiJoZW5rZUBobW1kZXV0c2NobGFuZC5kZSJ9.pCyBJ1GXKqvWunD3AUydOLWzE8hEaNIDLrbaQhIo_irU2hprG1r2Zon8Si18sewEN6OHuXTao5L_TtHHyRXu82WXPtFIhAbgFAZV-8EGyEZY6s6S4cUQ173B0NlBO8hJFKdpFusO4fbuf0mXPxFPt78LYB4CdxmaF2fC2HQg7C4DGqIHDaedtdsgL8H5zteJAwdmM-D3cLNJOmu3OgBQedgVrbVdb3hlkpodAnMexqOo2nCyKMmZ8My-yYcVRPP1bCm56fXvQWq2FQuRHGd4IzRQzJharHShejBKN6j4g8iJgeOkw7n18nV109fGGG5KWyV8qwGAOcA76nMK4HU7fA failed com.amazon.dlic.auth.http.jwt.keybyoidc.BadCredentialsException: Algorithm of JWT does not match algorithm of JWK (HS512 != RS256) at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.validateSignatureAlgorithm(JwtVerifier.java:96) ~[opensearch-security-2.8.0.0.jar:2.8.0.0] at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.getInitializedSignatureVerifier(JwtVerifier.java:104) ~[opensearch-security-2.8.0.0.jar:2.8.0.0] at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.getVerifiedJwtToken(JwtVerifier.java:64) ~[opensearch-security-2.8.0.0.jar:2.8.0.0] at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials0(AbstractHTTPJwtAuthenticator.java:115) [opensearch-security-2.8.0.0.jar:2.8.0.0] at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:97) [opensearch-security-2.8.0.0.jar:2.8.0.0] at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:94) [opensearch-security-2.8.0.0.jar:2.8.0.0] at java.security.AccessController.doPrivileged(AccessController.java:318) [?:?] at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials(AbstractHTTPJwtAuthenticator.java:94) [opensearch-security-2.8.0.0.jar:2.8.0.0] at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.extractCredentials(HTTPSamlAuthenticator.java:160) [opensearch-security-2.8.0.0.jar:2.8.0.0] at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:244) [opensearch-security-2.8.0.0.jar:2.8.0.0] at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:191) [opensearch-security-2.8.0.0.jar:2.8.0.0] at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:124) [opensearch-security-2.8.0.0.jar:2.8.0.0] at org.opensearch.rest.RestController.dispatchRequest(RestController.java:320) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:411) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.rest.RestController.dispatchRequest(RestController.java:249) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-2.8.0.0.jar:2.8.0.0] at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.8.0.jar:2.8.0] at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.8.0.jar:2.8.0] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.8.0.jar:2.8.0] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) [netty-handler-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) [netty-handler-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) [netty-handler-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.91.Final.jar:4.1.91.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.91.Final.jar:4.1.91.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.91.Final.jar:4.1.91.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.91.Final.jar:4.1.91.Final] at java.lang.Thread.run(Thread.java:833) [?:?] [2023-06-30T10:43:10,893][WARN ][o.o.s.a.BackendRegistry ] [lab51.company.lan] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'

Logs from OSD

{"type":"log","@timestamp":"2023-06-30T08:36:11Z","tags":["error","plugins","securityDashboards"],"pid":1416,"message":"OpenId authentication failed: Error: Authentication Exception"}
{"type":"response","@timestamp":"2023-06-30T08:36:11Z","tags":[],"pid":1416,"method":"get","statusCode":401,"req":{"url":"/auth/openid/login?state=P-A62Hg_MURogbt77xIxnX&session_state=5eb4ccef-44f5-41a9-9a92-08bb89e576e6&code=3d5caee6-66ac-4b74-8641-8b1f2ae02859.5eb4ccef-44f5-41a9-9a92-08bb89e576e6.da91734f-610a-4b6d-9f61-c6aa2e7aa985","method":"get","headers":{"host":"lab51.company.lan:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","accept-encoding":"gzip, deflate","accept-language":"de-DE,de;q=0.9"},"remoteAddress":"192.168.113.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"},"res":{"statusCode":401,"responseTime":95,"contentLength":9},"message":"GET /auth/openid/login?state=P-A62Hg_MURogbt77xIxnX&session_state=5eb4ccef-44f5-41a9-9a92-08bb89e576e6&code=3d5caee6-66ac-4b74-8641-8b1f2ae02859.5eb4ccef-44f5-41a9-9a92-08bb89e576e6.da91734f-610a-4b6d-9f61-c6aa2e7aa985 401 95ms - 9.0B"}
1 Like
2

Okay, with a lot of tests and head scratching and slightly going insane, I found the solution.

Important in opensearch_dashboards.yml is this snippet:

opensearch.requestHeadersAllowlist: ["securitytenant", "authorization", "WWW-Authenticate"]

After setting the header “WWW-Authenticate” it worked like a charm.

This setting in Keycloak is also important:

grafik
grafik894Ă—324 15.5 KB

You need to create the role “admin” and not “all_access” as I always thought.

Sascha

3

@henkes Just a small addon. You’ve configured OpenID with BasicAuth in OpenSearch Dashboards.
However, you’ve enabled saml in config.yml. You should disable or remove saml configuration as OpenSearch will try to authenticate your OpenID user in the order defined in your config.yml.

  1. basicauth
  2. saml
  3. openid

This may cause issues and possibly be the root cause of your original issue.

Regarding the all_access this is the internal role of OpenSearch. The admin is a built-in backend role in OpenSearch.
If you define more custom roles in keycloak then you must map them as backend_roles in the roles_mapping.yml.

2 Likes
4

@pablo
Thank you for the addition. I already removed the SAML part, since we don’t use it. SAML was a test.