Keycloak OpenID 401 Unauthorized

Security
1

I have a few questions to ask but first of all:
I’m running opensearch 2.1 cluster as a service on 6 separate VM’s and Opensearch Dashboards on separate machine too.
All of them are securely connected and form cluster without any problems.
I try to connect Keycloak as a mean of authorization but everytime I try to log in it’s always and never allows me to visit login page, just straight error:
{"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}

I can’t wrap my head around how to set security log level to debug. I think it’s not working because I get no logs from cluster on the 401 message.
My configuration files:

security “config.yml”

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal


      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            openid_connect_idp:
              enable_ssl: true
              verify_hostnames: false
              pemtrustedcas_filepath: /usr/share/opensearch/config/cert.pem
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: https://<<keycloak>>/realms/<<realm>>/.well-known/openid-configuration
            skip_users:
              - kibanaserver
        authentication_backend:
            type: noop

opensearch_dashboard.yml

#I have nginx redirecting all incoming trafic from 80 and 5601 to 443
server.port: 5601
server.host: 127.0.0.1

opensearch.hosts: ["https://opensearch1_host.com:9200", "https://opensearch2_host.com:9200",  "https://opensearch3_host.com:9200" ]
opensearch.ssl.verificationMode: none
opensearch.username: "kibanaserver"
opensearch.password: "kibanaserver"
opensearch.requestHeadersAllowlist: [ "Authorization", "security_tenant" ]
server.ssl.enabled: true
server.ssl.certificate: /usr/share/opensearch-dashboard/config/kibana.pem
server.ssl.key: /usr/share/opensearch-dashboard/config/kibana-key.pem
opensearch.ssl.certificateAuthorities: ["/usr/share/opensearch-dashboard/config/root-ca.pem"]
opensearch_security.auth.type: "openid"
opensearch_security.openid.connect_url: https://<<keycloak>>/realms/<<realm>>/.well-known/openid-configuration
opensearch_security.openid.client_id: client_id
opensearch_security.openid.header: "Authorization"
opensearch_security.openid.client_secret: <<client_id-secret>>
opensearch_security.openid.root_ca: /usr/share/opensearch-dashboard/config/idp.pem
opensearch_security.openid.scope: openid profile email
opensearch_security.openid.verify_hostnames: false

I’ll be more than happy to provide any useful information to solve this issue or any and every configuration you might need.

Also I have a question, when I visit my dashboards site does it have to redirect me to keycloack auth page? Or do I login from dashboards interface?

Thank you very much in advance!

2

@Saidose These settings have to point to Keycloak’s certificate. The CN or SAN of the certificate must match either an IP address or FQDN of the Keycloak.

opensearch_dashboards.yml

opensearch_security.openid.root_ca: /usr/share/opensearch-dashboard/config/idp.pem

config.yml

pemtrustedcas_filepath: /usr/share/opensearch/config/cert.pem

Also please check that OpenSearch Dashboards and OpenSearch can resolve FQDN of the keycloak.

When you use SAML or OpenID authentication, it will always redirect to an IDP authentication portal (i.e. Keycloak) and then redirect back to OpenSearch Dashboards.

1 Like
3

I’ve had a similar issue on 2.2.1. My configuration is very similar to OP’s, though using Docker.

Attempting to log into dashboards sends me to Keycloak where I am able to authenticate, then the browser is redirected back to the dashboards page where we get the 401.

Keycloak client config is almost all default except for the peer URLs.

Some logs:

opensearch-node1         | [2022-09-12T19:09:44,490][WARN ][stderr                   ] [opensearch-node1] Sep 12, 2022 7:09:44 PM org.apache.cxf.rs.security.jose.jws.JwsCompactConsumer verifySignatureWith
opensearch-node1         | [2022-09-12T19:09:44,490][WARN ][stderr                   ] [opensearch-node1] WARNING: Invalid Signature
opensearch-node1         | [2022-09-12T19:09:44,490][INFO ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [opensearch-node1] Extracting JWT token from xxxxx failed
opensearch-node1         | com.amazon.dlic.auth.http.jwt.keybyoidc.BadCredentialsException: Invalid JWT signature

The failing URL:

http://dashboards:5601/auth/openid/login?state=xxxx&session_state=xxxxx&code=xxxxx.xxxxx.xxxxx

Any workarounds or fixes?

4

Im also on 2.2.1 trying to setup OpenID but with Gitlab. I’m getting a Json response in the browser with 401 unauthorized. Doesnt matter if I’m logged in to Gitlab or not. We have an integration with Discourse and Grafana working so the Gitlab part should be ok. Does anyone have a working example for Gitlab CE?

dashboard.yml

server.basePath: /opensearch
server.rewriteBasePath: true
server.host: "0.0.0.0"
opensearch.hosts: ["https://localhost:9200"]
opensearch.ssl.verificationMode: none
opensearch.username: "kibanaserver"
opensearch.password: "xyz"
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false

# openid config
opensearch_security.auth.type: "openid"
opensearch_security.openid.connect_url: "https://xyz/.well-known/openid-configuration"
opensearch_security.openid.client_id: "xyz"
opensearch_security.openid.client_secret: "xyz"

config

    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            openid_connect_url: https://xyz/.well-known/openid-configuration
        authentication_backend:
          type: noop

xyz is commented out code

Im testing with the default docker-compose setup 3 node cluster

5

@drBenway Can you resolve FQDN of the Keycloak node from the OpenSearch and OpenSearch Dashboards nodes?

What type of TLS certificate do you use in Keycloak? If self-signed then you need to configure it in opensearch_dashboards.yml and config.yml.

@noahbailey Do you have challenge set to false in basicauth nad openid in config.yml?

6

Yes, both are set to false in my opensearch config.yml.

I also can confirm that the jwks and connect urls for my keycloak server are accessible from all nodes.

The config is practically a carbon copy of the example in the docs, only changes are for my server hostnames & secrets. I can share if it helps…

7

I will try it in a day since right now I’m performing stress tests on opensearch.
This reply was extremely helpful and I think I know in what direction I need to dig, Thank you!

My kibana and opensearch hosts resolve keycloak’s fqdn with specific certificate that i put in config.yml but it doesn’t redirect me to auth page, I think my major problem is redirection to keycloak auth page. I also don’t get any errors from my log file either.
How do I validate that I can connect to auth page though?

8

@noahbailey Could you share client config and mappers?
Did you map the User Realm Role?

9

@drBenway I’m not very familiar with GitLab OpenID authentication. However, this looks like groups sent in JWT token are not recognized or they are not sent at all.
I don’t see roles_key defined in the config.yml.

10

I’ve played with a couple configurations without success.

  • No roles, mappers etc. Default config only → Returns 401 when authenticating
  • Realm roles - Claim shows as realm_access.roles.all_access, but still getting 401
  • Client roles - Claim shows as resource_access.opensearch.roles.all_access, returns 401
  • Client scopes > User realm role > token claim named “roles” to map “all_access” → same result, 401

An interesting thing I’ve noticed is that the JWT that OpenSearch prints to the console is NOT the same as the one that I get from Keycloak, as it is missing several fields that are present in a regular token. I was using those tokens to try to debug, but they were causing other issues. Could this be related to my problem, or something unrelated?

I’m really not an expert on Keycloak, so any advice would be appreciated. I think I’ve checked all the boxes, but it’s possible I’ve missed something…

11

@noahbailey I’d like to see your current Keycloak config.
Would you mind sharing the client config?
i.e.
image

image
image725×372 20.6 KB

Also, please take screenshot of the Mappers in that client.
i.e.

image
image1050×348 24.1 KB

12

@noahbailey @drBenway Please share your roles_mapping.yml.
You should at least get the own_index role assigned.

By default, own_index role is assigned to any authenticated user.

13

I’m using the latest version of Keycloak (19.0.2 as of today), so my screenshots will look a little different.

(Going to put them in separate posts since that seems to be limited on new accounts. )

With this config I can get the JWT to look like this:

...
  "sid": "d6368e42-a8ad-48b3-81c4-c5673ca3adda",
  "email_verified": true,
  "roles": "all_access",
  "name": "Bob Smith",
  "preferred_username": "admin",
  "given_name": "Bob",
  "family_name": "Smith",
  "email": "admin@example.com"
}

So, it does include the roles key which should line up with what OSD expects. Still getting the 401 responses, and the same invalid token error messages.

I haven’t made any modifications to the roles_mapping.yml file, so this is the default that comes with the docker image:

all_access:
  reserved: false
  backend_roles:
  - "admin"
  description: "Maps admin to all_access"

own_index:
  reserved: false
  users:
  - "*"
  description: "Allow full access to an index named like the username"

logstash:
  reserved: false
  backend_roles:
  - "logstash"

kibana_user:
  reserved: false
  backend_roles:
  - "kibanauser"
  description: "Maps kibanauser to kibana_user"

readall:
  reserved: false
  backend_roles:
  - "readall"

manage_snapshots:
  reserved: false
  backend_roles:
  - "snapshotrestore"

kibana_server:
  reserved: true
  users:
  - "kibanaserver"

Thinking a bit more, is it possible this is a red herring? In my experience typically a permissions/roles issue gets you a 403 error, where here I’m seeing a 401 which seems to point in the direction of the server not being able to read/understand the JWT it’s getting from Keycloak…

Thanks, Pablo! Hope we can figure this out…

14

Client config:

image
image1259×1077 69 KB

15

Mappers (different in v19+)

image
image1112×819 44 KB

16

@noahbailey I suggest removing localhost from the client configuration and Hardcoded claim. Instead, you’ll need to configure the User Realm Role.

i.e.

image
image681×836 29.6 KB

17

@noahbailey Could you share your config.yml and opensearch_dashboards.yml files?

18

Made some changes:
Localhost is removed from the client config.
Mapper is changed to the User Realm Role type:

image
image542×765 37.1 KB

JWT has changed to reflect that, with the full list of roles in the multivalue roles claim.

{
...
  "email_verified": true,
  "roles": [
    "create-realm",
    "default-roles-master",
    "offline_access",
    "admin",
    "uma_authorization",
    "all_access"
  ],
  "name": "Bob Smith",
...
}

Here are the config files, mapped through docker.

/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml

---
_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
    authc:
      internal_auth:
        order: 0
        description: "HTTP basic authentication using the internal user database"
        http_enabled: true
        transport_enabled: true
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
    openid_auth_domain:
      http_enabled: true
      transport_enabled: true
      order: 1
      http_authenticator:
        type: openid
        challenge: false
        config:
          subject_key: preferred_username
          roles_key: roles
          openid_connect_url: http://keycloak:8080/realms/master/.well-known/openid-configuration
      authentication_backend:
        type: noop

/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml

timelion.ui.enabled: true
server.name: kibana
server.host: "0.0.0.0"
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
opensearch_security.auth.type: "openid"
opensearch_security.openid.connect_url: http://keycloak:8080/realms/master/.well-known/openid-configuration
opensearch_security.openid.client_id: kibana
opensearch_security.openid.client_secret: "xxxxxxxxxxx"
opensearch_security.openid.scope: openid email profile
opensearch.requestHeadersAllowlist: ["Authorization", "security_tenant"]
opensearch_security.openid.base_redirect_url: http://kibana:5601

Unfortunately, still having the same issue.
Browser responds: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}

After successfully authenticating with Keycloak, I’m being redirected back to OSD and landing on a URL like this: http://kibana:5601/auth/openid/login?state=k68wjN7ASGdrXFScT4aogT&session_state=aab9365d-546a-4e95-a6b2-f78389845613&code=4156e0f1-9fc7-495e-9c8e-9c61671f6ba6.aab9365d-546a-4e95-a6b2-f78389845613.51de4f57-32d1-4936-a58c-8a1e51f7843b

Logs from OpenSearch server, including full stacktrace:

opensearch-node1         | [2022-09-15T14:42:46,127][WARN ][o.o.s.h.HTTPBasicAuthenticator] [opensearch-node1] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
opensearch-node1         | [2022-09-15T14:42:46,146][WARN ][stderr                   ] [opensearch-node1] Sep 15, 2022 2:42:46 PM org.apache.cxf.rs.security.jose.jws.JwsCompactConsumer verifySignatureWith
opensearch-node1         | [2022-09-15T14:42:46,146][WARN ][stderr                   ] [opensearch-node1] WARNING: Invalid Signature
opensearch-node1         | [2022-09-15T14:42:46,147][INFO ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [opensearch-node1] Extracting JWT token from XXXXXXXXXXXXX failed
opensearch-node1         | com.amazon.dlic.auth.http.jwt.keybyoidc.BadCredentialsException: Invalid JWT signature
opensearch-node1         | 	at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.getVerifiedJwtToken(JwtVerifier.java:70) ~[opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials0(AbstractHTTPJwtAuthenticator.java:107) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:89) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:86) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at java.security.AccessController.doPrivileged(AccessController.java:318) [?:?]
opensearch-node1         | 	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials(AbstractHTTPJwtAuthenticator.java:86) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.extractCredentials(HTTPSamlAuthenticator.java:160) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:244) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:191) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:124) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:312) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:241) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.2.1.jar:2.2.1]
opensearch-node1         | 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.2.1.jar:2.2.1]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) [netty-handler-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) [netty-handler-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) [netty-handler-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at java.lang.Thread.run(Thread.java:833) [?:?]

Thanks for taking a look. Let me know if there’s more info I can provide.

19

(Tried to post earlier, but it got eaten by a spam filter)

Changed the hardcoded claim to realm roles, and removed ‘localhost’ from the client config as directed. The full list of roles are in the claim when the JWT is decoded:

{
...
  "email_verified": true,
  "roles": [
    "create-realm",
    "default-roles-master",
    "offline_access",
    "admin",
    "uma_authorization",
    "all_access"
  ],
  "name": "Bob Smith",
  "preferred_username": "admin",
...
}

Here is the opensearch config file:

---
_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
    authc:
      internal_auth:
        order: 0
        description: "HTTP basic authentication using the internal user database"
        http_enabled: true
        transport_enabled: true
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
    openid_auth_domain:
      http_enabled: true
      transport_enabled: true
      order: 1
      http_authenticator:
        type: openid
        challenge: false
        config:
          subject_key: preferred_username
          roles_key: roles
          openid_connect_url: http://keycloak:8080/realms/master/.well-known/openid-configuration
      authentication_backend:
        type: noop

The Opensearch Dashboards config file:

timelion.ui.enabled: true
server.name: kibana
server.host: "0.0.0.0"
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]

opensearch_security.auth.type: "openid"
opensearch_security.openid.connect_url: http://keycloak:8080/realms/master/.well-known/openid-configuration
opensearch_security.openid.client_id: kibana
opensearch_security.openid.client_secret: "xxxxxxxx"
opensearch_security.openid.scope: openid email profile
opensearch.requestHeadersAllowlist: ["Authorization", "security_tenant"]
opensearch_security.openid.base_redirect_url: http://kibana:5601

Still having the same error:

opensearch-node1         | [2022-09-15T17:48:01,310][WARN ][o.o.s.h.HTTPBasicAuthenticator] [opensearch-node1] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
opensearch-node1         | [2022-09-15T17:48:01,323][WARN ][stderr                   ] [opensearch-node1] Sep 15, 2022 5:48:01 PM org.apache.cxf.rs.security.jose.jws.JwsCompactConsumer verifySignatureWith
opensearch-node1         | [2022-09-15T17:48:01,323][WARN ][stderr                   ] [opensearch-node1] WARNING: Invalid Signature
opensearch-node1         | [2022-09-15T17:48:01,323][INFO ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [opensearch-node1] Extracting JWT token from xxxxx-jwt-goes-here-xxxxx failed
opensearch-node1         | com.amazon.dlic.auth.http.jwt.keybyoidc.BadCredentialsException: Invalid JWT signature
opensearch-node1         | 	at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.getVerifiedJwtToken(JwtVerifier.java:70) ~[opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials0(AbstractHTTPJwtAuthenticator.java:107) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:89) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:86) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at java.security.AccessController.doPrivileged(AccessController.java:318) [?:?]
opensearch-node1         | 	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials(AbstractHTTPJwtAuthenticator.java:86) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.extractCredentials(HTTPSamlAuthenticator.java:160) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:244) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:191) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:124) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:312) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:241) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-2.2.1.0.jar:2.2.1.0]
opensearch-node1         | 	at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.2.1.jar:2.2.1]
opensearch-node1         | 	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.2.1.jar:2.2.1]
opensearch-node1         | 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.2.1.jar:2.2.1]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) [netty-handler-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) [netty-handler-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) [netty-handler-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.79.Final.jar:4.1.79.Final]
opensearch-node1         | 	at java.lang.Thread.run(Thread.java:833) [?:?]

Similarly, browser is returning the 401 error:

{"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}

Are there any other factors that could impact signature validation of the tokens on the OS side?

Cheers

20

@noahbailey The security plugin config folder mentioned by you earlier is incorrect for version 2.2.1

/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml

This has changed in version 2.0.0. The new config folder is:

/usr/share/opensearch/config/opensearch-security/config.yml