OpenID authentication and "Authentication finally failed for null"

Hi

I have been strugling for a while to get OpenID authentication (Keycloak as IdP) to work with OD for ES & Kibana 1.8 and finally run out of options to try and would need help to get this working. Everything used here is ran on top of Kubernetes.

But starting from configs.

ES config.yml (imported after changes with securityadmin.sh):

_meta:
type: “config”
config_version: 2

config:
dynamic:
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: .+
authc:
basic_internal_auth_domain:
description: “Authenticate via HTTP Basic against internal users database”
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: internal
openid_auth_domain:
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
type: openid
challenge: false
config:
subject_key: preferred_username
roles_key: roles
enable_ssl: true
verify_hostnames: false
pemtrustedcas_filepath: /usr/share/elasticsearch/config/keycloak-root-ca.pem
openid_connect_url: https:///auth/realms//.well-known/openid-configuration
authentication_backend:
type: noop
authz:
roles_from_myldap:
description: “Authorize via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
rolebase: ‘ou=groups,dc=example,dc=com’
rolesearch: ‘(member={0})’
userroleattribute: null
userrolename: disabled
rolename: cn
resolve_nested_roles: true
userbase: ‘ou=people,dc=example,dc=com’
usersearch: ‘(uid={0})’
roles_from_another_ldap:
description: “Authorize via another Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap

And kibana.yml:

server.name: kibana
server.host: “0.0.0.0”
elasticsearch.hosts: ${ELASTICSEARCH_URL}
elasticsearch.requestTimeout: 360000
server.ssl.enabled: true
server.ssl.key: /usr/share/kibana/config/kibana-key.pem
server.ssl.certificate: /usr/share/kibana/config/kibana-crt.pem
elasticsearch.ssl.certificateAuthorities: /usr/share/kibana/config/kibana-root-ca.pem
elasticsearch.ssl.verificationMode: none
elasticsearch.username: “kibanaserver”
elasticsearch.password: “”
elasticsearch.requestHeadersWhitelist: [“Authorization”, “security_tenant”, “securitytenant”, “x-forwarded-for”, “x-forwarded-by”]
opendistro_security.cookie.secure: true
opendistro_security.cookie.password: ${COOKIE_PASS}
opendistro_security.auth.type: “openid”
opendistro_security.openid.connect_url: “https:///auth/realms//.well-known/openid-configuration”
opendistro_security.openid.client_id: “kibana”
opendistro_security.openid.client_secret: “”
opendistro_security.openid.root_ca: “/usr/share/kibana/config/keycloak-root-ca.pem”
opendistro_security.openid.scope: “openid”
opendistro_security.openid.base_redirect_url:
logging.verbose: true

I also set all possible options for logging in ES side, but those don’t seem to have any impact to logs:

status = error

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

rootLogger.level = info
rootLogger.appenderRef.console.ref = console

logger.token.name = com.amazon.dlic.auth.http.saml.Token
logger.token.level = debug

logger.opendistro_security.name = com.amazon.opendistroforelasticsearch.security
logger.opendistro_security.level = trace
logger.opendistro_security.appenderRef.rolling.ref = rolling
logger.opendistro_security.appenderRef.rolling_old.ref = rolling_old
logger.opendistro_security.additivity = false

logger.opendistro_security.name = com.amazon.dlic.auth.http.jwt
logger.opendistro_security.level = trace

Anyway, whenever trying authentication the only thing coming up in the logs (via K8S) is this:

[2020-06-29T10:27:56,203][WARN ][c.a.o.s.h.HTTPBasicAuthenticator] [es-master-2] No ‘Basic Authorization’ header, send 401 and ‘WWW-Authenticate Basic’
[2020-06-29T10:27:56,777][WARN ][c.a.o.s.a.BackendRegistry] [es-master-2] Authentication finally failed for null from 10.144.211.228:46872

I first thought that something is wrong in Kibana side, but after finally trying authentication with curl it looks that the issue is in ElasticSearch. I used following to fetch token from Keycloak and then trying to authenticate towards ES:

RESULT=curl -k --noproxy '*' -d 'client_id=kibana' -d 'username=<uid>' -d 'password=<password>' -d 'grant_type=password' -d 'client_secret=<secret>' -d 'scope=openid' 'https://<Keycloak address>/auth/realms/<realm>/protocol/openid-connect/token'
TOKEN=echo $RESULT | sed 's/.*access_token":"\([^"]*\).*/\1/'
curl -k --noproxy ‘*’ -H “Authorization: Bearer $TOKEN”

The result was exactly the same as trying with Kibana. So something seems to click between ES and Keycloak, but I just can’t figure what. Everything looks to be OK in keycloak’s client config and I also allowed “Web Origins” from * just in case.

Note: Authenthication with Keycloak works fine with Grafana. so IdP side looks to be OK.

Any ideas what to try next?

I forgot to mention that access token provided by Keycloak looks OK:

{
“exp”: 1593424216,
“iat”: 1593423916,
“jti”: “14d3fab1-ba86-4ba5-bd51-617473b8313a”,
“iss”: “https://keycloak address/auth/realms/realm”,
“aud”: “account”,
“sub”: “e6fdfbc4-d552-4824-8466-92249601c496”,
“typ”: “Bearer”,
“azp”: “kibana”,
“session_state”: “91746596-c205-4a4b-9376-e62f9507471e”,
“acr”: “1”,
“allowed-origins”: [“*”],
“realm_access”: {
“roles”: [“offline_access”, “uma_authorization”]
},
“resource_access”: {
“account”: {
“roles”: [“manage-account”, “manage-account-links”, “view-profile”]
}
},
“scope”: “openid email profile”,
“email_verified”: false,
“name”: “XXX”,
“preferred_username”: “xxx”,
“given_name”: “XXX”,
“email”: “xxx@xxx.com
}

I had finally time to continue troubleshooting above issue, but still no luck with OIDC. However, I managed to get better logs out from ES after setting rootLogger.level to debug. For some reason increasing log level only for security module didn’t not just work.

Anyway, below is logged in situation when I try connection to ES with curl by using OIDC bearer token (Kibana seems to have issues with OIDC in 1.10.1):

[2020-10-26T16:21:57,212][DEBUG][c.a.o.s.a.BackendRegistry] [es-master-2] Check authdomain for rest internal/0 or 2 in total
[2020-10-26T16:21:57,212][WARN ][c.a.o.s.h.HTTPBasicAuthenticator] [es-master-2] No ‘Basic Authorization’ header, send 401 and ‘WWW-Authenticate Basic’
[2020-10-26T16:21:57,212][DEBUG][c.a.o.s.a.BackendRegistry] [es-master-2] Check authdomain for rest noop/1 or 2 in total
[2020-10-26T16:21:57,233][DEBUG][c.a.o.s.a.BackendRegistry] [es-master-2] ‘ElasticsearchSecurityException[Authentication backend failed]’ extracting credentials from jwt-key-by-oidc http authenticator
org.elasticsearch.ElasticsearchSecurityException: Authentication backend failed
at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials0(AbstractHTTPJwtAuthenticator.java:111) ~[opendistro_security-1.10.1.0.jar:1.10.1.0]
at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.access$000(AbstractHTTPJwtAuthenticator.java:47) ~[opendistro_security-1.10.1.0.jar:1.10.1.0]
at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:90) ~[opendistro_security-1.10.1.0.jar:1.10.1.0]
at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:87) ~[opendistro_security-1.10.1.0.jar:1.10.1.0]
at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials(AbstractHTTPJwtAuthenticator.java:87) ~[opendistro_security-1.10.1.0.jar:1.10.1.0]
at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:411) [opendistro_security-1.10.1.0.jar:1.10.1.0]
at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter.checkAndAuthenticateRequest(OpenDistroSecurityRestFilter.java:177) [opendistro_security-1.10.1.0.jar:1.10.1.0]
at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter.access$000(OpenDistroSecurityRestFilter.java:66) [opendistro_security-1.10.1.0.jar:1.10.1.0]
at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter$1.handleRequest(OpenDistroSecurityRestFilter.java:113) [opendistro_security-1.10.1.0.jar:1.10.1.0]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:236) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:318) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176) [elasticsearch-7.9.1.jar:7.9.1]
at com.amazon.opendistroforelasticsearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opendistro_security-1.10.1.0.jar:1.10.1.0]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:318) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:308) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-7.9.1.jar:7.9.1]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) [transport-netty4-client-7.9.1.jar:7.9.1]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.9.1.jar:7.9.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-26T16:21:57,235][DEBUG][c.a.o.s.a.BackendRegistry] [es-master-2] User still not authenticated after checking 2 auth domains
[2020-10-26T16:21:57,235][WARN ][c.a.o.s.a.BackendRegistry] [es-master-2] Authentication finally failed for null from 10.144.190.240:42016

So, does above error indicate that ES is not able to connect to Keycloak or is this about something else?

Well, I got it finally working at least with curl:

curl -k --noproxy ‘*’ -H “Authorization: Bearer $TOKEN” “https://elastic
{“error”:{“root_cause”:[{“type”:“security_exception”,“reason”:“no permissions for [cluster:monitor/main] and User [name=xxx, backend_roles=, requestedTenant=null]”}],“type”:“security_exception”,“reason”:“no permissions for [cluster:monitor/main] and User [name=xxx, backend_roles=, requestedTenant=null]”},“status”:403}

And after configuring permissions (had to give * as external identity for the time being, for some reason exact user ID didn’t work)

curl -k --noproxy ‘*’ -H “Authorization: Bearer $TOKEN” “https://elastic
{
“name” : “es-master-1”,
“cluster_name” : “logs”,
“cluster_uuid” : “zsaKCh5SSK-cWbZ8JiMJ5A”,
“version” : {
“number” : “7.9.1”,
“build_flavor” : “oss”,
“build_type” : “tar”,
“build_hash” : “083627f112ba94dffc1232e8b42b73492789ef91”,
“build_date” : “2020-09-01T21:22:21.964974Z”,
“build_snapshot” : false,
“lucene_version” : “8.6.2”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}

Looks that the documentation given for OIDC config here is not correct:

I stumbled to this and it finally helped. I used the last config option presented in it:

https://github.com/opendistro-for-elasticsearch/security/issues/419

Did you find what should External Identity be for OIDC user? Have the same issue. When mapping is done through yaml file user get placed in “Internal user” section even though there is no such Internal user. It is impossible to add OIDC user to the same section through GUI

Well, External Identity was not involved at end, as it was not related to users. I got it working by adding new internal user via GUI with same name as OIDC user. After that it was then possible to use user in role definitions. I just used a random password when creating the user.

After above was done, I was able to use REST endpoints & indexes allowed for user via curl.

It is not very convenient in my mind - you need to add external (OIDC ) users to internal database and than configure permissions. I am using different approach - in Kibana I create role with permissions and in keycloak I create role with the same name as in kibana, create group and add role to the group. Now I can just add new OIDC user to this group and role with the same name in kibana will be assigned during login (just need to create user role mapping in OIDC client configuration in order to pass roles in JWT )

Hmm, thanks about the idea. I think that tried that but didn’t get it working, perhaps because I’m federating users from corporate LDAP to Keycloak. I think that I didn’t just find a way to define groups because of that, but I have to take a second look. However, I managed to define roles applied for anybody who has account and can thus log in. At the end I have to define additional roles only for few specific users, which means that I don’t to create too many internal users.

We actually are using the same federated users from corporate LDAP. We defined default group assignment for all authenticated through LDAP users too. Because of corporate policies we do not have ability to define groups in corporate LDAP servers and ended up just manually assigning groups in Keycloak when user login for the first time. But if you do have rights to create and join users to groups in corporate side you may create groups with the same names as roles in kibana and than use role mapping in federated provider configuration to map LDAP group to kibana role

@JiiHoo Is this now resolved? The best way is to use groups in Keycloak which are then passed via JWT and mapped to backend roles in elastic, just as @mmamaenko has done.

Yes, it has been working for me fine for a while. I haven’t got time to try with external groups yet, as current setup is enough for my needs. But will take a look into it later on.

Hi, I’m facing the same issue in my opensearch cluster with opensearch dashboard and keycloak ad IdP.

opensearch version: 2.19.1
keycloak version: 24.1
opensearch-dashboard: 2.19.1

opensearch and opensearch-dashboard share the same keycloak client which has a protocol mapper role2claim with the role_key opensearch_roles.
The client has also the role admin as preliminary test, which has been added in the opensearch-security yaml file.

The opensearch node and the opensearch-dashboard are placed in to different virtual machines and they can communicates without problems.

the issue is that I got Authentication finally failed for null when I try to login in the opensearch dashboard through keycloak.

I run securityadmin.sh after any change of the security configuration and after each restart of openserach and the scripts apply the configuration successfully.

I found several close issue, but none of them solve my issue.

Below my configuration files:

opensearch-dashboard.yml

home.disableWelcomeScreen: true
home.disableNewThemeModal: true

opensearch.username: "kibanaserver"
opensearch.password: "<password>"
opensearch.ssl.certificateAuthorities: [ "/usr/share/opensearch-dashboards/rootCA.pem" ]
opensearch.ssl.verificationMode: none
opensearch_security.auth.type: "openid"
opensearch_security.openid.connect_url: "<wellknown url to my IsP"
opensearch_security.openid.client_id: "opensearch-keycloak-client"
opensearch_security.openid.client_secret: "<keycloak-client-secret>"
opensearch_security.openid.scope: "openid email profile offline_access roles"
opensearch_security.openid.trust_dynamic_headers: true
opensearch_security.openid.base_redirect_url: "https://opensearch.mydomain.org"
opensearch_security.openid.root_ca: "/usr/share/opensearch-dashboards/ca-certificates.crt"

opensearch.yml

plugins.security.authcz.admin_dn:
  - "OU=root@internal-ca.cloud.mydomain.org,O=mkcert development certificate"


plugins.security.disabled: False
plugins.security.ssl.http.enabled: True

plugins.security.ssl.http.pemtrustedcas_filepath: /opt/opensearch/config/rootCA.pem
plugins.security.ssl.http.pemcert_filepath: /opt/opensearch/config/opensearch.pem
plugins.security.ssl.http.pemkey_filepath: /opt/opensearch/config/opensearch.key

plugins.security.ssl.transport.pemtrustedcas_filepath: /opt/opensearch/config/rootCA.pem
plugins.security.ssl.transport.pemcert_filepath: /opt/opensearch/config/opensearch.pem
plugins.security.ssl.transport.pemkey_filepath: /opt/opensearch/config/opensearch.key

config.yml

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
      http:
      anonymous_auth_enabled: false
 authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            openid_connect_idp:
              enable_ssl: true
              pemtrustedcas_filepath: /etc/ssl/certs/ca-certificates.crt
              verify_hostnames: false
            roles_key: opensearch_roles
            openid_connect_url: <wellknown url to my IdP>
            client_id: opensearch-keycloak-client
            client_secret: <keycloak-client-secret>
        authentication_backend:
          type: noop

roles_mapping.yml

all_access:
  reserved: false
  backend_roles:
    - "admin"
  description: "Maps admin to all_access"

kibana_server:
  reserved: true
  users:
  - "kibanaserver"

kibana_user:
  reserved: false
  backend_roles:
  - "kibanauser"
  description: "Maps kibanauser to kibana_user"

internal_users.yml

admin:
  hash: "<password>"
  reserved: true
  backend_roles:
  - "admin"
  description: "Demo admin user"


kibanaserver:
  hash: "<password>"
  reserved: false
  description: "Demo OpenSearch Dashboards user"

if I issue

curl -v  -XGET https://localhost:9200 -u 'admin:<password>' --insecure

no issue appears.

If I try to connect to opensearch dashboard with a keycloak user with the admin role mapped, I receive the following error in the dashboard log:

OpenId authentication failed: Error: Authentication Exception

and in the opensearch node log:

No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2025-04-30T11:44:36,075][WARN ][o.o.s.a.BackendRegistry  ] [opensearch] Authentication finally failed for null from <server:port>

@colbacc8 did you have a look at Keycloak integration, if the solution doesn’t work for you, can you share your jwt token and openID Client export?

Dear @Anthony, I’ve solved the issue yesterday evening.

the problem was the certification chain which was not fully appropriate. I had to regenerate the public certificates (I use letsencrypty) and merge the certificates in a pem file. Finally, I used the pem file in the opensearch.yml and config.yml files of opensearch.

1 Like