Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.15.0
Dashboard 2.15.0
Docker image
Describe the issue:
I’ve somewhat successfully configured SAML connector to corporate Azure Entra ID (former Azure Active Directory). But whilst trying to log in from Dashboard, I always end up with plain:
{"statusCode":500,"error":"Internal Server Error","message":"Internal Error"}
After I authenticate with Microsoft and being redirected back to Dashboard. Seems like, Opensearch receives correct response from MS and is able to generate JWT token.
This token seems to be passed back to Dashboards but then when it is being used for the first time whilst calling
const user = await this.securityClient.authenticateWithHeader(request, 'authorization', credentials.authorization);
in /plugins/securityDashboards/server/auth/types/saml/routes.js
.
Security audit log does not say anything useful, just plan LOGIN FAILED
or something similarly generic. Only thing remotely useful I was able to find was in Opensearch’s logs when I enabled debug for entire project:
[2024-07-16T15:50:22,381][DEBUG][c.o.s.a.SamlResponse ] [opensearch-node1] SAMLResponse has NameID --> my@email.com
[2024-07-16T15:50:22,381][DEBUG][c.o.s.a.SamlResponse ] [opensearch-node1] SAMLResponse has NameID Format --> urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress
[2024-07-16T15:50:22,381][DEBUG][c.o.s.a.SamlResponse ] [opensearch-node1] SAMLResponse has NameID Format --> urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress
[2024-07-16T15:50:22,388][DEBUG][c.o.s.a.SamlResponse ] [opensearch-node1] SAMLResponse has attributes: {httplink://schemas.microsoft.com/identity/claims/tenantid=[{id}], httplink://schemas.microsoft.com/identity/claims/objectidentifier=[{id}], httplink://schemas.microsoft.com/identity/claims/displayname=[Some username], httplink://schemas.microsoft.com/ws/2008/06/identity/claims/groups=[99999999-9999-9999-9999-999999999999, 99999999-9999-9999-9999-999999999998, 99999999-9999-9999-9999-999999999997, 99999999-9999-9999-9999-999999999996, 99999999-9999-9999-9999-999999999995, 99999999-9999-9999-9999-999999999994, 99999999-9999-9999-9999-999999999993, 99999999-9999-9999-9999-999999999992], httplink://schemas.microsoft.com/identity/claims/identityprovider=[httpslink://sts.windows.net/{id}/], httplink://schemas.microsoft.com/claims/authnmethodsreferences=[httplink://schemas.microsoft.com/ws/2008/06/identity/authenticationmethod/password, httplink://schemas.microsoft.com/claims/multipleauthn], httplink://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname=[Somefirstname], httplink://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname=[Somesurname], httplink://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress=[my@email.com], httplink://schemas.xmlsoap.org/ws/2005/05/identity/claims/name=[my@email.com]}
[2024-07-16T15:50:22,396][DEBUG][c.a.d.a.h.s.Token ] [opensearch-node1] Created JWT: <redacted>.<redacted>.<redacted>
{"alg":"HS512"}
{"sub":"my@email.com","nbf":1721145022,"roles":["99999999-9999-9999-9999-999999999999","99999999-9999-9999-9999-999999999998","99999999-9999-9999-9999-999999999997","99999999-9999-9999-9999-999999999996","99999999-9999-9999-9999-999999999995","99999999-9999-9999-9999-999999999994","99999999-9999-9999-9999-999999999993","99999999-9999-9999-9999-999999999992"],"saml_ni":"my@email.com","saml_nif":"email","exp":1721148622,"saml_si":"_53c1fbac-a71c-47d4-b300-8f9ac0041d00"}
[2024-07-16T15:50:22,402][DEBUG][o.o.t.TransportService ] [opensearch-node1] Action: indices:data/write/bulk[s][p]
[2024-07-16T15:50:22,407][DEBUG][o.o.s.a.BackendRegistry ] [opensearch-node1] Check authdomain for rest internal/0 or 2 in total
[2024-07-16T15:50:22,407][WARN ][o.o.s.h.HTTPBasicAuthenticator] [opensearch-node1] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2024-07-16T15:50:22,407][DEBUG][o.o.s.a.BackendRegistry ] [opensearch-node1] Check authdomain for rest internal/1 or 2 in total
[2024-07-16T15:50:22,416][DEBUG][o.o.s.a.BackendRegistry ] [opensearch-node1] Can not authenticate my@email.com due to exception
com.google.common.util.concurrent.UncheckedExecutionException: OpenSearchSecurityException[empty passwords not supported]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2087) ~[guava-32.1.3-jre.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:4019) ~[guava-32.1.3-jre.jar:?]
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933) ~[guava-32.1.3-jre.jar:?]
at org.opensearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:579) [opensearch-security-2.15.0.0.jar:2.15.0.0]
at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:331) [opensearch-security-2.15.0.0.jar:2.15.0.0]
at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:309) [opensearch-security-2.15.0.0.jar:2.15.0.0]
at org.opensearch.security.ssl.http.netty.Netty4HttpRequestHeaderVerifier.channelRead0(Netty4HttpRequestHeaderVerifier.java:91) [opensearch-security-2.15.0.0.jar:2.15.0.0]
at org.opensearch.security.ssl.http.netty.Netty4HttpRequestHeaderVerifier.channelRead0(Netty4HttpRequestHeaderVerifier.java:38) [opensearch-security-2.15.0.0.jar:2.15.0.0]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.110.Final.jar:4.1.110.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) [netty-handler-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) [netty-handler-4.1.110.Final.jar:4.1.110.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) [netty-handler-4.1.110.Final.jar:4.1.110.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) [netty-handler-4.1.110.Final.jar:4.1.110.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) [netty-codec-4.1.110.Final.jar:4.1.110.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) [netty-codec-4.1.110.Final.jar:4.1.110.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) [netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.110.Final.jar:4.1.110.Final]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.OpenSearchSecurityException: empty passwords not supported
at org.opensearch.security.auth.internal.InternalAuthenticationBackend.authenticate(InternalAuthenticationBackend.java:124) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:589) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:579) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) ~[guava-32.1.3-jre.jar:?]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) ~[guava-32.1.3-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) ~[guava-32.1.3-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) ~[guava-32.1.3-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081) ~[guava-32.1.3-jre.jar:?]
... 45 more
[2024-07-16T15:50:22,423][DEBUG][o.o.s.a.BackendRegistry ] [opensearch-node1] Cannot authenticate rest user my@email.com (or add roles) with authdomain internal/1 of [AuthDomain [backend=org.opensearch.security.auth.internal.InternalAuthenticationBackend@3fcc58f8, httpAuthenticator=org.opensearch.security.http.HTTPBasicAuthenticator@2e9e0c37, order=0, challenge=false], AuthDomain [backend=org.opensearch.security.auth.internal.InternalAuthenticationBackend@3fcc58f8, httpAuthenticator=com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator@1564b5a6, order=1, challenge=true]], try next
[2024-07-16T15:50:22,423][DEBUG][o.o.s.a.BackendRegistry ] [opensearch-node1] User still not authenticated after checking 2 auth domains
[2024-07-16T15:50:22,423][DEBUG][o.o.s.a.BackendRegistry ] [opensearch-node1] Rerequest with class com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator
[2024-07-16T15:50:22,425][DEBUG][c.o.s.a.AuthnRequest ] [opensearch-node1] AuthNRequest --> <samlp:AuthnRequest xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" ID="ONELOGIN_d999999-9999-9999-9999-b74df739882d" Version="2.0" IssueInstant="2024-07-16T15:50:22Z" Destination="https://login.microsoftonline.com/{id}/saml2" ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" AssertionConsumerServiceURL="https://my.server.com/_opendistro/_security/saml/acs"><saml:Issuer>https://my.server.com</saml:Issuer><samlp:NameIDPolicy Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified" AllowCreate="true" /></samlp:AuthnRequest>
[2024-07-16T15:50:22,426][DEBUG][o.o.s.a.BackendRegistry ] [opensearch-node1] Rerequest class com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator failed
[2024-07-16T15:50:22,427][WARN ][o.o.s.a.BackendRegistry ] [opensearch-node1] Authentication finally failed for my@email.com from 10.0.0.1:39224
Configuration:
opensearch-security/config.yml:
_meta:
type: "config"
config_version: 2
config:
dynamic:
# Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
# Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
# Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
#filtered_alias_mode: warn
#do_not_fail_on_forbidden: false
#kibana:
# Kibana multitenancy
#multitenancy_enabled: true
#private_tenant_enabled: true
#default_tenant: ""
#server_username: kibanaserver
#index: '.kibana'
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
doc/config/valve.html#Remote_IP_Valve
authc:
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
saml_auth_domain:
order: 1
description: "SAML provider"
http_enabled: true
transport_enabled: false
http_authenticator:
type: saml
challenge: true
config:
idp:
entity_id: {url}
metadata_url: {url}
verify_hostnames: true
enable.ssl: true
sp:
entity_id: {url}
authentication_backend:
type: noop
kibana_url: {url}
subject_key: "httplink://schemas.xmlsoap.org/ws/2005/05/identity/claims/name"
roles_key: "httplink://schemas.microsoft.com/ws/2008/06/identity/claims/groups"
exchange_key: '64 characters long string'
jwt_clock_skew_tolerance_seconds: 7200 # from testing earlier on - does not make any difference at all
Any ideas what I might be missing? There’s no nginx or any sort of proxy anywhere and all communication is running via HTTPS configured on both opensearch transport/rest and dashboard as well.