Configuration for SAML login in OpenSearch with Keycloak

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OS: 2.14.0

I can’t seem to manage to tie correctly OpenSearch - SAML - Keycloak.
I’m currently getting a few 401 responses when trying to use the button to login with SAML.
My OpenSearch service is active through a virtual machine set up for me, and Keycloak runs locally on port 8080.

Configuration:

authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      saml_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 0
        http_authenticator:
          type: saml
          challenge: true
          config:
            idp:
              metadata_url: 'http://localhost:8080/realms/os/protocol/saml'
            sp:
              entity_id: opensearch.devenv.dev
            kibana_url: '//my-domain/app/dashboards'
            roles_key: Role
        authentication_backend:
          type: noop
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: false
opensearch_security.multitenancy.tenants.preferred: [Global, Private]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false

#Enable SAML login
opensearch_security.auth.multiple_auth_enabled: true
opensearch_security.auth.type: ["basicauth", "saml"]
server.xsrf.allowlist: ["/_opendistro/_security/saml/acs/idpinitiated", "/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/logout", "/_plugins/_security/saml/acs/idpinitiated", "/_plugins/_security/saml/acs", "/_plugins/_security/saml/logout"]

I created a client in Keycloak with SAML protocol.
Client ID: ‘opensearch.devenv.dev’
Root url: ‘//my-domain’
Valid redirect URIs: ‘//my-domain/*’
IDP-Initiated SSO URL name: ‘//my-domain/_opendistro/_security/saml/acs/idpinitiated’
Master SAML Processing URL: ‘//my-domain/_opendistro/_security/saml/acs’
Force POST binding: On
Include AuthnStatement: On
Sign documents: On

I created a mapper in Client scopes → opensearch.devenv.dev-dedicated. The name is Role, category: Role Mapper, type: Role list

Relevant Logs or Screenshots:
These are my 401 responses:
"res":{"statusCode":401,"responseTime":3,"contentLength":9},"message":"GET /api/v1/auth/type?dataSourceId= 401 3ms - 9.0B"}

"res":{"statusCode":401,"responseTime":3,"contentLength":9},"message":"GET /api/v1/multitenancy/tenant 401 3ms - 9.0B"}

"res":{"statusCode":401,"responseTime":2,"contentLength":9},"message":"GET /api/v1/configuration/account 401 2ms - 9.0B"}

"res":{"statusCode":401,"responseTime":2,"contentLength":9},"message":"GET /api/v1/configuration/account?dataSourceId= 401 2ms - 9.0B"}

I’m obviously changing ‘my-domain’ with the correct value.
I am running Keycloak in Debug and i see nothing happening when i try to login in OpenSearch.

I had to cancel some ‘https’ because it’s my first post.
Any help would be great.

I remembered i have some logs from OpenSearch in my terminal aswell.

[2024-08-19T14:47:17,810][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [os0.devenv.dev] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_29: Next refresh cycle for metadata provider 'http://localhost:8080/realms/os/protocol/saml' will occur on '2024-08-19T12:48:17.810067575Z' ('2024-08-19T14:48:17.810067575+02:00[Europe/Rome]' local time)
[2024-08-19T14:47:20,139][ERROR][o.o.s.m.r.i.HTTPMetadataResolver] [os0.devenv.dev] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_30: Error retrieving metadata from http://localhost:8080/realms/os/protocol/saml: Connect to localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused
[2024-08-19T14:47:20,139][ERROR][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [os0.devenv.dev] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_30: Error occurred while attempting to refresh metadata from 'http://localhost:8080/realms/os/protocol/saml'
net.shibboleth.utilities.java.support.resolver.ResolverException: Error retrieving metadata from http://localhost:8080/realms/os/protocol/saml
at org.opensaml.saml.metadata.resolver.impl.HTTPMetadataResolver.fetchMetadata(HTTPMetadataResolver.java:239) ~[opensaml-saml-impl-4.3.0.jar:?]
        at com.amazon.dlic.auth.http.saml.SamlHTTPMetadataResolver.access$001(SamlHTTPMetadataResolver.java:31) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at com.amazon.dlic.auth.http.saml.SamlHTTPMetadataResolver.lambda$fetchMetadata$0(SamlHTTPMetadataResolver.java:44) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
        at com.amazon.dlic.auth.http.saml.SamlHTTPMetadataResolver.fetchMetadata(SamlHTTPMetadataResolver.java:44) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensaml.saml.metadata.resolver.impl.AbstractReloadingMetadataResolver.refresh(AbstractReloadingMetadataResolver.java:364) [opensaml-saml-impl-4.3.0.jar:?]
        at org.opensaml.saml.metadata.resolver.impl.AbstractReloadingMetadataResolver$RefreshMetadataTask.run(AbstractReloadingMetadataResolver.java:685) [opensaml-saml-impl-4.3.0.jar:?]
        at java.base/java.util.TimerThread.mainLoop(Timer.java:566) [?:?]
        at java.base/java.util.TimerThread.run(Timer.java:516) [?:?]
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
        at org.opensaml.saml.metadata.resolver.impl.HTTPMetadataResolver.fetchMetadata(HTTPMetadataResolver.java:212) ~[opensaml-saml-impl-4.3.0.jar:?]
        ... 8 more
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.connect0(Native Method) ~[?:?]
        at java.base/sun.nio.ch.Net.connect(Net.java:589) ~[?:?]
        at java.base/sun.nio.ch.Net.connect(Net.java:578) ~[?:?]
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:583) ~[?:?]
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
        at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?]
        at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
        at org.opensaml.saml.metadata.resolver.impl.HTTPMetadataResolver.fetchMetadata(HTTPMetadataResolver.java:212) ~[opensaml-saml-impl-4.3.0.jar:?]
        ... 8 more
[2024-08-19T14:47:20,140][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [os0.devenv.dev] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_30: Next refresh cycle for metadata provider 'http://localhost:8080/realms/os/protocol/saml' will occur on '2024-08-19T12:48:20.139883926Z' ('2024-08-19T14:48:20.139883926+02:00[Europe/Rome]' local time)

Hey @RayleighSDK

Look like a connection issue.

Error retrieving metadata from http://localhost:8080/realms/os/protocol/saml

perhaps try something that can reach of the network , IP Address or FQDN instead of localhost.

I had issues awhile back, maybe this might help.

Hi @Gsmitt @RayleighSDK This is my working config.

      saml_auth:
        order: 1
        description: "SAML provider"
        http_enabled: true
        transport_enabled: false
        http_authenticator:
          type: "saml"
          challenge: true
          config:
            idp:
              metadata_url: "http://keycloak.pablo.local:8080/realms/opensearch/protocol/saml/descriptor"
              entity_id: "http://keycloak.pablo.local:8080/realms/opensearch"
            sp:
              entity_id: "docker2-saml"
            kibana_url: "https://docker2.pablo.net:5601/"
            roles_key: "Role"
            exchange_key: "123"

Your metadata_url is missing descriptor at the end. Just test this link in the browser. The link must provide xml descriptor file.

Also add this line to opensearch_dashboards.yml

opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
1 Like

is ‘exchange_key’ mandatory?
At the moment i had to make some changes. First of all i configured Keycloak on its own virtual machine, with valid certificates, with ansible. Now i just have to make the connection between the 2 machines happen.
The configuration itself shouldn’t be that different.

saml_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: saml
          challenge: True
          config:
            idp:
              #enable_ssl: True
              #pemtrustedcas_filepath: /etc/opensearch/my-cert.pem
              metadata_url: https://keycloak-vm:8443/realms/my-realm/protocol/saml/descriptor
              entity_id: https://keycloak-vm:8443/realms/my-realm
            sp:
              entity_id: https://opensearch-vm
            kibana_url: https://opensearch-vm
        authentication_backend:
                type: noop
#Enable SAML login
opensearch_security.auth.type: ["basicauth", "saml"]
opensearch_security.auth.multiple_auth_enabled: true
server.xsrf.allowlist: ["/_opendistro/_security/saml/acs/idpinitiated", "/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/logout", "/_plugins/_security/saml/acs/idpinitiated", "/_plugins/_security/saml/acs", "/_plugins/_security/saml/logout"]

But i still have errors.
These are from tailing the logs of opensearch:

[2024-08-26T12:09:34,160][WARN ][c.a.d.a.h.s.HTTPSamlAuthenticator] [opensearch-vm] roles_key is not configured, will only extract subject from SAML
[2024-08-26T12:09:34,212][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [opensearch-vm] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_2: New metadata successfully loaded for 'https://keycloak-vm:8443/realms/my-realm/protocol/saml/descriptor'
[2024-08-26T12:09:34,212][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [opensearch-vm] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_2: Next refresh cycle for metadata provider 'https://keycloak-vm:8443/realms/my-realm/protocol/saml/descriptor' will occur on '2024-08-26T13:09:34.179220654Z' ('2024-08-26T15:09:34.179220654+02:00[Europe/Rome]' local time)
[2024-08-26T12:09:34,213][WARN ][c.a.d.a.h.s.AuthTokenProcessorHandler] [opensearch-vm] roles_key is not configured, will only extract subject from SAML
[2024-08-26T12:09:34,214][ERROR][c.a.d.a.h.s.HTTPSamlAuthenticator] [opensearch-vm] Error creating HTTPSamlAuthenticator. SAML authentication will not work
java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null
	at java.base/java.util.Base64$Decoder.decode(Base64.java:593) ~[?:?]
	at com.amazon.dlic.auth.http.saml.AuthTokenProcessorHandler.createJwkFromSettings(AuthTokenProcessorHandler.java:260) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.amazon.dlic.auth.http.saml.AuthTokenProcessorHandler.<init>(AuthTokenProcessorHandler.java:113) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.<init>(HTTPSamlAuthenticator.java:148) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:432) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) [?:?]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:430) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:329) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:102) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:285) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:570) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.notifyConfigurationListeners(ConfigurationRepository.java:559) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:554) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.loadConfigurationWithLock(ConfigurationRepository.java:538) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:531) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:522) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.action.configupdate.TransportConfigUpdateAction.nodeOperation(TransportConfigUpdateAction.java:128) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.action.configupdate.TransportConfigUpdateAction.nodeOperation(TransportConfigUpdateAction.java:52) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:328) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:324) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:207) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:211) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:106) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:841) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:114) [opensearch-index-management-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1059) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.14.0.jar:2.14.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-08-26T12:09:34,217][WARN ][o.o.s.s.ReflectionHelper ] [opensearch-vm] Unable to enable 'com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator' due to java.lang.reflect.InvocationTargetException
[2024-08-26T12:09:34,218][ERROR][o.o.s.s.DynamicConfigModelV7] [opensearch-vm] Unable to initialize auth domain saml_auth_domain=AuthcDomain [http_enabled=true, order=1, http_authenticator=HttpAuthenticator [challenge=true, type=saml, config={idp={enable_ssl=true, pemtrustedcas_filepath=/etc/opensearch/DEVCA_DBM.pem, metadata_url=https://keycloak-vm:8443/realms/my-realm/protocol/saml/descriptor, entity_id=https://keycloak-vm:8443/realms/my-realm}, sp={entity_id=https://opensearch-vm}, kibana_url=https://opensearch-vm}], authentication_backend=AuthcBackend [type=noop, config={}], description=null] due to OpenSearchException[java.lang.reflect.InvocationTargetException]; nested: InvocationTargetException; nested: RuntimeException[java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null]; nested: NullPointerException[Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null];
org.opensearch.OpenSearchException: java.lang.reflect.InvocationTargetException
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:73) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:432) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:430) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:329) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:102) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:285) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:570) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.notifyConfigurationListeners(ConfigurationRepository.java:559) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:554) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.loadConfigurationWithLock(ConfigurationRepository.java:538) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:531) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:522) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.action.configupdate.TransportConfigUpdateAction.nodeOperation(TransportConfigUpdateAction.java:128) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.action.configupdate.TransportConfigUpdateAction.nodeOperation(TransportConfigUpdateAction.java:52) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:328) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:324) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:207) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:211) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:106) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:841) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:114) [opensearch-index-management-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1059) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.14.0.jar:2.14.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	... 29 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null
	at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.<init>(HTTPSamlAuthenticator.java:154) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	... 29 more
Caused by: java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null
	at java.base/java.util.Base64$Decoder.decode(Base64.java:593) ~[?:?]
	at com.amazon.dlic.auth.http.saml.AuthTokenProcessorHandler.createJwkFromSettings(AuthTokenProcessorHandler.java:260) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.amazon.dlic.auth.http.saml.AuthTokenProcessorHandler.<init>(AuthTokenProcessorHandler.java:113) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.<init>(HTTPSamlAuthenticator.java:148) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	... 29 more

These are from the journal of opensearch-dashboards:

Error: failed parsing SAML config
at SecurityClient.getSamlHeader (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:214:15
at runMicrotasks (<anonymous>)
Aug 26 12:12:05 opensearch-vm opensearch-dashboards[27194]: at processTicksAndRejections (internal/process/task_queues.js:95:5)
Aug 26 12:12:05 opensearch-vm opensearch-dashboards[27194]: at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/saml/routes.ts:78:30
Aug 26 12:12:05 opensearch-vm opensearch-dashboards[27194]: at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:174:44)
Aug 26 12:12:05 opensearch-vm opensearch-dashboards[27194]: at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:140:50)
Aug 26 12:12:05 opensearch-vm opensearch-dashboards[27194]: at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
Aug 26 12:12:05 opensearch-vm opensearch-dashboards[27194]: at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)
Aug 26 12:12:05 opensearch-vm opensearch-dashboards[27194]: at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)
Aug 26 12:12:05 opensearch-vm opensearch-dashboards[27194]: at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
Aug 26 12:12:05 opensearch-vm opensearch-dashboards[27194]: at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)
{"type":"log","@timestamp":"2024-08-26T10:12:05Z","tags":["error","plugins","securityDashboards"],"pid":27194,"message":"Failed to get saml header: Error: Error: failed parsing SAML config"}
{"type":"error","@timestamp":"2024-08-26T10:12:05Z","tags":[],"pid":27194,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n
{"type":"response","@timestamp":"2024-08-26T10:12:05Z","tags":[],"pid":27194,"method":"get","statusCode":500,"req":{"url":"/auth/saml/login?nextUrl=%2F&redirectHash=false","method":"get","headers":{"host":"opensearch-vm"

I’m not quite sure how to configure ‘roles_key’ if it’s really needed. I did add a role mapping in keycloak going to my client, then Client Scopes, then chosing the ‘dedicated’ to my client roles, then adding a ‘Role list’ with the name ‘Role’.

Could you try downloading the metadata descriptor file(https://keycloak-vm:8443/realms/my-realm/protocol/saml/descriptor) from Keycloak and replacing metadata_url with metadata_file?
i.e.

              metadata_file: "/usr/share/opensearch/config/descriptor"

This approach is because your Keycloak is probably using self-signed certificates. The SAML authentication doesn’t have the option to trust all certificates.

@RayleighSDK I get HTTP 500 when the exchange key is missing.

As soon as i get in the office i will try using the metadata_file tag, but i’m not quite sure that will change everything. The certificates i’m working with were provided to me by the firm where i work, and they were signed the same way as the other certificates for the other services.

The version of configuration i have now is pretty “clean”. I can’t quite remember the correct files names since i work mainly with templates, but i think my config.yml has the basic configuration plus the 2 lines of code where i enabled multifactor authentication and i say the 2 types are [“basicauth”, “saml”].
Then the (maybe) opensearch_dashboards.yml has simply “enable_ssl: true” and the permtrustedcas filepath for my Keycloak trusted certificate in the “idp” configuration. Also i added the “exchange_key”.

With this version, i get an error along the lines “Could not read entity descriptor for https://opensearch-vm”, and before that is an error of “reReAuthentication()”, or something like thay. Again, i will check again and send logs when i’m in the office.
That problem seems to have something to do with my Opensearch cluster tho, because recently, when i use ansible-playbook to configure my opensearch again with the changes i made, it gets stuck on checking that my cluster is green or yellow. I do check manually that the cluster is green, but i don’t know much about anything to be able to make a hypothesis.
My Opensearch service runs on 2 machines which are behind a load balancer(opensearch0 and opensearch1), while Keycloak runs on a single one(keycloak0).

Unfortunately i’m new both at where i work, i’m a junior who has never touched Linuxor virtual machines, never heard of ansible, and this whole ordeal is mine to deal with.
It’s a lot of trial and error, and guesses, and Copilot and ChatGPT can’t help much when there’s this level of personalization

At the moment i can’t even start my OpenSearch service because of the cluster issue i mentioned.
In my configuration file for ansible, there’s a task that checks that the cluster status is either yellow or green. It authenticates to _cluster/health using the admin details and password, and it gets back a 401 not authorized instead of 200. If i log as root on the machines myself and i do ‘curl -u ‘admin:password’ 127.0.0.1:9200/_cluster/health?pretty=true’ it works perfectly and i see:

{
  "cluster_name" : "my-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 5,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Once i let all retries play out, i get this error:
fatal: [opensearch-vm]: FAILED! => {"attempts": 60, "changed": false, "content": "Authentication finally failed", "content_length": "29", "content_type": "text/plain; charset=UTF-8", "elapsed": 0, "msg": "Status code was 401 and not [200]: HTTP Error 401: Unauthorized", "redirected": false, "status": 401, "url": "http://localhost:9200/_cluster/health?pretty=true"}

The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_ansible.legacy.uri_payload_dB54rX/ansible_ansible.legacy.uri_payload.zip/ansible/modules/uri.py", line 724, in main
fatal: [opensearch-vm]: FAILED! => {
    "attempts": 60,
    "changed": false,
    "content": "Authentication finally failed",
    "content_length": "29",
    "content_type": "text/plain; charset=UTF-8",
    "elapsed": 0,
    "invocation": {
        "module_args": {
            "attributes": null,
            "body": null,
            "body_format": "raw",
            "ca_path": null,
            "ciphers": null,
            "client_cert": null,
            "client_key": null,
            "creates": null,
            "decompress": true,
            "dest": null,
            "follow_redirects": "safe",
            "force": false,
            "force_basic_auth": false,
            "group": null,
            "headers": {},
            "http_agent": "ansible-httpget",
            "method": "GET",
            "mode": null,
            "owner": null,
            "remote_src": false,
            "removes": null,
            "return_content": true,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": null,
            "status_code": [
                200
            ],
            "timeout": 30,
            "unix_socket": null,
            "unredirected_headers": [],
            "unsafe_writes": false,
            "url": "http://localhost:9200/_cluster/health?pretty=true",
            "url_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "url_username": "admin",
            "use_gssapi": false,
            "use_netrc": true,
            "use_proxy": true,
            "validate_certs": true
        }
    },
    "msg": "Status code was 401 and not [200]: HTTP Error 401: Unauthorized",
    "redirected": false,
    "status": 401,
    "url": "http://localhost:9200/_cluster/health?pretty=true"
}

It’s for this reason that i created a script to wipe OpenSearch, and try a new clean slate:

#!/bin/bash
set -e

systemctl stop opensearch-dashboards.service || echo "OpenSearch-Dashboards not running..."
yum --assumeyes remove opensearch-dashboards || echo "OpenSearch-Dashboards rpm not installed..."

systemctl stop opensearch.service || echo "OpenSearch not running..."
yum --assumeyes remove opensearch || echo "OpenSearch rpm not installed..."

rm -rf \
    /etc/init.d/opensearch \
    /etc/opensearch \
    /etc/sysconfig/opensearch \
    /usr/lib/sysctl.d/opensearch.conf \
    /usr/lib/systemd/system/opensearch* \
    /usr/lib/tmpfiles.d/opensearch.conf \
    /var/lib/opensearch/* \
    /var/log/opensearch \
    /var/run/opensearch \
    /etc/init.d/opensearch-dashboards \
    /etc/default/opensearch-dashboards \
    /etc/opensearch-dashboards \
    /usr/lib/tmpfiles.d/opensearch-dashboards.conf \
    /var/lib/opensearch-dashboards/* \
    /var/log/opensearch-dashboards \
    /var/run/opensearch-dashboards

But apparently not even this can help me anymore.
Nothing has changed in my configuration regarding the vm or anything, at some point it just gives me this issue.
The logs aren’t much help either.
I need to fix this before i can start going back to my OpenSearch problems. I can’t even check what i said earlier and provide logs/screenshots till the service starts.

I managed to start OpenSearch and OpenSearch Dashboards.
My configuration:

authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: True
        transport_enabled: True
        order: 0
        http_authenticator:
          type: basic
          challenge: True
        authentication_backend:
          type: intern
      saml_auth_domain:
        description: "SAML protocol"
        http_enabled: True
        transport_enabled: True
        order: 1
        http_authenticator:
          type: saml
          challenge: True
          config:
            idp:
              enable_ssl: True
              pemtrustedcas_filepath: /etc/opensearch/keys/public-certificate.pem
              metadata_url: https://keycloak-vm:8443/realms/my-realm/protocol/saml/descriptor
              entity_id: https://opensearch.devenv.dev
            sp:
              entity_id: https://opensearch.devenv.dev
            kibana_url: https://opensearch.devenv.dev
            roles_key: 'Role'
            exchange_key: '123'
        authentication_backend:
          type: noop

That trustedCA unfortunately is the OpenSearch one, i’m assuming is not correct.

#Enable SAML authentication
opensearch_security.auth.type: ["basicauth", "saml"]
opensearch_security.auth.multiple_auth_enabled: true
server.xsrf.allowlist: ["/_opendistro/_security/saml/acs/idpinitiated", "/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/logout", "/_plugins/_security/saml/acs/idpinitiated", "/_plugins/_security/saml/acs", "/_plugins/_security/saml/logout"]

Logs of my errors:

[2024-08-27T13:24:07,740][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [opensearch-vm] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_7: Next refresh cycle for metadata provider 'https://keycloak-vm8443/realms/my-realm/protocol/saml/descriptor' will occur on '2024-08-27T11:25:07.737069819Z' ('2024-08-27T13:25:07.737069819+02:00[Europe/Rome]' local time)
[2024-08-27T13:24:20,314][ERROR][c.a.d.a.h.s.HTTPSamlAuthenticator] [opensearch-vm] Error creating HTTPSamlAuthenticator. SAML authentication will not work
org.opensearch.OpenSearchException: Unable to read /etc/opensearch/keys/DEVCA_DBM.pem (/etc/opensearch/keys/public-certificate.pem). Please make sure this files exists and is readable regarding to permissions. Property: idp.pemtrustedcas_filepath

[2024-08-27T13:24:20,320][ERROR][o.o.s.s.DynamicConfigModelV7] [opensearch-vm]] Unable to initialize auth domain saml_auth_domain=AuthcDomain [http_enabled=true, order=1, http_authenticator=HttpAuthenticator [challenge=true, type=saml, config={idp={enable_ssl=true, pemtrustedcas_filepath=/etc/opensearch/keys/public-certificate.pem, metadata_url=https://keycloak-vm:8443/realms/my-realm/protocol/saml/descriptor, entity_id=https://opensearch.devenv.dev}, sp={entity_id=https://opensearch.devenv.dev}, kibana_url=https://opensearch.devenv.dev, roles_key=Role, exchange_key=8a2h8ajasdfhsdiydfn7dtd6d5ashsd89a2h8ajasdHhsdiyLfn7dtd6d5ashsdI}], authentication_backend=AuthcBackend [type=noop, config={}], description=SAML protocol] due to OpenSearchException[java.lang.reflect.InvocationTargetException]; nested: InvocationTargetException; nested: RuntimeException[OpenSearchException[Unable to read /etc/opensearch/keys/public-certificate.pem (/etc/opensearch/keys/public-certificate.pem). Please make sure this files exists and is readable regarding to permissions. Property: idp.pemtrustedcas_filepath]]; nested: OpenSearchException[Unable to read /etc/opensearch/keys/public-certificate.pem (/etc/opensearch/keys/public-certificate.pem). Please make sure this files exists and is readable regarding to permissions. Property: idp.pemtrustedcas_filepath];

If i change to ‘metadata_file’:

config:
            idp:
              metadata_file: /etc/opensearch/keycloak_metadata.xml
              entity_id: https://opensearch.devenv.dev

I get these logs:

[2024-08-27T13:56:24,997][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [opensearch-vm] Metadata Resolver SamlFilesystemMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_12: New metadata successfully loaded for '/etc/opensearch/keycloak_metadata.xml'
[2024-08-27T13:56:24,997][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [opensearch-vm] Metadata Resolver SamlFilesystemMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_12: Next refresh cycle for metadata provider '/etc/opensearch/keycloak_metadata.xml' will occur on '2024-08-27T14:56:24.994262049Z' ('2024-08-27T16:56:24.994262049+02:00[Europe/Rome]' local time)
[2024-08-27T13:56:42,413][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [opensearch-vm] Metadata Resolver SamlFilesystemMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_13: New metadata successfully loaded for '/etc/opensearch/keycloak_metadata.xml'
[2024-08-27T13:56:42,413][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [opensearch-vm] Metadata Resolver SamlFilesystemMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_13: Next refresh cycle for metadata provider '/etc/opensearch/keycloak_metadata.xml' will occur on '2024-08-27T14:56:42.410252995Z' ('2024-08-27T16:56:42.410252995+02:00[Europe/Rome]' local time)
[2024-08-27T13:56:47,722][ERROR][o.o.s.m.r.i.HTTPMetadataResolver] [opensearch-vm] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Error retrieving metadata from https://keycloak-vm-dev:8443/realms/my-realm/protocol/saml/descriptor: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[2024-08-27T13:56:47,722][ERROR][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [opensearch-vm] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Error occurred while attempting to refresh metadata from 'https://keycloak-vm.dev:8443/realms/my-realm/protocol/saml/descriptor'
net.shibboleth.utilities.java.support.resolver.ResolverException: Error retrieving metadata from https://keycloak-vm.dev:8443/realms/my-realm/protocol/saml/descriptor

It looks like somehow it still falls back into https. Also, there are links in the .xml anyway, so i need to figure out this thing with the certificates. Any help is appreciated

@RayleighSDK Did you update the security config with securityadmin.sh script when you’ve changed to metadata_fle?

Could you run the below command to confirm that new config is there?

curl --insecure -u admin:<password> https://<OpenSearch_FQDN_or_IP>:9200/_plugins/_security/api/securityconfig?pretty

The pemtrustedcas_filepath must contain Keycloak’s certificate or its RootCA. However, I never got this working on my side and ended using metadata_file.

The metadata_url will perfectly work with commercial IDPs like Azure, AWS, Okta or PingID.

This is the response from the securityconfig:

{
  "config" : {
    "dynamic" : {
      "filtered_alias_mode" : "warn",
      "disable_rest_auth" : false,
      "disable_intertransport_auth" : false,
      "respect_request_indices_options" : false,
      "kibana" : {
        "multitenancy_enabled" : true,
        "private_tenant_enabled" : false,
        "default_tenant" : "",
        "server_username" : "kibanaserver",
        "index" : ".kibana",
        "sign_in_options" : [
          "BASIC"
        ]
      },
      "http" : {
        "anonymous_auth_enabled" : false,
        "xff" : {
          "enabled" : false,
          "internalProxies" : "192\\.168\\.0\\.10|192\\.168\\.0\\.11",
          "remoteIpHeader" : "X-Forwarded-For"
        }
      },
      "authc" : {
        "basic_internal_auth_domain" : {
          "http_enabled" : true,
          "order" : 0,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "intern",
            "config" : { }
          },
          "description" : "Authenticate via HTTP Basic against internal users database"
        },
        "saml_auth_domain" : {
          "http_enabled" : true,
          "order" : 1,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "saml",
            "config" : {
              "idp" : {
                "pemtrustedcas_filepath" : "/etc/opensearch/keys/root-certificate-for-all-services-in-the-office.pem",
                "metadata_file" : "/etc/opensearch/keycloak_metadata.xml",
                "entity_id" : "https://opensearch.devenv.dev"
              },
              "sp" : {
                "entity_id" : "https://opensearch.devenv.dev"
              },
              "kibana_url" : "https://opensearch.devenv.dev",
              "roles_key" : "Role",
              "exchange_key" : "key-copied-literally-from-opensearch-documentation"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "SAML protocol"
        }
      },
      "authz" : { },
      "auth_failure_listeners" : { },
      "do_not_fail_on_forbidden" : false,
      "multi_rolespan_enabled" : true,
      "hosts_resolver_mode" : "ip-only",
      "do_not_fail_on_forbidden_empty" : false,
      "on_behalf_of" : {
        "enabled" : false
      }
    }
  }
}

Also, i never really used securityadmin.sh because whenever i needed to check whatever changes i applied, i would simply re-run my ansible-playbook which configures Opensearch.
If i go into ‘/usr/share/opensearch/plugins/opensearch-security/tools’ and then i run ‘./securityadmin.sh’, i get:

**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
which: no java in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use

And it literally finishes with ‘will use’.

Now, after running my ansible-playbook this morning, i get this in the logs:

[2024-08-28T09:12:56,467][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [os0.devenv.dev] Metadata Resolver SamlFilesystemMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_2: New metadata successfully loaded for '/etc/opensearch/keycloak_metadata.xml'
[2024-08-28T09:12:56,468][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [os0.devenv.dev] Metadata Resolver SamlFilesystemMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_2: Next refresh cycle for metadata provider '/etc/opensearch/keycloak_metadata.xml' will occur on '2024-08-28T10:12:56.463522749Z' ('2024-08-28T12:12:56.463522749+02:00[Europe/Rome]' local time)

So nothing happens, i think? And not only that, but it says it will make a new refresh cycle 3 hours after i ran my command. So i can’t really wait 3 hours for each sweep. This part never happened till now tbh.

Also the rootCA you were saying. The one i passed, it’s the certificate that’s identical for all services in Development, so it should be okay(i suppose)

To sum it up, my configuration now is:

authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: True
        transport_enabled: True
        order: 0
        http_authenticator:
          type: basic
          challenge: True
        authentication_backend:
          type: intern
      saml_auth_domain:
        description: "SAML protocol"
        http_enabled: True
        transport_enabled: True
        order: 1
        http_authenticator:
          type: saml
          challenge: True
          config:
            idp:
              enable_ssl: true
              pemtrustedcas_filepath: /etc/opensearch/keys/{{ elk_ca_subject | cn_from_subject }}.pem
              metadata_url: https://keycloak-vm:8443/realms/my-realm/protocol/saml/descriptor
            sp:
              entity_id: https://opensearch.devenv.dev
            kibana_url: https://opensearch.devenv.dev
            roles_key: 'Role'
            exchange_key: key
        authentication_backend:
          type: noop

And all i get in the console pertaining the SAML config is:

[2024-08-28T11:54:36,491][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [os0.devenv.dev] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_5: New metadata successfully loaded for 'https://scm0.devenv.dev:8443/realms/DBM_REALM/protocol/saml/descriptor'
[2024-08-28T11:54:36,492][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [os0.devenv.dev] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_5: Next refresh cycle for metadata provider 'https://scm0.devenv.dev:8443/realms/DBM_REALM/protocol/saml/descriptor' will occur on '2024-08-28T12:54:36.474760943Z' ('2024-08-28T14:54:36.474760943+02:00[Europe/Rome]' local time)

Makes me sad not even seeing errors anymore.

I took out the entity_id just to see if it gives me an error, and it doesn’t. At this point i’m not even sure anymore how Keycloak and OpenSearch communicate or see each other.

I went and looked for the file that gives me the 'Error: failed parsing SAML config.
This is the content:

async getSamlHeader(request) {
    try {
      // response is expected to be an error
      await this.esClient.asScoped(request).callAsCurrentUser('opensearch_security.authinfo', {
        [_common.AUTH_TYPE_PARAM]: _common.AuthType.SAML
      });
    } catch (error) {
      // the error looks like
      // wwwAuthenticateDirective:
      //   '
      //     X-Security-IdP realm="Open Distro Security"
      //     location="https://<your-auth-domain.com>/api/saml2/v1/sso?SAMLRequest=<some-encoded-string>"
      //     requestId="<request_id>"
      //   '

      if (!error.wwwAuthenticateDirective) {
        throw error;
      }
      try {
        const locationRegExp = /location="(.*?)"/;
        const requestIdRegExp = /requestId="(.*?)"/;
        const locationExecArray = locationRegExp.exec(error.wwwAuthenticateDirective);
        const requestExecArray = requestIdRegExp.exec(error.wwwAuthenticateDirective);
        if (locationExecArray && requestExecArray) {
          return {
            location: locationExecArray[1],
            requestId: requestExecArray[1]
          };
        }
        throw Error('failed parsing SAML config');
      } catch (parsingError) {
        console.log(parsingError);
        throw new Error(parsingError);
      }
    }
    throw new Error(`Invalid SAML configuration.`);
  }

Could you change the basic auth challenge to false?

1 Like

I did.
The failed parsing SAML config is gone, but i still get 500.
This is what i find when i use journalctl on opensearch-dashboards service:

{"type":"log","@timestamp":"2024-08-28T10:28:27Z","tags":["error","plugins","securityDashboards"],"pid":11136,"message":"Failed to get saml header: Authentication Exception :: {\"path\":\"/_plugins/_security/authinfo\",\"query\":{\"auth_type\":\"saml\"},\"statusCode\":401,\"response\":\"Authentication finally failed\"}"}

"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:127:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:83:19)\n    at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:79:17)\n    at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http
/router/router.js:175:34)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:140:50)\n    at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http://opensearch.devenv.dev/auth/saml/login?nextUrl=%2F&redirectHash=false","message":"Internal Server Error"}

{"type":"response","@timestamp":"2024-08-28T10:28:27Z","tags":[],"pid":11136,"method":"get","statusCode":500,"req":{"url":"/auth/saml/login?nextUrl=%2F&redirectHash=false","method":"get","headers":{"host":"opensearch.devenv.dev","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://opensearch.devenv.dev/auth/saml/captureUrlFragment?nextUrl=%2F","upgrade-insecure-requests":"1","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-origin","te":"trailers","x-forwarded-proto":"https","x-unique-id":"C0A87A01:D41A_C0A87A98:01BB_66CEFBCB_0154:0451","x-forwarded-for":"192.168.122.1","connection":"close"},"remoteAddress":"192.168.122.220","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0","referer":"https://opensearch.devenv.dev/auth/saml/captureUrlFragment?nextUrl=%2F"},"res":{"statusCode":500,"responseTime":28,"contentLength":9}"message":"GET /auth/saml/login?nextUrl=%2F&redirectHash=false 500 28ms - 9.0B"}

I checked Keycloak’s logs, nothing reaches it yet.

I also tried with metadata_file instead of metadata_url. Nothing changes. Now i have metadata_file as configuration.

This is the final method where it fails(at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)):

async _execute() {
    268 
    269         this.info.acceptEncoding = this._core.compression.accept(this);
    270 
    271         try {
    272             await this._onRequest();
    273         }
    274         catch (err) {
    275             Bounce.rethrow(err, 'system');
    276             return this._reply(err);
    277         }
    278 
    279         this._lookup();
    280         this._setTimeouts();
    281         await this._lifecycle();
    282         this._reply();
    283     }

Error is at line 281

@RayleighSDK Do you even get Keycloak’s login page?

I managed to make everything work, even if it took a bit of touching Keycloak too (beside the same things you see in every guide, and a lot of things are not that clear).

As soon as i can, i’l try and make a short “guide” attached to the post i created. I could delete some replies so it’s less messy