OS-Dashboards login redirecting when using Keycloak and self-signed certs

Security
1

When trying to login to Opensearch Dashboards, dashboard is not loading, with an “The page isnt working, <kibana/os-dash>.internal.lan redirected too many times” error. Login is using OIDC configuration with Keycloak as the SSO connect provider. Can see that a JWT token is being returned to Opensearch with a valid role that Opensearch can use. Using truststore configured in Opensearch.yml. OSS, OS-Dash running as containers, v1.2.0 and default configuration, except that self-signed certs are used everywhere.

When I put the truststore in as runtime parameter (not an Opensearch.yml ssl configuration), SSO from OS-Dash to Opensearch works as expected. Error I am getting in Opensearch is:

[2022-01-15T11:06:31,459][DEBUG][c.a.d.a.h.j.k.SelfRefreshingKeySet] [ossearch.internal.lan] performRefresh(pNYNaCIKS5HmZcc8zs-TMtjPnC_cY04BVbJ5amioNhI)
[2022-01-15T11:06:31,459][INFO ][c.a.d.a.h.j.k.SelfRefreshingKeySet] [ossearch.internal.lan] Performing refresh 1
[2022-01-15T11:06:31,614][INFO ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [ossearch.internal.lan] com.amazon.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Authentication backend failed
[2022-01-15T11:06:31,614][WARN ][o.o.s.h.HTTPBasicAuthenticator] [ossearch.internal.lan] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-15T11:06:31,615][WARN ][o.o.s.a.BackendRegistry  ] [ossearch.internal.lan] Authentication finally failed for null from 10.0.5.8:45846
[2022-01-15T11:06:31,614][WARN ][c.a.d.a.h.j.k.SelfRefreshingKeySet] [ossearch.internal.lan] KeySetProvider threw error
com.amazon.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Error while getting https://sso.internal.lan/auth/realms/master/.well-known/openid-configuration: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:149) ~[opensearch-security-1.2.0.0.jar:1.2.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.get(KeySetRetriever.java:70) ~[opensearch-security-1.2.0.0.jar:1.2.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.SelfRefreshingKeySet$1.run(SelfRefreshingKeySet.java:214) [opensearch-security-1.2.0.0.jar:1.2.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

And in Opensearch Dashboards:

log   [11:10:16.305] [error][data][opensearch] [ConnectionError]: connect ECONNREFUSED ossearch.internal.lan:9200
log   [11:10:16.728] [error][data][opensearch] [ConnectionError]: connect ECONNREFUSED ossearch.internal.lan:9200
...

Note, I have verified that the truststore in the running OSS container works from OSS to keycloak, by adding SSLPoke to image and testing:

verifying truststore on the broken instance (the one with above error); mounting SSLPoke and using truststore.jks:

Also, if I add -Djavax.net.ssl.trustStore=/usr/share/opensearch/config/truststore.jks to the container run command, I am not getting this issue (ie go to OSS url, redirects to keycloak, JWT token is returned, and I am logged into OS-Dashboards:

docker run  -e OPENSEARCH_JAVA_OPTS="-Djavax.net.ssl.trustStore=/usr/share/opensearch/config/truststore.jks" opensearchproject/opensearch:1.2.0
1 Like
2

@searchymcsearchface this is a security issue. Could you move it to the proper group?

@damo Please share your config.yml.

1 Like
3

@pablo:

node.name: oss1.internal.lan
cluster.name: docker-cluster
network.host: 0.0.0.0

discovery.seed_hosts:
  - oss1.internal.lan
  - oss2.internal.lan

cluster.initial_master_nodes:
  - oss1.internal.lan
  - oss2.internal.lan

plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.truststore_filepath: truststore.jks
plugins.security.ssl.transport.truststore_password: XXXXXXXX
plugins.security.ssl.transport.keystore_filepath: keystore.jks
plugins.security.ssl.transport.keystore_password: XXXXXXXX
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.truststore_filepath: truststore.jks
plugins.security.ssl.http.truststore_password: XXXXXXXX
plugins.security.ssl.http.keystore_filepath: keystore.jks
plugins.security.ssl.http.keystore_password: XXXXXXXX

plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=oss1.internal.lan'
  - 'CN=oss2.internal.lan'
plugins.security.nodes_dn:
  - 'CN=oss1.internal.lan'
  - 'CN=oss2.internal.lan'

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpointernal.", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3

kind regards;

4

@damo this is elasticsearch.yml. I’d like to see your config.yml file which holds security plugin configuration.

5

thanks;

---
_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: openid
          challenge: false
          config:
            enable_ssl: true
            verify_hostnames: false
            subject_key: name
            roles_key: roles
            openid_connect_url: https://sso.internal.lan/auth/realms/master/.well-known/openid-configuration
            skip_users:
            - kibanaserver
        authentication_backend:
          type: noop
      ldap1:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: true
        order: 3
          ...
    authz:
      ldap:
        description: "ldap"
        http_enabled: true
        transport_enabled: true
        authorization_backend:
          type: ldap
          config:
            enable_ssl: false
            enable_start_tls: false
6

@damo When you use self-signed certificates you have to specify IDP’s (keycloak) certificate in config.yml.

Try below config

  openid_auth_domain:
    http_enabled: true
    transport_enabled: true
    order: 4
    http_authenticator:
      type: openid
      challenge: false
      config:
        verify_hostnames: false
        subject_key: name
        roles_key: roles
        openid_connect_url: https://sso.internal.lan/auth/realms/master/.well-known/openid-configuration
		openid_connect_url.pemtrustedcsa_filepath: <filepath_to_keycloak_certificate>
		openid_connect_url.enable_ssl: true
        skip_users:
        - kibanaserver
    authentication_backend:
      type: noop
7

Thank you @pablo . Unfortunately not yet working for me. Im pretty sure I tried this before, along with the method of not using truststores but cert filepaths directly, and in config.yml above, including the certs as inline data. Still could have something wrong of course.

After following above, redirect message from OS-Dashboards/kibana still same, and still getting …SunCertPathBuilderException: unable to find valid certification path to requested target… in the Opensearch container stacktrace.

FYI, this is digging deeper into the state of the container:

$ pwd
/usr/share/opensearch/plugins/opensearch-security
$ cat /etc/system-release
Amazon Linux release 2 (Karoo)
$  
$ pwd
/usr/share/opensearch/plugins/opensearch-security
$ 
$ grep -r connect_url .
./securityconfig/config.yml:            openid_connect_url: https://sso.internal.lan/auth/realms/master/.well-known/openid-configuration
./securityconfig/config.yml:            openid_connect_url.pemtrustedcsa_filepath: /usr/share/opensearch/config/truststore.jks
./securityconfig/config.yml:            openid_connect_url.enable_ssl: true
$ 
$ java -Djavax.net.ssl.trustStore=/usr/share/opensearch/config/truststore.jks -jar /tmp/utils/SSLPoke-1.0.jar "sso.internal.lan" 9200
Successfully connected
$ 
$ keytool -list -storepass $PASS -keystore /usr/share/opensearch/config/truststore.jks
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

lan, Dec 9, 2021, trustedCertEntry, 
Certificate fingerprint (SHA-256): 08:13:09:10:5D:62:16:72:46:79:2C:94:A9:0D:40:36:75:AB:42:66:E0:4D:1F:11:2B:1E:3D:59:AC:AD:1C:47
mykey, Dec 10, 2021, trustedCertEntry, 
Certificate fingerprint (SHA-256): 81:23:CC:6C:75:40:10:21:59:1E:BE:B6:8B:97:F2:E1:E8:62:87:7F:0C:D9:24:0B:CE:52:09:10:D2:6D:E0:0D
$