OpenID Login Loop (OpenID + AWS Cognito)

Hey - maybe someone can point out what I have wrong regarding my OpenID Connect setup with AWS Cognito. Any help greatly appreciated!

I am using OpenDistro For Elasticsearch v1.13.2 (amazon/opendistro-for-elasticsearch:1.13.2) but I just cannot get it to connect via OpenID to Amazon Cognito - no matter what I seem to do it appears to be stuck in a login loop. For avoidance of doubt, I am not using AWS Elasticsearch Service - this is just some simple containers.

I’ve included a picture below indicating the network activity. Kibana and Elasticsearch are configured for OpenID. I think the Cognito half is fine and it is going wrong when navigating back to Kibana, but I am not sure. I believe the user being logged in to has the admin role

When accessing Kibana, I believe the following occurs (based on the image below):

  1. Kibana is accessed, which redirects to /auth/openid/login?nextUrl=/
  2. This next page then redirects to my AWS OAuth endpoint as seen in the “location” HTTP response hader
    • https://<aws-auth-endpoint-id>.auth.eu-west-2.amazoncognito.com/oauth2/authorize?client_id=<clientid>&response_type=code&redirect_uri=https://<hostname>-elk.dev.internal:5601/auth/openid/login&state=beBx_3sG9xjezfgOHHFyfR&scope=openid
  3. Login completes on Cognito (user completes auth flow if no saved session already). Cognito redirects back to the redirect_uri location, back on Kibana along with the code URI parameter (which I have no reason to believe isn’t valid).
  4. Kibana redirects again to /auth/openid/login. Infinite loop begins.
  5. Browser detects and breaks loop after approx 5 iterations with a “The page isn’t redirecting properly” warning.

I have attempted to enable more logging but have not been able to find any useful logs. Not sure if I’ve done it wrong.

I have read quite a lot of the OpenID related posts on the forum. I see there have been others with infinite loops but I have not been able to verify if it is the same issue. In some cases Kibana did not have rights to login as kibana/kibanaserver into Elasticsearch but I don’t believe this is the case. Regardless, I’ve posted my security config files below.

Can anyone see what I’ve done wrong? Or any way I can get increased logging? The Cognito auth flow looks like it should be valid, I just can’t work out why Kibana rejects it and goes back into a loop!

Network Connectivity upon Login

/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
    authc:
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: "https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_$$REDACTED_ID$$/.well-known/openid-configuration"
            skip_users:
              - kibanaro
              - kibanaserver
              - admin
        authentication_backend:
          type: noop

      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal # Original value was "intern" but docs say "internal"?
		  
    # No authz entries actually set up -these are the defaults
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
            rolesearch: '(member={0})'
            # Specify the name of the attribute which value should be substituted with {2} above
            userroleattribute: null
            # Roles as an attribute of the user entry
            userrolename: disabled
            #userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(uid={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            #skip_users:
            #  - 'cn=Michael Jackson,ou*people,o=TEST'
            #  - '/\S*/'
      roles_from_another_ldap:
        description: "Authorize via another Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap

kibana.yml

server.host: "<kibanahostname>-elk.dev.internal"
elasticsearch.hosts: http://<elasticsearchhostname>-elk.dev.internal:9200
elasticsearch.ssl.verificationMode: none
elasticsearch.username: kibanaserver
elasticsearch.password: kibanaserver
elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

server.ssl.enabled: true
server.ssl.certificate: "/usr/share/kibana/config/server.pem.crt" # PEM
server.ssl.key: "/usr/share/kibana/config/server.pem.key" # PEM

# Use OpenIDConnect (OIDC) logon
opendistro_security.auth.type: "openid"
opendistro_security.openid.connect_url: "https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_$$REDACTED_ID$$/.well-known/openid-configuration"
opendistro_security.openid.client_id: "$$REDACTED_CLIENT_ID$$"
opendistro_security.openid.client_secret: "$$REDACTED_CLIENT_SECRET$$"
opendistro_security.openid.scope: "openid"  # Also tested with default of "openid profile email address phone" 
#opendistro_security.openid.header: "Authorization" # Defaults to "Authorization"
opendistro_security.openid.base_redirect_url: "https://<hostname>-elk.dev.internal:5601"

opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.preferred: ["Private", "Global"]
opendistro_security.readonly_mode.roles: ["kibana_read_only"]

# Use this setting if you are running kibana without https
opendistro_security.cookie.secure: true

newsfeed.enabled: false
telemetry.optIn: false
telemetry.enabled: false
security.showInsecureClusterWarning: false
map.includeElasticMapsService: false

Can you post the logs of Kibana and Elasticsearch?

  1. Logs when Kibana/Elasticsearch starts up. It will throw out some error log if openId’s endpoint has any problem(like self-sign cert) when Kibana/Elasticsearch try to connect the endpoint.
  2. Logs when you try to login. It can help you to locate the problem.it’s stuck at Kibana or Elasticsearch.

Hey,

The OpenID endpoint is not self-signed and the cert is accepted by the CA list - it’s the AWS Cognito endpoint (literally https://cognito-idp.eu-west-2.amazonaws.com),

Regarding the logs I’ve just realized I’ve made a dumb mistake - I previously couldn’t get it to print any useful logging but I’d made a typo when sorting out the logs.

I now have the following on ODFE logs:

    2021-07-27T18:21:07.814+01:00	[2021-07-27T17:21:07,814][WARN ][c.a.o.s.h.HTTPBasicAuthenticator] [<cluster-name>] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
	2021-07-27T18:21:07.815+01:00	[2021-07-27T17:21:07,815][INFO ][c.a.d.a.h.j.k.JwtVerifier] [<cluster-name>] Escaped Key ID from JWT Token
	2021-07-27T18:21:07.817+01:00	[2021-07-27T17:21:07,816][WARN ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [<cluster-name>] Failed to get subject from JWT claims, check if subject_key 'preferred_username' is correct.
	2021-07-27T18:21:07.817+01:00	[2021-07-27T17:21:07,816][ERROR][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [<cluster-name>] No subject found in JWT token
	2021-07-27T18:21:07.817+01:00	[2021-07-27T17:21:07,816][WARN ][c.a.o.s.a.BackendRegistry] [<cluster-name>] Authentication finally failed for null from 10.0.252.44:58696

I guess that seems pretty clear - it says there’s no preferred_username in the JWT token served (I had explicitly set http_authenticator.config.subject_key to it’s default value), and I guess this is the problem.

I presume this should be set with a non-changeable ID for that user (e.g. username, or something similar if username is changeable).

I think the correct value here is probably cognito:username. Looking online, apparently a username can be re-used later if freed up. If that is a problem then sub should be used which is a UUID and supposedly always unique (i.e. can’t be reused later of course, unlike cognito:username).

I’ll give this a try tomorrow.
My OpenID experience is minimal but I have used OAuth many times. Is preferred_username a common field? It’s definitely not one of the mandatory ones (I’m not sure if there is a mandatory OpenID Connect claim that fills the “username”/“user id” role?)

Links
AWS doc showing some of the JWT fields:
How can I decode and verify the signature of an Amazon Cognito JSON Web Token?

AWS support thread discussing sub and cognito:username:
Cognito User Pool field “sub” unique ?

In my case, the JWT token doesn’t has the key: ‘preferred_username’.
So I removed the subject key in config.yml and it works for me.

I had evidently misremembered what the docs said, as it does say it’ll use “the subject” (the “sub” claim) if not present - it merely mentions that “preferred_username” is a common claim that some IdP provides use.

The literal text:

The key in the JSON payload that stores the user’s name. If not defined, the subject registered claim is used. Most IdP providers use the preferred_username claim. Optional.

I did try this at one point, but I must have had a different error at the same time. I think this will likely boil down to the mistake I made with the logging config resulting in no useful logging, then various retries with different configs not noticing the real issue.
I’ll try adjusting the config soon.

FYI, I dont think this was preferred_username issue? I encountered this error every morning: my elasticsearch cluster goes offline periodically, when everything comes back online, kibana is in exactly the same loop. subject_key: preferred_username has always been set in eg, elasticsearch/securityconfig/config.yml.

In this loop state, i’m not seeing any traffic hitting, Keycloak/sso.

In this state, I completely stop my es cluster, same login URL going around.

Restarting kibana (container) fixes issue.

@nick_cloud did you solve the infinite loop ?
Thanks

Sorry, I genuinely thought I had replied to this thread already and asked for it to be closed.

In an unrelated turn of events I had to migrate from OpenSearch to Elastic-co Elasticsearch. Therefore I never actually resolved this, sorry. I do use OIDC on Elasticsearch and Kibana though, and did not experience these issues - but that’s not to say that perhaps I hadn’t made a dumb configuration mistake on the OpenSearch instance.

I see you have a similar thread for Keycloak - there were some other Keycloak threads previously that I found when I created this post.

Sorry I can’t really help further, as I never actually resolved the issue. What is clear though is that multiple people are having this issue, and OpenSearch has made connecting OIDC simpler than Elasticsearch by using the .well-known discovery document instead of configuring multiple properties, and so you’d think this would work more easily - which leads me to suspect that something isn’t quite correct here (or at least isn’t obvious) in OpenSearch.

Thanks for your answer.
There’s something happening with the OIDC configuration callback using keycloak to authenticate on Google. The google authenticator works very well but the f*** infinite loop drives me crazy.
I have other posts but no opensearch support on that. Maybe the hard configurations are better than the automatic ones.

Good luck :slight_smile: