OIDC `pemtrustedcas_filepath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` has no effect

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 2.18.0
opensearch-k8s-operator: 2.7.0

Describe the issue:
I have an OpenSearch cluster deployed via the opensearch-k8s-operator, with Dashboards OIDC SSO configured. SSO works fine when I configure config.yml → config → dynamic → authc → openid_auth_domain → http_authenticator → config → openid_connect_idp → pemtrustedcas_content with the CA certificate content for the OIDC URL’s TLS certificate.

But when I configure pemtrustedcas_filepath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem then SSO fails. i.e. After successfully logging in and being redirected back to OpenSearch, Dashboards displays 401 Unauthorized.

According to the documentation the full file path can be used. It does not have to be relative to the config directory.

/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem is already present in the opensearch and opensearch-dashboards container images, so I owuld prefer to use that file rather than duplicate its contents in OpenSearch security configuration.

As an experiment I put the entire contents of /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem into pemtrustedcas_content, and that worked fine. So it doesn’t seem to be a problem with the file’s content.

Also /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem is a text file, not a symbolic link.

Configuration:

pemtrustedcas_filepath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

Relevant Logs or Screenshots:
Dashboards:

{"type":"log","@timestamp":"2024-11-25T15:31:34Z","tags":["error","plugins","securityDashboards"],"pid":1,"message":"OpenId authentication failed: Error: Response Error: 400 Bad Request"}
{"type":"response","@timestamp":"2024-11-25T15:31:33Z","tags":[],"pid":1,"method":"get","statusCode":302,"req":{"url":"/auth/openid/login?code=REDACTED&state=REDACTED","method":"get","headers":{"host":"REDACTED","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-CA,en-US;q=0.7,en;q=0.3","priority":"u=0, i","referer":"https://REDACTED/","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-origin","sec-fetch-user":"?1","te":"trailers","upgrade-insecure-requests":"1","x-forwarded-for":"10.42.0.1","x-forwarded-host":"REDACTED","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"traefik-d7c9c5778-794bg","x-real-ip":"10.42.0.1"},"remoteAddress":"10.42.0.8","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0","referer":"https://REDACTED/"},"res":{"statusCode":302,"responseTime":432,"contentLength":9},"message":"GET /auth/openid/login?code=REDACTED&state=REDACTED 302 432ms - 9.0B"}
{"type":"response","@timestamp":"2024-11-25T15:31:33Z","tags":[],"pid":1,"method":"get","statusCode":302,"req":{"url":"/auth/openid/login?code=REDACTED&state=REDACTED","method":"get","headers":{"host":"REDACTED","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-CA,en-US;q=0.7,en;q=0.3","priority":"u=0, i","referer":"https://REDACTED/","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-origin","sec-fetch-user":"?1","te":"trailers","upgrade-insecure-requests":"1","x-forwarded-for":"10.42.0.1","x-forwarded-host":"REDACTED","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"traefik-d7c9c5778-794bg","x-real-ip":"10.42.0.1"},"remoteAddress":"10.42.0.8","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0","referer":"https://REDACTED/"},"res":{"statusCode":302,"responseTime":432,"contentLength":9},"message":"GET /auth/openid/login?code=REDACTED&state=REDACTED 302 432ms - 9.0B"}
{"type":"response","@timestamp":"2024-11-25T15:31:34Z","tags":[],"pid":1,"method":"get","statusCode":302,"req":{"url":"/auth/openid/login","method":"get","headers":{"host":"REDACTED","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-CA,en-US;q=0.7,en;q=0.3","priority":"u=0, i","referer":"https://REDACTED/","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-origin","sec-fetch-user":"?1","te":"trailers","upgrade-insecure-requests":"1","x-forwarded-for":"10.42.0.1","x-forwarded-host":"REDACTED","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"traefik-d7c9c5778-794bg","x-real-ip":"10.42.0.1"},"remoteAddress":"10.42.0.8","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0","referer":"https://REDACTED"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"GET /auth/openid/login 302 2ms - 9.0B"}
{"type":"log","@timestamp":"2024-11-25T15:31:35Z","tags":["error","plugins","securityDashboards"],"pid":1,"message":"OpenId authentication failed: Error: Authentication Exception"}
{"type":"response","@timestamp":"2024-11-25T15:31:34Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/auth/openid/login?code=REDACTED&state=REDACTED","method":"get","headers":{"host":"REDACTED","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-CA,en-US;q=0.7,en;q=0.3","priority":"u=0, i","referer":"https://REDACTED/","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"cross-site","sec-fetch-user":"?1","te":"trailers","upgrade-insecure-requests":"1","x-forwarded-for":"10.42.0.1","x-forwarded-host":"REDACTED","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"traefik-d7c9c5778-794bg","x-real-ip":"10.42.0.1"},"remoteAddress":"10.42.0.8","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0","referer":"https://REDACTED"},"res":{"statusCode":401,"responseTime":830,"contentLength":9},"message":"GET /auth/openid/login?code=REDACTED&state=REDACTED 401 830ms - 9.0B"}
{"type":"response","@timestamp":"2024-11-25T15:31:35Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/favicon.ico","method":"get","headers":{"host":"REDACTED","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0","accept":"image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-CA,en-US;q=0.7,en;q=0.3","priority":"u=6","referer":"https://REDACTED/auth/openid/login?code=REDACTED&session_state=REDACTED3","sec-fetch-dest":"image","sec-fetch-mode":"no-cors","sec-fetch-site":"cross-site","te":"trailers","x-forwarded-for":"10.42.0.1","x-forwarded-host":"REDACTED","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"traefik-d7c9c5778-794bg","x-real-ip":"10.42.0.1"},"remoteAddress":"10.42.0.8","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0","referer":"https://REDACTED/auth/openid/login?code=REDACTED&session_state=REDACTED"},"res":{"statusCode":401,"responseTime":1,"contentLength":9},"message":"GET /favicon.ico 401 1ms - 9.0B"}

Cluster:

[2024-11-25T15:31:35,281][WARN ][o.o.s.h.HTTPBasicAuthenticator] [cluster-managers-0] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'

Screenshot:

OIDC not working
OIDC not working660×328 22.5 KB

2

Hi @paul.mossman,

How do you map your IDP roles/groups/user to internal OpenSearch roles?

Would you mind sharing your config.yml?

best,
mj

3

Hi @Mantas,

I used the operator example/demo securityconfig-secret.yaml as a template for my own securityconfig.

I made minor changes only, except for config.yml to configure OIDC:

_meta:
  type: "config"
  config_version: "2"
config:
  dynamic:
    http:
      anonymous_auth_enabled: false
    authc:
      # Basic authentication (Username/Password) is always required for the
      # REST API clients
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        authentication_backend:
          type: internal
        http_authenticator:
          type: basic
          challenge: true

      # See: https://opensearch.org/docs/latest/security/authentication-backends/openid-connect/#openid-connect-url
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0 # i.e. Before basic authentication
        authentication_backend:
          type: noop # Here we disable authentication backend since we will be using Azure AD for that
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: email # Users will be able to login with email and show it as username
            roles_key: roles # Use roles from App registrations.
            # App registration IDP metadata file URL
            openid_connect_url: "https://login.microsoftonline.com/REDACTED/v2.0/.well-known/openid-configuration"
            openid_connect_idp:
              enable_ssl: true
              pemtrustedcas_filepath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

Note that when I use pemtrustedcas_content: ... instead (which works), then that content of course appears in config.yml instead.

Thank you.

-Paul

4

@paul.mossman, what is returned as a roles in your JWT from idp to OS?

Do you have a sample of JWT?

Best,
mj

5

The user has the admin role.

How could the role be related to the problem though? The issue seems to be with OpenSearch’s HTTP GET to https://login.microsoftonline.com to retrieve the Public Key for the kid in the JWT. I’ve configured enable_ssl: true so OpenSearch tries to validate the TLS certificate of https://login.microsoftonline.com. That fails when pemtrustedcas_filepath is configured, but is successful when pemtrustedcas_content is configured.

Out of curiosity I looked for the JWT using Chrome Developer Tools, but am not able to find it.

-Paul