Infinite loop with Authentik OpenID Connect

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.12 docker
debian 12
chrome/firefox

Describe the issue:

OUTDATED See next reply instead: Infinite loop with Authentik OpenID Connect - #3 by mmorg

I setup Opensearch + OpenSearch dashboard. Configured with Authentik OIDC but upon login I get stuck in an infinite loop => Authentik sends me back to opendashboard, opendashboard sends me back to authentik etc.

I see in the logs of the dashboard. I don’t see anything happening in opensearch.

OpenId authentication failed: Error: no handler found for uri [/_plugins/_security/authinfo] and method [GET] | type=log @timestamp=2024-03-27T22:23:10Z tags=["error","plugins","securityDashboards"] pid=1

After which it starts over and keeps ending up here.

I can see the tokens being created in Authentik.

I can connect to the api through the dashboard container.

I’ve been at it for quite some hours, could use another pair of eyes :slight_smile:

Configuration:

docker-compose.yml

services:
  opensearch-api:
    image: opensearchproject/opensearch:2.12.0
    environment:
      - "discovery.type=single-node"
      - "bootstrap.memory_lock=true"
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"
      - "TZ=Europe/Amsterdam"
    volumes:
      - data:/usr/share/opensearch/data
      - ./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - ./certs/opensearch.pem:/usr/share/opensearch/config/opensearch.pem
      - ./certs/opensearch-key.pem:/usr/share/opensearch/config/opensearch-key.pem
      - ./opensearch_security_internal_users.yml:/usr/share/opensearch/config/security/internal_users.yml
      - ./opensearch_security_config.yml:/usr/share/opensearch/config/security/config.yml
      - ./opensearch.yml:/usr/share/opensearch/config/opensearch.yml

    opensearch-dashboard:
      image: opensearchproject/opensearch-dashboards:2.12.0
      environment:
        - "OPENSEARCH_HOSTS=http://opensearch-api:9200"
        - "TZ=Europe/Amsterdam"
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.opensearch-dashboard.rule=Host(`opensearch.my-domain.com`)"
        - "traefik.http.routers.opensearch-dashboard.entrypoints=websecure"
        - "traefik.http.routers.opensearch-dashboard.tls=true"
        - "traefik.http.routers.opensearch-dashboard.tls.certresolver=dns"
        - "traefik.http.routers.opensearch-dashboard.service=opensearch-dashboard"
        - "traefik.http.services.opensearch-dashboard.loadbalancer.server.port=5601"
      volumes:
        - ./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml

OpenSearch API Container

/usr/share/opensearch/config/security/internal_users.yml

_meta:
  type: "internalusers"
  config_version: 2

admin:
  hash: "$2SNIP"
  reserved: true
  backend_roles:
    - "admin"
  description: "admin user"

kibanaserver:
  hash: "$2SNIP"
  reserved: true
  description: "OpenSearch Dashboards server"

/usr/share/opensearch/config/security/config.yml

basic_internal_auth_domain:
  http_enabled: true
  transport_enabled: true
  order: 0
  http_authenticator:
    type: basic
    challenge: false
  authentication_backend:
    type: internal
openid_auth_domain:
  http_enabled: true
  transport_enabled: true
  order: 1
  http_authenticator:
    type: openid
    challenge: false
    config:
      subject_key: preferred_username
      openid_connect_idp:
        enable_ssl: true
      roles_key: groups
      openid_connect_url: https://auth.my-domain.com/application/o/opensearch/.well-known/openid-configuration
  authentication_backend:
    type: noop

/usr/share/opensearch/config/opensearch.yml

cluster:
  name: opensearch-my-domain

network:
  host: 0.0.0.0

discovery:
  type: single-node

plugins:
  security:
    authcz.admin_dn: CN=admin,OU=SSL,O=Test,L=Test,C=NL
    ssl:
      transport:
        pemcert_filepath: opensearch.pem
        pemkey_filepath: opensearch-key.pem
        pemtrustedcas_filepath: root-ca.pem
        enforce_hostname_verification: false
        resolve_hostname: false
      http:
        enabled: false

OpenSearch Dashboard Container

/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml

server:
  host: "0.0.0.0"
  name: "opensearch.my-domain.com"

opensearch_security:
  auth:
    type: "openid"
  openid:
    connect_url: "https://auth.my-domain.com/application/o/opensearch/.well-known/openid-configuration"
    client_id: "SNIP"
    client_secret: "SNIP"
    scope: "openid profile email"
    logout_url: "https://auth.my-domain.com/application/o/opensearch/end-session/"
    base_redirect_url: "https://opensearch.my-domain.com"
    verify_hostnames: false

opensearch:
  username: "kibanaserver"
  password: "SNIP"

Relevant Logs or Screenshots:

Full log:

GET /auth/openid/login?code=SNIP&state=SNIP 302 527ms - 9.0B | type=response @timestamp=2024-03-27T22:23:09Z tags=[] pid=1 method=get statusCode=302 req={"url":"/auth/openid/login?code=SNIP&state=SNIP","method":"get","headers":{"host":"opensearch.my-domain.com","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","sec-ch-ua":"\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Linux\"","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-site","upgrade-insecure-requests":"1","x-forwarded-for":"192.168.1.55","x-forwarded-host":"opensearch.my-domain.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"e3557ae279a4","x-real-ip":"192.168.1.55"},"remoteAddress":"172.18.0.2","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"} res={"statusCode":302,"responseTime":527,"contentLength":9}

GET /auth/openid/login 302 1ms - 9.0B | type=response @timestamp=2024-03-27T22:23:10Z tags=[] pid=1 method=get statusCode=302 req={"url":"/auth/openid/login","method":"get","headers":{"host":"opensearch.my-domain.com","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","sec-ch-ua":"\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Linux\"","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-site","upgrade-insecure-requests":"1","x-forwarded-for":"192.168.1.55","x-forwarded-host":"opensearch.my-domain.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"e3557ae279a4","x-real-ip":"192.168.1.55"},"remoteAddress":"172.18.0.2","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"} res={"statusCode":302,"responseTime":1,"contentLength":9}

OpenId authentication failed: Error: no handler found for uri [/_plugins/_security/authinfo] and method [GET] | type=log @timestamp=2024-03-27T22:23:10Z tags=["error","plugins","securityDashboards"] pid=1

Hey @mmorg

Out of curiosity where did you find these settings for OpenSearch-dashboard file?

Normally these setting are only made in the security configuration file.

Then you run…

./securityadmin.sh

That would be in the config section for the Dashboard: OpenID Connect - OpenSearch Documentation

Those options don’t seem to be available in that config file: OpenID Connect - OpenSearch Documentation

So I found a few mistakes I had made. I fixed those and ran that command a bunch of times, now I’m getting a 401, which is a step at least.

{"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}

Updates:

API

Wrong directory

/usr/share/opensearch/config/security/internal_users.yml => /usr/share/opensearch/config/opensearch-security/internal_users.yml

Wrong directory and missing keys, the YML here threw my off because it doesn’t contain the config.dynamic.authc keys.

/usr/share/opensearch/config/security/config.yml => /usr/share/opensearch/config/opensearch-security/config.yml

_meta:
  type: "config"
  config_version: 2

# these keys are needed
config:
  dynamic:
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            openid_connect_idp:
              enable_ssl: true
              verify_hostnames: false
              # this is the last thing I tried, but it didn't work
              pemtrustedcas_filepath: /usr/share/opensearch/config/certificates/authentik.pem
            roles_key: roles
            openid_connect_url: https://auth.my-domain.com/application/o/opensearch/.well-known/openid-configuration
        authentication_backend:
          type: noop

Docker Compose

Fixed wrong paths and added every single certificate I could think of

services:
  opensearch-api:
    image: opensearchproject/opensearch:2.12.0
    environment:
      - "bootstrap.memory_lock=true"
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "TZ=Europe/Amsterdam"
    volumes:
      - data:/usr/share/opensearch/data
      - ./opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      ## certs
      - ./certs/authentik.pem:/usr/share/opensearch/config/authentik.pem
      - ./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - ./certs/admin.pem:/usr/share/opensearch/config/admin.pem
      - ./certs/admin-key.pem:/usr/share/opensearch/config/admin-key.pem
      - ./certs/node.pem:/usr/share/opensearch/config/node.pem
      - ./certs/node-key.pem:/usr/share/opensearch/config/node-key.pem
      ## security
      - ./opensearch_security_config.yml:/usr/share/opensearch/config/opensearch-security/config.yml
      - ./opensearch_security_internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml

  opensearch-dashboard:
    image: opensearchproject/opensearch-dashboards:2.12.0
    environment:
      - "TZ=Europe/Amsterdam"
    volumes:
      - ./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
      - ./certs/root-ca.pem:/usr/share/opensearch-dashboards/config/root-ca.pem

/usr/share/opensearch/config/opensearch.yml

network:
  host: 0.0.0.0

discovery:
  type: single-node

plugins:
  security:
    authcz:
      # needed to be a list
      admin_dn:
        - "CN=admin,OU=SSL,O=Test,L=Test,C=NL"
    nodes_dn:
      - "CN=node,OU=SSL,O=Test,L=Test,C=NL"
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
    audit:
      type: internal_opensearch
    restapi:
      # added probably not needed
      roles_enabled:
        ["all_access", "admin", "kibanauser", "security_rest_api_access"]
    ssl:
      transport:
        enabled: true
        pemcert_filepath: admin.pem
        pemkey_filepath: admin-key.pem
        pemtrustedcas_filepath: root-ca.pem
        enforce_hostname_verification: false
        resolve_hostname: false
      # enabled this and generated certs for it with the guide on the website
      http:
        enabled: true
        pemcert_filepath: node.pem
        pemkey_filepath: node-key.pem
        pemtrustedcas_filepath: root-ca.pem

opendistro_security:
  audit:
    config:
      disabled_rest_categories: NONE
      disabled_transport_categories: NONE

/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml

server:
  host: "0.0.0.0"
  name: "opensearch.my-domain.com"

opensearch_security:
  auth:
    type: "openid"
  openid:
    connect_url: "https://auth.my-domain.com/application/o/opensearch/.well-known/openid-configuration"
    client_id: "SNIP"
    client_secret: "SNAP"
    # added roles here and added a property mapping in authentik, added admin role to my user
    scope: "openid profile email roles"
    logout_url: "https://auth.my-domain.com/application/o/opensearch/end-session/"
    base_redirect_url: "https://opensearch.my-domain.com"
    # verify_hostnames: false

opensearch:
  hosts:
    - "https://opensearch-api:9200"
  ssl:
    # added ssl root ca for opensearch https connection
    verificationMode: certificate
    certificateAuthorities:
      - "/usr/share/opensearch-dashboards/config/root-ca.pem"
  username: "kibanaserver"
  password: "SNOOP"

With all this I’m stuck with a 401 error message when trying to login with Authentik OIDC. Basic auth works to get to the OpenSearch api with the kibanaserver username and password.

Last thing I did: I thought it had to do with the certificate of my IDP. It was using a letsencrypt signed certificate. I changed that to a self signed certificate, and added the cert to the opensearch container and in /usr/share/opensearch/config/opensearch-security/config.yml. Then ran:

bash plugins/opensearch-security/tools/securityadmin.sh -cd config/opensearch-security -icl -nhnv -cacert config/root-ca.pem -cert config/admin.pem -key config/admin-key.pem -h localhost

But still getting 401 unfortunately. Feel like I’m close but missing something

Oh I see now, thanks :+1: . I was curious because I will be working with this type of setup soon , I’m testing out a new IDP called Zitadel.

:laughing: I do the same thing.

I assume your keeping an eye on the logs when you execute SSO?

When you test your SSO by chance do you use Chrome? if so have you tried using incognito?

If you haven’t fix this by tomorrow, I will lab this out. It looks correct or i may be over looking the issue.

@mmorg I’ve noticed that your opensearch_dashboards.yml is missing the following line.

opensearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

Added, but still 401 :frowning:

Zitadel looks really cool!

Tried incognito, sends me to authentik login page, then redirects me back to opensearch after succesful auth. Then I see the 401 again :slight_smile:

Really appreciate you looking over this. I’m pretty much out of ideas :smiley: The only thing I can think of is to manually define the roles and role_mappings in the dashboard. But the default ones look good and would match pretty much what I would add. My user has the roles key in the jwt with value: admin,all_access.

Logs from SSO. I can see roles being requested so that’s good. I don’t see the client secret, I’m assuming that’s because opensearch doesn’t do direct auth but lets you get a session first. Then uses that somehow (not 100% sure how these flows should work.

{"auth_via": "session", "domain_url": "auth.my-domain.com", "event": "/if/flow/default-provider-authorization-implicit-consent/?client_id=SNIP&response_type=code&redirect_uri=https%3A%2F%2Fopensearch.my-domain.com%2Fauth%2Fopenid%2Flogin&state=zrHbmFJWXrXr-V9mWFbQaQ&scope=openid+profile+email+roles", "host": "auth.my-domain.com", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 45, "remote": "192.168.1.55", "request_id": "f4fe5bdfc0d5468ea1de0e9f2de2cca6", "runtime": 20, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-03-28T16:21:02.303903", "user": "me", "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"}
{"domain_url": null, "event": "/ws/client/", "level": "info", "logger": "authentik.asgi", "pid": 45, "remote": "192.168.1.55", "schema_name": "public", "scheme": "ws", "timestamp": "2024-03-28T16:21:02.470420", "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"}
{"action": "authorize_application", "auth_via": "session", "client_ip": "192.168.1.55", "context": {"authorized_application": {"app": "authentik_core", "model_name": "application", "name": "OpenSearch", "pk": "103f8222960948ae93503a7f39b6a41f"}, "flow": "15954af0422248e0bd7bd9ddbb05afa8", "http_request": {"args": {"client_id": "SNIP", "redirect_uri": "https://opensearch.my-domain.com/auth/openid/login", "response_type": "code", "scope": "openid profile email roles", "state": "zrHbmFJWXrXr-V9mWFbQaQ"}, "method": "GET", "path": "/api/v3/flows/executor/default-provider-authorization-implicit-consent/", "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"}, "scopes": "profile roles email openid"}, "domain_url": "auth.my-domain.com", "event": "Created Event", "host": "auth.my-domain.com", "level": "info", "logger": "authentik.events.models", "pid": 45, "request_id": "2b73320e58244fd5b8cb9c798ca12fc5", "schema_name": "public", "timestamp": "2024-03-28T16:21:02.520952", "user": {"email": "my-name@me.com", "pk": 5, "username": "me"}}
{"auth_via": "session", "domain_url": "auth.my-domain.com", "event": "Task published", "host": "auth.my-domain.com", "level": "info", "logger": "authentik.root.celery", "pid": 45, "request_id": "2b73320e58244fd5b8cb9c798ca12fc5", "schema_name": "public", "task_id": "29baac75c14c4ad0be3743bd3180407b", "task_name": "authentik.events.tasks.event_notification_handler", "timestamp": "2024-03-28T16:21:02.544312"}
{"auth_via": "session", "domain_url": "auth.my-domain.com", "event": "/api/v3/flows/executor/default-provider-authorization-implicit-consent/?query=client_id%3DSNIP%26response_type%3Dcode%26redirect_uri%3Dhttps%253A%252F%252Fopensearch.my-domain.com%252Fauth%252Fopenid%252Flogin%26state%3DzrHbmFJWXrXr-V9mWFbQaQ%26scope%3Dopenid%2Bprofile%2Bemail%2Broles", "host": "auth.my-domain.com", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 45, "remote": "192.168.1.55", "request_id": "2b73320e58244fd5b8cb9c798ca12fc5", "runtime": 52, "schema_name": "public", "scheme": "https", "status": 200, "timestamp": "2024-03-28T16:21:02.551466", "user": "me", "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"}

Logs from Dashboard show that it’s performing auth succesfully with oidc, but then fails. I can’t tell if it’s sending the Authorization header though to the API.

GET / 302 1ms - 9.0B | type=response @timestamp=2024-03-28T16:39:38Z tags=[] pid=1 method=get statusCode=302 req={"url":"/","method":"get","headers":{"host":"opensearch.my-domain.com","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","sec-ch-ua":"\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Linux\"","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1","upgrade-insecure-requests":"1","x-forwarded-for":"192.168.1.55","x-forwarded-host":"opensearch.my-domain.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"e3557ae279a4","x-real-ip":"192.168.1.55"},"remoteAddress":"172.18.0.2","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"} res={"statusCode":302,"responseTime":1,"contentLength":9}
GET /auth/openid/captureUrlFragment?nextUrl=%2F 200 1ms - 9.0B | type=response @timestamp=2024-03-28T16:39:38Z tags=[] pid=1 method=get statusCode=200 req={"url":"/auth/openid/captureUrlFragment?nextUrl=%2F","method":"get","headers":{"host":"opensearch.my-domain.com","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","sec-ch-ua":"\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Linux\"","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1","upgrade-insecure-requests":"1","x-forwarded-for":"192.168.1.55","x-forwarded-host":"opensearch.my-domain.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"e3557ae279a4","x-real-ip":"192.168.1.55"},"remoteAddress":"172.18.0.2","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"} res={"statusCode":200,"responseTime":1,"contentLength":9}
GET /auth/openid/captureUrlFragment.js 200 2ms - 9.0B | type=response @timestamp=2024-03-28T16:39:38Z tags=[] pid=1 method=get statusCode=200 req={"url":"/auth/openid/captureUrlFragment.js","method":"get","headers":{"host":"opensearch.my-domain.com","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","accept":"*/*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","referer":"https://opensearch.my-domain.com/auth/openid/captureUrlFragment?nextUrl=/","sec-ch-ua":"\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Linux\"","sec-fetch-dest":"script","sec-fetch-mode":"no-cors","sec-fetch-site":"same-origin","x-forwarded-for":"192.168.1.55","x-forwarded-host":"opensearch.my-domain.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"e3557ae279a4","x-real-ip":"192.168.1.55"},"remoteAddress":"172.18.0.2","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","referer":"https://opensearch.my-domain.com/auth/openid/captureUrlFragment?nextUrl=/"} res={"statusCode":200,"responseTime":2,"contentLength":9}
GET /auth/openid/login?nextUrl=%2F&redirectHash=false 302 1ms - 9.0B | type=response @timestamp=2024-03-28T16:39:38Z tags=[] pid=1 method=get statusCode=302 req={"url":"/auth/openid/login?nextUrl=%2F&redirectHash=false","method":"get","headers":{"host":"opensearch.my-domain.com","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","referer":"https://opensearch.my-domain.com/auth/openid/captureUrlFragment?nextUrl=/","sec-ch-ua":"\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Linux\"","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-origin","upgrade-insecure-requests":"1","x-forwarded-for":"192.168.1.55","x-forwarded-host":"opensearch.my-domain.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"e3557ae279a4","x-real-ip":"192.168.1.55"},"remoteAddress":"172.18.0.2","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","referer":"https://opensearch.my-domain.com/auth/openid/captureUrlFragment?nextUrl=/"} res={"statusCode":302,"responseTime":1,"contentLength":9}
OpenId authentication failed: Error: Authentication Exception | type=log @timestamp=2024-03-28T16:39:39Z tags=["error","plugins","securityDashboards"] pid=1
GET /auth/openid/login?code=SNIP&state=SNAP-QI 401 500ms - 9.0B | type=response @timestamp=2024-03-28T16:39:39Z tags=[] pid=1 method=get statusCode=401 req={"url":"/auth/openid/login?code=SNIP&state=SNAP-QI","method":"get","headers":{"host":"opensearch.my-domain.com","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","sec-ch-ua":"\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Linux\"","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-site","upgrade-insecure-requests":"1","x-forwarded-for":"192.168.1.55","x-forwarded-host":"opensearch.my-domain.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"e3557ae279a4","x-real-ip":"192.168.1.55"},"remoteAddress":"172.18.0.2","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"} res={"statusCode":401,"responseTime":500,"contentLength":9}

Logs from API is very interesting, I can’t tell whether it’s even trying to validate the OIDC token. It just errors out with basic auth header is missing.

[2024-03-28T17:39:39,848][WARN ][o.o.s.h.HTTPBasicAuthenticator] [8c7ade493a5a] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2024-03-28T17:39:39,848][WARN ][o.o.s.a.BackendRegistry  ] [8c7ade493a5a] Authentication finally failed for null from 172.18.0.3:34708

@mmorg I’ve installed Authentik in my lab and successfully connected with OpenSearch using OpenID authentication.

image

image

The difference between my configs and yours is the treafik configuration. I understand that treafik is a reverse proxy between OpenSearch Dashboards and Authentik.

I’ve seen scenarios where reverse proxy was dropping the Authorization header from the IdP to OpenSearch Dashboards even when OpenSearch Dashboards had that header whitelisted.

Could you try to connect OpenSearch Dashboards and Authentik directly?

I used groups instead to comply with the token.

roles_key: "groups"
{
    "iss": "https://docker6.pablo.local/application/o/opensearch/",
    "sub": "a80f936d72dcb1169570bd8da9b8fc440c8f997d559dcd7d82a41e4d2c3acb1f",
    "aud": "HA8aGtpZkFwKU6zRwZoLujX4iMsehoU3FN1C28Gf",
    "exp": 1711660907,
    "iat": 1711659107,
    "auth_time": 1711659107,
    "acr": "goauthentik.io/providers/oauth2/default",
    "amr": [
        "pwd"
    ],
    "email": "<email>",
    "email_verified": true,
    "name": "authentik Default Admin",
    "given_name": "authentik Default Admin",
    "preferred_username": "akadmin",
    "nickname": "akadmin",
    "groups": [
        "authentik Admins",
        "admin"
    ]
}

I tried this but I get stuck in the loop again.

I added to my docker-compose this:

    ports:
      - "5601:5601"

Changed in opensearch_dashboards.yml and also updated authentik redirect uris.

base_redirect_url: "http://my-domain.com:5601"

It’s just using dns now, traefik only listens on 80 and 443.

Did this too, created an admin group and added myself to it.

In the logs of the dashboard I just see a bunch of 302s. Nothing logged in the API.

@mmorg Could you run the following command on your OpenSearch pod and send the output?

curl --insecure --cert <admin_cert> --key <admin_key> -XGET https://<OpenSearch>:9200/_plugins/_security/api/securityconfig?pretty

Sure thing:

{                         
  "config" : {
    "dynamic" : {
      "filtered_alias_mode" : "warn",   
      "disable_rest_auth" : false,
      "disable_intertransport_auth" : false,
      "respect_request_indices_options" : false,
      "kibana" : {              
        "multitenancy_enabled" : true,
        "private_tenant_enabled" : true,
        "default_tenant" : "",
        "server_username" : "kibanaserver",
        "index" : ".kibana"     
      },                  
      "http" : {
        "anonymous_auth_enabled" : false,
        "xff" : {
          "enabled" : false,
          "internalProxies" : "10\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|192\\.168\\.\\d{1,3}\\.\\d{1,3}|169\\.254\\.\\d{1,3}\\.\\d{1,3}|127\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|172\\.1[6-9]{1}\\.\\d{1,3}\\
.\\d{1,3}|172\\.2[0-9]{1}\\.\\d{1,3}\\.\\d{1,3}|172\\.3[0-1]{1}\\.\\d{1,3}\\.\\d{1,3}",
          "remoteIpHeader" : "X-Forwarded-For"
        }                               
      },                                       
      "authc" : {       
        "openid_auth_domain" : {
          "http_enabled" : true,
          "order" : 1,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "openid",
            "config" : {
              "subject_key" : "preferred_username",
              "openid_connect_idp" : {
                "enable_ssl" : true,
                "verify_hostnames" : false,
                "pemtrustedcas_filepath" : "/usr/share/opensearch/config/authentik.pem"
              },
              "roles_key" : "groups",
              "openid_connect_url" : "https://auth.my-domain.com/application/o/opensearch/.well-known/openid-configuration"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          }
        },
        "basic_internal_auth_domain" : {
          "http_enabled" : true,
          "order" : 0,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "internal",
            "config" : { }
          }
        }
      },
      "authz" : { },
      "auth_failure_listeners" : { },
      "do_not_fail_on_forbidden" : false,
      "multi_rolespan_enabled" : true,
      "hosts_resolver_mode" : "ip-only",
      "do_not_fail_on_forbidden_empty" : false,
      "on_behalf_of" : {
        "enabled" : false
      }
    }
  }
}

@mmorg One more thing I couldn’t get HTTPS working in my configuration.
So I communicate with Authentik over port 80. I didn’t have to change much, only used HTTP instead of HTTPS for the Authentik URL in both config.yml and opensearch_dashboards.yml.

In the past, I had issues when my Opensearch Dashboards frontend was running on HTTP and IdP was still on HTTPS.

Hey,

There fairly new, I found them out last year when we had to move away from Keycloak. So far in my testing the source code is easy to build and they do have cloud instance for free.

I’m in!

Now I can try re-adding things 1 by 1 and see where it actually fails - I do want to run everything through SSL :slight_smile:

Thanks so much!

One last question:

The self signed certificates expire in 1 year. How do you handle rotation? And is it possible to use letsencrypt? I couldn’t find a lot of information about this.

1 Like

Now that I’m thinking about it. When I started using http, I had to clear my cookies because it was trying to overwrite a secure cookie.

This whole thing could have just been authentik sending a stale cookie or something else that’s fixable by clearing cookies.

Ok, the problem is not the cookie.

It’s the SSL on authentik. As soon as I turn that on, I go back to the 401 unauthorized.

EDIT

OKAY! I feel a little stupid, it says this in the logs when I startup the API. I didn’t see it because I was looking through portainer most of the time and it limits it to 100 lines, unless you extend it BEFORE it finishes booting:

It literally says: Error creating JWT authenticator. JWT authentication will not work

[2024-03-28T23:12:38,748][ERROR][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [b714b34258aa] Error creating JWT authenticator. JWT authentication will not work
com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from null
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:338) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:196) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:117) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:131) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:47) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:88) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:426) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) [?:?]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:424) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:323) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:101) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:285) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:430) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:419) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:402) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.initalizeClusterConfiguration(ConfigurationRepository.java:227) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.lambda$initOnNodeStart$0(ConfigurationRepository.java:318) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.OpenSearchException: Empty file path for plugins.security.ssl.transport.truststore_filepath
	at org.opensearch.security.support.PemKeyReader.checkPath(PemKeyReader.java:211) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.support.PemKeyReader.resolve(PemKeyReader.java:290) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.support.PemKeyReader.resolve(PemKeyReader.java:276) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:327) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]

	... 23 more
[2024-03-28T23:12:38,753][WARN ][o.o.s.s.ReflectionHelper ] [b714b34258aa] Unable to enable 'com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator' due to java.lang.reflect.InvocationTargetException
[2024-03-28T23:12:38,762][ERROR][o.o.s.s.DynamicConfigModelV7] [b714b34258aa] Unable to initialize auth domain openid_auth_domain=AuthcDomain [http_enabled=true, order=1, http_authenticator=HttpAuthenticator [challenge=false, type=openid, config={subject_key=preferred_username, openid_connect_idp={enable_ssl=true, verify_hostnames=false}, roles_key=groups, openid_connect_url=https//auth.my-domain.com/application/o/opensearch/.well-known/openid-configuration}], authentication_backend=AuthcBackend [type=noop, config={}], description=null] due to OpenSearchException[java.lang.reflect.InvocationTargetException]; nested: InvocationTargetException; nested: RuntimeException[com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from null]; nested: SSLConfigException[Error loading trust store from null]; nested: OpenSearchException[Empty file path for plugins.security.ssl.transport.truststore_filepath];
org.opensearch.OpenSearchException: java.lang.reflect.InvocationTargetException
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:73) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:426) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:424) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:323) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:101) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:285) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:430) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:419) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:402) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.initalizeClusterConfiguration(ConfigurationRepository.java:227) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.lambda$initOnNodeStart$0(ConfigurationRepository.java:318) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]

	... 12 more
Caused by: java.lang.RuntimeException: com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from null
	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:93) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]

	... 12 more
Caused by: com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from null
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:338) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:196) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:117) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:131) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:47) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:88) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]

	... 12 more
Caused by: org.opensearch.OpenSearchException: Empty file path for plugins.security.ssl.transport.truststore_filepath
	at org.opensearch.security.support.PemKeyReader.checkPath(PemKeyReader.java:211) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.support.PemKeyReader.resolve(PemKeyReader.java:290) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at org.opensearch.security.support.PemKeyReader.resolve(PemKeyReader.java:276) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:327) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:196) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:117) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:131) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:47) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:88) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]

	... 12 more

So final issue to get this wrapped up with SSL and everything… how can I get the API to trust my letsencrypt SSL cert on the authentik endpoints. I think that’s where it’s failing

Added letsencrypt root CA to the security config from

pemtrustedcas_filepath: /usr/share/opensearch/config/letsencrypt-root.pem

Also added to opensearch_dashboards:

root_ca: "/usr/share/opensearch-dashboards/config/letsencrypt-root.pem" 

Now I get this:

[2024-03-28T23:25:20,782][WARN ][o.o.s.h.HTTPBasicAuthenticator] [f1102c59aee0] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2024-03-28T23:25:20,818][INFO ][c.a.d.a.h.j.k.SelfRefreshingKeySet] [f1102c59aee0] Performing refresh 1
[2024-03-28T23:25:20,883][INFO ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [f1102c59aee0] com.amazon.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Authentication backend failed
[2024-03-28T23:25:20,883][WARN ][o.o.s.a.BackendRegistry  ] [f1102c59aee0] Authentication finally failed for null from 172.18.0.3:35904
[2024-03-28T23:25:20,885][WARN ][c.a.d.a.h.j.k.SelfRefreshingKeySet] [f1102c59aee0] KeySetProvider threw error
com.amazon.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Error while getting https//auth.my-domain.com/application/o/opensearch/.well-known/openid-configuration: org.apache.http.client.ClientProtocolException
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:167) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.get(KeySetRetriever.java:72) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.SelfRefreshingKeySet$1.run(SelfRefreshingKeySet.java:213) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.apache.http.client.ClientProtocolException
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:140) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	... 7 more
Caused by: org.apache.http.ProtocolException: Target host is not specified
	at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:140) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	... 7 more
[2024-03-28T23:25:22,950][WARN ][o.o.s.h.HTTPBasicAuthenticator] [f1102c59aee0] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2024-03-28T23:25:22,950][INFO ][c.a.d.a.h.j.k.SelfRefreshingKeySet] [f1102c59aee0] Performing refresh 2
[2024-03-28T23:25:22,951][INFO ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [f1102c59aee0] com.amazon.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Authentication backend failed
[2024-03-28T23:25:22,951][WARN ][o.o.s.a.BackendRegistry  ] [f1102c59aee0] Authentication finally failed for null from 172.18.0.3:35904
[2024-03-28T23:25:22,951][WARN ][c.a.d.a.h.j.k.SelfRefreshingKeySet] [f1102c59aee0] KeySetProvider threw error
com.amazon.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Error while getting https//auth.my-domain.com/application/o/opensearch/.well-known/openid-configuration: org.apache.http.client.ClientProtocolException
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:167) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.get(KeySetRetriever.java:72) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.SelfRefreshingKeySet$1.run(SelfRefreshingKeySet.java:213) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.apache.http.client.ClientProtocolException
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:140) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	... 7 more
Caused by: org.apache.http.ProtocolException: Target host is not specified
	at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:140) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	... 7 more

Also:

[2024-03-28T23:34:18,112][WARN ][c.a.d.a.h.j.k.SelfRefreshingKeySet] [0bd154a35e27] KeySetProvider threw error
com.amazon.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Error while getting https//auth.my-domain.com/application/o/opensearch/.well-known/openid-configuration: org.apache.http.client.ClientProtocolException
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:167) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.get(KeySetRetriever.java:72) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.SelfRefreshingKeySet$1.run(SelfRefreshingKeySet.java:213) [opensearch-security-2.12.0.0.jar:2.12.0.0]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.apache.http.client.ClientProtocolException
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:140) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	... 7 more
Caused by: org.apache.http.ProtocolException: Target host is not specified
	at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
	at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:140) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
	... 7 more

@mmorg I tried to replace the SSL certificate of the Authentik service (443) but I couldn’t find the way. In the Authentik documentation, letsencrypt is mentioned but I don’t know if it rotates the service certs or just certs in the keystore.
I’ll try to fight a bit with that. Keycloak was much simpler in that matter.

I also couldn’t get HTTPS connection from OpenSearch to Authentik as Authentik’s certificate didn’t contain the DNS name in the SAN.
Anyway, this is outside of the OpenSearch security plugin at this point.

Not sure how to do that. I’m using traefik as a proxy for this. And I share the traefik letsencrypt certs with Authentik. Then in the provider for opensearch in authentik I set it to the certificate that matches the domain name.

Agreed this is out of scope. I can take it from here :slight_smile:

I’m trying a few more things, I’ll edit this post if I figure it out in case anyone stumbles on this thread.

Thanks so much again!!