OpenSearch Dashboards Integration with Keycloak - Issues with Refresh Tokens

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch: opensearchproject/opensearch:2.16.0. (opensearch-k8s-operator)
Opensearch Dashboard: opensearchproject/opensearch-dashboards:2.16.0
K8s: 1.26
Chrome Version 127.0.6533.100 (Official Build) (arm64)

Describe the issue:

Hi everyone!

I’ve been facing an issue while integrating OpenSearch Dashboards with Keycloak. The specific problem occurs during the authorization process through Keycloak. After entering my login credentials, I receive a 502 error in the browser.

Through multiple configuration tests, I’ve found that this issue arises when both an access_token and a refresh_token are returned in the authentication response. However, as soon as I disable the “Use refresh tokens” option in Keycloak settings or set refresh_tokens: false in the Dashboard settings, the authorization works fine, and I’m able to access the Dashboard.

In our setup, Keycloak is configured with federation to Active Directory (AD). Users and groups are synchronized with the domain, and we have the following setup: there is a group named js-devops-team that is pulled from AD, and a role mapping is configured for this group, which is mapped to the opensearch_admin role created earlier.

In real-world use cases, this solution is inconvenient because the session duration would then be equal to the token’s lifespan. Once the token expires, the user would have to log in again, and with a token lifetime of only 5 minutes, this isn’t practical. I would prefer not to disable refresh token usage or extend the access token’s lifetime.

Additionally, it’s worth mentioning that the OpenSearch Dashboard is installed separately from the operator using a Helm chart.

Has anyone else faced this issue, and if so, how did you resolve it? Any insights or suggestions would be greatly appreciated!

Thanks in advance!

Configuration:

opensearch dashboards: values.yaml

# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

# Default values for opensearch-dashboards.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

opensearchHosts: "https://opensearch-cluster.observability.svc.cluster.local:9200"
replicaCount: 1

image:
  repository: "opensearchproject/opensearch-dashboards"
  tag: "2.16.0"
  pullPolicy: "IfNotPresent"

startupProbe:
  tcpSocket:
    port: 5601
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 20
  successThreshold: 1
  initialDelaySeconds: 10

livenessProbe:
  tcpSocket:
    port: 5601
  periodSeconds: 20
  timeoutSeconds: 5
  failureThreshold: 10
  successThreshold: 1
  initialDelaySeconds: 10

readinessProbe:
  tcpSocket:
    port: 5601
  periodSeconds: 20
  timeoutSeconds: 5
  failureThreshold: 10
  successThreshold: 1
  initialDelaySeconds: 10

imagePullSecrets: []
nameOverride: ""
fullnameOverride: "os-dashboard"

serviceAccount:
  create: true
  annotations: {}
  name: ""

rbac:
  create: true

podAnnotations: {}

dashboardAnnotations: {}

extraEnvs:
  - name: OPENID_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: opensearch-oidc-secret-4
        key: client-secret

envFrom: []

secretMounts:
  - name: certs
    secretName: opensearch-cluster-http-cert
    path: /usr/share/opensearch-dashboards/certs/

extraVolumes:
  - name: sbis-crts
    secret:
      secretName: sbis-crts

extraVolumeMounts:
  - name: sbis-crts
    mountPath: /usr/share/opensearch-dashboards/certs/sbis-crts/
    readOnly: true

extraInitContainers: ""

extraContainers: ""

podSecurityContext: {}

securityContext:
  capabilities:
    drop:
      - ALL
  runAsNonRoot: true
  runAsUser: 1000

config:
  opensearch_dashboards.yml:
    # opensearch.ssl.verificationMode: none
    logging.verbose: true
    logging.root.level: debug
    server:
      name: "os-dashboard"
      host: "os-dashboard"
      port: "5601"

    opensearch:
      username: kibanaserver
      password: kibanaserver
      ssl:
        certificateAuthorities:
          - "/usr/share/opensearch-dashboards/certs/ca.crt"
        certificate: "/usr/share/opensearch-dashboards/certs/tls.crt"
        key: "/usr/share/opensearch-dashboards/certs/tls.key"
        verificationMode: none #"certificate"
        alwaysPresentCertificate: false
      requestHeadersAllowlist:
        - securitytenant
        - Authorization
        - security_tenant
    opensearch_security:
      session:
        keepalive: true
        ttl: 86400000
      cookie:
        secure: true
        ttl: 86400000
        password: "redacted"
      auth:
        type: ["basicauth","openid"]
        multiple_auth_enabled: true
      openid:
        extra_storage:
          additional_cookies: 3
          cookie_prefix: security_authentication_oidc
        base_redirect_url: "https://opensearch-ng.prod.domain.local"
        connect_url: "https://claims-keycloak.dev.domain.local/sso/realms/opensearch/.well-known/openid-configuration"
        client_id: "opensearch-dashboard"
        client_secret: "${OPENID_CLIENT_SECRET}"
        scope: "openid profile email"
        verify_hostnames: true
        root_ca: "/usr/share/opensearch-dashboards/certs/sbis-crts/CA.crt"
        trust_dynamic_headers: "true"
        header: Authorization
        refresh_tokens: true
      readonly_mode:
        roles:
          - kibana_read_only
      multitenancy:
        enabled: false
      ui:
        openid:
          login:
            buttonname: "Login with KeyCloak"
            showbrandimage: true
    home:
      disableWelcomeScreen: true

opensearchDashboardsYml:
  defaultMode:

priorityClassName: ""

opensearchAccount:
  secret: "os-admin-dashboard-secret"
  keyPassphrase:
    enabled: false

labels:
  sidecar.istio.io/inject: true

hostAliases: []

serverHost: "0.0.0.0"

service:
  type: ClusterIP
  port: 5601
  labels: {}
  annotations: {}
  loadBalancerSourceRanges: []
  httpPortName: http

ingress:
  enabled: false
  annotations: {}
  labels: {}
  hosts:
    - host: chart-example.local
      paths:
        - path: /
          backend:
            serviceName: ""
            servicePort: ""
  tls: []

resources:
  requests:
    cpu: "1000m"
    memory: "4096M"
  limits:
    cpu: "1000m"
    memory: "4096M"

autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 5
  targetCPU: "80"
  targetMemory: "80"

updateStrategy:
  type: "Recreate"

nodeSelector:
  node-role.kubernetes.io/observability: ""
tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/infra
  - effect: NoSchedule
    key: node-role.kubernetes.io/observability

affinity: {}

topologySpreadConstraints: []

extraObjects: []

lifecycle: {}

plugins:
  enabled: false
  installList: []

opensearch: securityconfig-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: os-securityconfig-secret
  namespace: observability
type: Opaque
stringData:
  action_groups.yml: |-
    _meta:
      type: "actiongroups"
      config_version: 2
    my-action-group:
      reserved: false
      hidden: false
      allowed_actions:
      - "indices:data/write/index*"
      - "indices:data/write/update*"
      - "indices:admin/mapping/put"
      - "indices:data/write/bulk*"
      - "read"
      - "write"
      static: false
  internal_users.yml: |-
    _meta:
      type: "internalusers"
      config_version: 2
    admin:
      hash: "redacted"
      reserved: true
      backend_roles:
      - "admin"
      description: "admin user"
    dashboarduser:
      hash: "redacted"
      reserved: true
      description: "OpenSearch Dashboards user"
      backend_roles:
      - "admin"
  nodes_dn.yml: |-
    _meta:
      type: "nodesdn"
      config_version: 2
  whitelist.yml: |-
    _meta:
      type: "whitelist"
      config_version: 2
  tenants.yml: |-
    _meta:
      type: "tenants"
      config_version: 2
  roles_mapping.yml: |-
    _meta:
      type: "rolesmapping"
      config_version: 2
    all_access:
      reserved: false
      backend_roles:
      - "admin"
      description: "Maps admin to all_access"
    admin_kc_role:
      reserved: false
      backend_roles:
      - "opensearch_admin"
      description: "Maps Keycloak admin role to OpenSearch admin role"
    dashboard_server:
      reserved: true
      users:
      - "dashboarduser"
  roles.yml: |-
    _meta:
      type: "roles"
      config_version: 2
    admin_kc_role:
      cluster_permissions:
        - 'cluster_all'
      index_permissions:
        - index_patterns:
            - '*'
          allowed_actions:
            - 'indices_all'
      static: false
      reserved: false
      hidden: false
      description: "Administrator role with full access"
  config.yml: |-
    _meta:
      type: "config"
      config_version: "2"
    config:
      dynamic:
        http:
          anonymous_auth_enabled: false
        authc:
          basic_internal_auth_domain:
            description: "Internal User Database"
            http_enabled: true
            transport_enabled: true
            order: "1"
            http_authenticator:
              type: basic
              challenge: true
            authentication_backend:
              type: internal
          openid_auth_domain:
            description: "Keycloak OpenID Connect"
            http_enabled: true
            transport_enabled: true
            order: 0
            http_authenticator:
              type: openid
              challenge: false
              config:
                openid_connect_idp:
                  enable_ssl: true
                  verify_hostnames: false
                  trust_all: true
                subject_key: preferred_username
                roles_key: roles
                openid_connect_url: "https://claims-keycloak.dev.domain.local/sso/realms/opensearch/.well-known/openid-configuration"
                client_id: "opensearch-dashboard"
                client_secret: "${OPENID_CLIENT_SECRET}"
                jwt_clock_skew_tolerance_seconds: 20
            authentication_backend:
              type: noop

access_token

{
  "exp": 1724440608,
  "iat": 1724440308,
  "jti": "2b4f7417-b327-4dfc-aae7-edf4f3e48fc6",
  "iss": "https://claims-keycloak.dev.domain.local/sso/realms/opensearch",
  "sub": "0c7da72f-0852-497b-bc2c-0d7a93567ad1",
  "typ": "Bearer",
  "azp": "opensearch-dashboard",
  "session_state": "35e426fa-0325-4532-8b3d-1790bb8f1161",
  "acr": "1",
  "allowed-origins": [
    "*"
  ],
  "scope": "openid email profile",
  "sid": "35e426fa-0325-4532-8b3d-1790bb8f1161",
  "email_verified": false,
  "roles": [
    "default-roles-opensearch",
    "offline_access",
    "uma_authorization",
    "opensearch_admin"
  ],
  "name": "Ageev Anton Ageev",
  "preferred_username": "aaageev",
  "given_name": "Anton Ageev",
  "family_name": "Anton",
  "email": "aaageev@domain.com"
}

refresh_token

{
  "exp": 1724442108,
  "iat": 1724440308,
  "jti": "4720ed26-04c8-44c5-8c4c-7da97d53973d",
  "iss": "https://claims-keycloak.dev.domain.local/sso/realms/opensearch",
  "aud": "https://claims-keycloak.dev.domain.local/sso/realms/opensearch",
  "sub": "0c7da72f-0852-497b-bc2c-0d7a93567ad1",
  "typ": "Refresh",
  "azp": "opensearch-dashboard",
  "session_state": "35e426fa-0325-4532-8b3d-1790bb8f1161",
  "scope": "openid email profile",
  "sid": "35e426fa-0325-4532-8b3d-1790bb8f1161"
}

Relevant Logs or Screenshots:

Screenshot 2024-08-24 at 00.31.37
Screenshot 2024-08-24 at 00.31.371766×1172 119 KB

2

Screenshot 2024-08-24 at 00.33.18
Screenshot 2024-08-24 at 00.33.181920×1415 73.8 KB

3

Screenshot 2024-08-24 at 00.43.09
Screenshot 2024-08-24 at 00.43.092664×642 31.6 KB

4

Just a quick clarification regarding the configuration URLs:

  • base_redirect_url: "https://opensearch-ng.prod.domain.local"
  • connect_url: "https://claims-keycloak.dev.domain.local/sso/realms/opensearch/.well-known/openid-configuration"

These URLs are intentionally set this way and are not errors. Keycloak is being used in our development environment (dev contour), and all necessary access permissions are in place for this setup.

5

Have you tried to extend the TTL in your configuration?

opensearch_security.cookie.ttl: 86400000
opensearch_security.session.ttl: 86400000
opensearch_security.session.keepalive: true

best,
mj

6

Yeah I think We’re using Keycloak in our development environment (dev contour)

7

Thank you to everyone who responded. The solution to the problem was related to the fact that corporate devices have a DLP agent installed, which seemingly adds something to the request. On PCs without the DLP agent, everything worked fine, even though the initial value for the reverse proxy was set to proxy_buffer_size 128k;

I also tried this configuration, but it turned out the issue was not related to timeouts:

opensearch_security.cookie.ttl: 86400000
opensearch_security.session.ttl: 86400000
opensearch_security.session.keepalive: true

By setting proxy_buffer_size 512k;, the dashboard started working on corporate devices with the DLP agent.

Here is the final configuration for the Nginx server:

location / {
    proxy_pass http://upstream/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Port 443;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_buffer_size 512k;
    proxy_buffers 16 1m;
    proxy_busy_buffers_size 1m;
    proxy_read_timeout 1200s;
    client_max_body_size 1G;
    client_body_buffer_size 40M;
}
2 Likes
8

excellent - thank you for coming back with the summary for everyone