401 Unauthorized with Keycloak OpenID

Hello community,

This is another topic where you end up having 401 Unauthorized issue with Keycloak:

{
  "statusCode": 401,
  "error": "Unauthorized",
  "message": "Unauthorized"
}

Environment: EC2(opensearch cluster) + EKS(opendashboards)

OpenSearch: 2.19.3

OpenDashboards: 2.19.3


OS nodes work in containers, here are the envs of master-nodes:

cluster.name: "opensearch"
node.name: "{{ ansible_hostname }}"

network.bind_host: "_local_,_site_"

node.roles: "master-eligible,cluster_manager,clustermanager-eligible"

discovery.seed_hosts: "{{ master_ips }}"
cluster.initial_cluster_manager_nodes: "{{ master_ips }}"

plugins.security.ssl.transport.pemcert_filepath: "/usr/share/opensearch/config/certs/{{ ansible_fqdn }}.crt"
plugins.security.ssl.transport.pemkey_filepath: "/usr/share/opensearch/config/certs/{{ ansible_fqdn }}.key"
plugins.security.ssl.transport.pemtrustedcas_filepath: "/usr/share/opensearch/config/certs/{{ ansible_fqdn }}.issuer.crt"

plugins.security.ssl.transport.enforce_hostname_verification: "false"
plugins.security.ssl.http.enabled: "true"

plugins.security.ssl.http.pemcert_filepath: "/usr/share/opensearch/config/certs/{{ ansible_fqdn }}.crt"
plugins.security.ssl.http.pemkey_filepath: "/usr/share/opensearch/config/certs/{{ ansible_fqdn }}.key"
plugins.security.ssl.http.pemtrustedcas_filepath: "/usr/share/opensearch/config/certs/{{ ansible_fqdn }}.issuer.crt"

plugins.security.ssl_cert_reload_enabled: "true"

plugins.security.nodes_dn: "CN=*.{{ domain }}"
plugins.security.authcz.admin_dn: "CN=opensearch-admin"

plugins.security.ssl.http.enabled_protocols: "TLSv1.2,TLSv1.3"
plugins.security.allow_default_init_securityindex: "true"
plugins.security.audit.type: "internal_opensearch"
plugins.security.enable_snapshot_restore_privilege: "true"
plugins.security.check_snapshot_restore_write_privileges: "true"
plugins.security.restapi.roles_enabled: "all_access"
plugins.security.system_indices.permission.enabled: "true"
plugins.security.system_indices.enabled: "true"
plugins.security.system_indices.indices: ".opendistro-alerting-config, .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*, .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*, .opendistro-notifications-*, .opendistro-notebooks, .opendistro-asynchronous-search-response*"

logger.org.opensearch.discovery: "INFO"

bootstrap.memory_lock: "true"

cluster.routing.allocation.disk.watermark.low: "20gb"
cluster.routing.allocation.disk.watermark.high: "15gb"
cluster.routing.allocation.disk.watermark.flood_stage: "7gb"
cluster.max_shards_per_node: "5000"

DISABLE_INSTALL_DEMO_CONFIG: "true"

config.yml:

_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    http:
      anonymous_auth_enabled: true
    kibana:
      multitenancy_enabled: false
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: internal
      openid_auth_domain:
        description: "OpenID Connect"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            openid_connect_idp:
              enable_ssl: true
              verify_hostnames: false
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: "https://auth.corp/auth/realms/employee/.well-known/openid-configuration"
        authentication_backend:
          type: noop

roles_mapping.yml:

---
_meta:
  type: "rolesmapping"
  config_version: 2

all_access:
  reserved: false
  users:
  - "admin"

internal_users.yml:

---
_meta:
  type: "internalusers"
  config_version: 2

admin:
  hash: "{{ admin_password_hash }}"
  reserved: true
  opendistro_security_roles:
  - "all_access"

I don’t do much with ‘internal’ files, all i want is to set admin password and map admin to all_access role. All other things are done with API:

roles:

{
  "oidc-admin": {
    "reserved": false,
    "hidden": false,
    "description": "Added by Terraform",
    "cluster_permissions": [
      "*"
    ],
    "index_permissions": [
      {
        "index_patterns": [
          "*"
        ],
        "dls": "",
        "fls": [],
        "masked_fields": [],
        "allowed_actions": [
          "*"
        ]
      }
    ],
    "tenant_permissions": [],
    "static": false
  },
  "oidc-reader": {
    "reserved": false,
    "hidden": false,
    "description": "Added by Terraform",
    "cluster_permissions": [
      "cluster:admin/opendistro/reports/menu/download",
      "cluster:admin/opensearch/observability/get",
      "cluster:admin/opendistro/reports/definition/list",
      "indices:data/read/mget*",
      "cluster:admin/opensearch/ql/datasources/read",
      "indices:data/write/bulk"
    ],
    "index_permissions": [
      {
        "index_patterns": [
          ".kibana_*"
        ],
        "dls": "",
        "fls": [],
        "masked_fields": [],
        "allowed_actions": [
          "write"
        ]
      },
      {
        "index_patterns": [
          "*"
        ],
        "dls": "",
        "fls": [],
        "masked_fields": [],
        "allowed_actions": [
          "read",
          "search",
          "get"
        ]
      }
    ],
    "tenant_permissions": [],
    "static": false
  }
}

mappings:

{
  "oidc-reader": {
    "hosts": [],
    "users": [],
    "reserved": false,
    "hidden": false,
    "backend_roles": [
      "reader"
    ],
    "and_backend_roles": []
  },
  "oidc-admin": {
    "hosts": [],
    "users": [],
    "reserved": false,
    "hidden": false,
    "backend_roles": [
      "admin"
    ],
    "and_backend_roles": [],
    "description": "Added by Terraform"
  }
}

opendahboards-config.yml:

home:
  disableWelcomeScreen: true
opensearch:
  hosts:
    - https://master-1.corp:9200
    - https://master-2.corp:9200
    - https://master-3.corp:9200
  password: ...
  requestHeadersAllowlist:
    - Authorization
  ssl:
    certificateAuthorities: /usr/share/dashboards/certs/ca.crt
    verificationMode: full
  username: opendashboards
opensearch_security:
  auth:
    multiple_auth_enabled: "true"
    type:
      - basicauth
      - openid
  cookie:
    ttl: 86400000
  multitenancy:
    enabled: false
  openid:
    base_redirect_url: https://opendashboards.corp.com
    client_id: opendashboards
    client_secret: ...
    connect_url: https://auth.corp/auth/realms/employee/.well-known/openid-configuration
    refresh_tokens: true
    scope: openid profile email
    verify_hostnames: false
  session:
    keepalive: true
    ttl: 86400000
  ui:
    openid:
      login:
        buttonname: Log in with Google (Keycloak)
server:
  host: 0.0.0.0
  name: OpenDashboards
  port: "5601"

In keycloak i have a client with 2 roles: admin and reader, i bind them to groups with users. Also there is a mapper of User Client Role type with roles Token Claim Name. My user gets the role correctly:

"scope": "openid profile email",
"email_verified": true,
"roles": [
  "admin"
],
"name": "Maxim",
"preferred_username": "maxim@corp.com",

So i expect to log in with user from preferred_username key and oidc-admin role, but i end up with {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}.

What am i doing wrong?

@maxemontio Thank you for providing the full details. Can you change the challenge flags as follows and try again:

basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      openid_auth_domain:
        description: "OpenID Connect"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: true

Are there any errors in the browser dev tools?

Thank you, that worked!

There are errors in browser dev tools - i get them every time when i open login page without trying to login, opendashboards logs show the same lines with 401, but everything works anyway (chrome 139.0.7258.128 on macos):

/app/login?:654 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-75XtnrpDA0UHDMcl7S8lvswryIOd0RqgacRh0AMOgdk='), or a nonce ('nonce-...') is required to enable inline execution.

bootstrap.js:48 ^ A single error about an inline script not firing due to content security policy is expected!
data.plugin.js:1 Application Not available.
core.entry.js:15 
 POST https://opendashboards.corp.com/api/ism/accountInfo 401 (Unauthorized)
core.entry.js:15 Detected an unhandled Promise rejection.
Error
core.entry.js:15 Uncaught (in promise) HttpFetchError
    at fetch_Fetch.fetchResponse (core.entry.js:15:411759)
    at async interceptResponse (core.entry.js:15:406632)
    at async core.entry.js:15:409466
core.entry.js:15 
 GET https://opendashboards.corp.com/api/v1/restapiinfo 401 (Unauthorized)
securityDashboards.plugin.js:15 HttpFetchError
    at fetch_Fetch.fetchResponse (core.entry.js:15:411759)
    at async interceptResponse (core.entry.js:15:406632)
    at async core.entry.js:15:409466
core.entry.js:15 
 GET https://opendashboards.corp.com/api/v1/configuration/account?dataSourceId= 401 (Unauthorized)
core.entry.js:15 
 GET https://opendashboards.corp.com/api/v1/auth/dashboardsinfo?dataSourceId= 401 (Unauthorized)
core.entry.js:15 
 GET https://opendashboards.corp.com/api/v1/auth/type?dataSourceId= 401 (Unauthorized)
core.entry.js:15 
 GET https://opendashboards.corp.com/api/v1/configuration/account 401 (Unauthorized)
core.entry.js:15 
 GET https://opendashboards.corp.com/api/v1/configuration/account?dataSourceId= 401 (Unauthorized)

I am confused, because documentation says to set challenge to false for openid_auth_domain. Could you possibly give some details about that?

Also i had to specify openid_auth_domain.http_authenticator.config.openid_connect_idp.pemtrustedcas_filepath option with /etc/pki/tls/cert.pem value - it came unexpected because our keycloak is publicly accessible with letsencrypt certificates, so i thought that option was unnecessary.

By the way, i expected that my role oidc-admin would be exactly the same as all_access, but it isn’t, the only difference i noticed for now is that you can’t see security tab in management section. So i had to map admin backend role with all_access role to have that tab accessible.

@maxemontio Glad this worked for you. The challenge flag is like saying, where do you want to stop if not authenticated after this step. Therefore setting it true after basic_auth would prevent the security plugin from going on to the next step.

Regarding the oidc-admin, You just need to add this to your plugins.security.restapi.roles_enabled: in opensearch.yml, where currently you probably have all_access listed.

That worked as well

Thank you, @Anthony!