Issue with Multi-Tenancy and Tenant-specific Dashboard saving in Opensearch Dashboards

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch Version: 2.11.1
OpenSearch Dashboards Version: 2.11.1
Authentication: Keycloak with OpenID Connect

Multi-tenancy is enabled.
The tenant, ddp role and roles mapping are created in the dashboard ui.
A custom tenant named ddp is created for grouping specific dashboards and visualizations.
Users authenticated via Keycloak are mapped to corresponding roles in OpenSearch, granting them access to the ddp tenant.

Describe the issue:

When users try to save a dashboard in the ddp tenant, the operation seems to default to the global tenant, despite the ddp tenant being selected.

Configuration:

securityconfig: 
data:
        action_groups.yml: |-
          _meta:
            type: "actiongroups"
            config_version: 2
        internal_users.yml: |-
          _meta:
            type: "internalusers"
            config_version: 2
          admin:
            hash: "{{ .ADMIN_PASSWORD_HASH }}"
            reserved: true
            backend_roles:
            - "admin"
            description: "admin user"
        nodes_dn.yml: |-
          _meta:
            type: "nodesdn"
            config_version: 2
        whitelist.yml: |-
          _meta:
            type: "whitelist"
            config_version: 2
        tenants.yml: |-
          _meta:
            type: "tenants"
            config_version: 2
        roles_mapping.yml: |-
          _meta:
            type: "rolesmapping"
            config_version: 2
          all_access:
            reserved: false
            backend_roles:
            - "admin"
            description: "Maps admin to all_access"
          own_index:
            reserved: false
            users:
            - "*"
            description: "Allow full access to an index named like the username"
          readall:
            reserved: false
            backend_roles:
            - "readall"
          manage_snapshots:
            reserved: false
            backend_roles:
            - "snapshotrestore"
          dashboard_server:
            reserved: true
            users:
            - "dashboarduser"
        roles.yml: |-
          _meta:
            type: "roles"
            config_version: 2
          dashboard_read_only:
            reserved: true
          security_rest_api_access:
            reserved: true
          # Allows users to view monitors, destinations and alerts
          alerting_read_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opendistro/alerting/alerts/get'
              - 'cluster:admin/opendistro/alerting/destination/get'
              - 'cluster:admin/opendistro/alerting/monitor/get'
              - 'cluster:admin/opendistro/alerting/monitor/search'
          # Allows users to view and acknowledge alerts
          alerting_ack_alerts:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opendistro/alerting/alerts/*'
          # Allows users to use all alerting functionality
          alerting_full_access:
            reserved: true
            cluster_permissions:
              - 'cluster_monitor'
              - 'cluster:admin/opendistro/alerting/*'
            index_permissions:
              - index_patterns:
                  - '*'
                allowed_actions:
                  - 'indices_monitor'
                  - 'indices:admin/aliases/get'
                  - 'indices:admin/mappings/get'
          # Allow users to read Anomaly Detection detectors and results
          anomaly_read_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opendistro/ad/detector/info'
              - 'cluster:admin/opendistro/ad/detector/search'
              - 'cluster:admin/opendistro/ad/detectors/get'
              - 'cluster:admin/opendistro/ad/result/search'
              - 'cluster:admin/opendistro/ad/tasks/search'
              - 'cluster:admin/opendistro/ad/detector/validate'
              - 'cluster:admin/opendistro/ad/result/topAnomalies'
          # Allows users to use all Anomaly Detection functionality
          anomaly_full_access:
            reserved: true
            cluster_permissions:
              - 'cluster_monitor'
              - 'cluster:admin/opendistro/ad/*'
            index_permissions:
              - index_patterns:
                  - '*'
                allowed_actions:
                  - 'indices_monitor'
                  - 'indices:admin/aliases/get'
                  - 'indices:admin/mappings/get'
          # Allows users to read Notebooks
          notebooks_read_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opendistro/notebooks/list'
              - 'cluster:admin/opendistro/notebooks/get'
          # Allows users to all Notebooks functionality
          notebooks_full_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opendistro/notebooks/create'
              - 'cluster:admin/opendistro/notebooks/update'
              - 'cluster:admin/opendistro/notebooks/delete'
              - 'cluster:admin/opendistro/notebooks/get'
              - 'cluster:admin/opendistro/notebooks/list'
          # Allows users to read observability objects
          observability_read_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opensearch/observability/get'
          # Allows users to all Observability functionality
          observability_full_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opensearch/observability/create'
              - 'cluster:admin/opensearch/observability/update'
              - 'cluster:admin/opensearch/observability/delete'
              - 'cluster:admin/opensearch/observability/get'
          # Allows users to read and download Reports
          reports_instances_read_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opendistro/reports/instance/list'
              - 'cluster:admin/opendistro/reports/instance/get'
              - 'cluster:admin/opendistro/reports/menu/download'
          # Allows users to read and download Reports and Report-definitions
          reports_read_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opendistro/reports/definition/get'
              - 'cluster:admin/opendistro/reports/definition/list'
              - 'cluster:admin/opendistro/reports/instance/list'
              - 'cluster:admin/opendistro/reports/instance/get'
              - 'cluster:admin/opendistro/reports/menu/download'
          # Allows users to all Reports functionality
          reports_full_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opendistro/reports/definition/create'
              - 'cluster:admin/opendistro/reports/definition/update'
              - 'cluster:admin/opendistro/reports/definition/on_demand'
              - 'cluster:admin/opendistro/reports/definition/delete'
              - 'cluster:admin/opendistro/reports/definition/get'
              - 'cluster:admin/opendistro/reports/definition/list'
              - 'cluster:admin/opendistro/reports/instance/list'
              - 'cluster:admin/opendistro/reports/instance/get'
              - 'cluster:admin/opendistro/reports/menu/download'
          # Allows users to use all asynchronous-search functionality
          asynchronous_search_full_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opendistro/asynchronous_search/*'
            index_permissions:
              - index_patterns:
                  - '*'
                allowed_actions:
                  - 'indices:data/read/search*'
          # Allows users to read stored asynchronous-search results
          asynchronous_search_read_access:
            reserved: true
            cluster_permissions:
              - 'cluster:admin/opendistro/asynchronous_search/get'
          # Allows user to use all index_management actions - ism policies, rollups, transforms
          index_management_full_access:
            reserved: true
            cluster_permissions:
              - "cluster:admin/opendistro/ism/*"
              - "cluster:admin/opendistro/rollup/*"
              - "cluster:admin/opendistro/transform/*"
            index_permissions:
              - index_patterns:
                  - '*'
                allowed_actions:
                  - 'indices:admin/opensearch/ism/*'
          # Allows users to use all cross cluster replication functionality at leader cluster
          cross_cluster_replication_leader_full_access:
            reserved: true
            index_permissions:
              - index_patterns:
                  - '*'
                allowed_actions:
                  - "indices:admin/plugins/replication/index/setup/validate"
                  - "indices:data/read/plugins/replication/changes"
                  - "indices:data/read/plugins/replication/file_chunk"
          # Allows users to use all cross cluster replication functionality at follower cluster
          cross_cluster_replication_follower_full_access:
            reserved: true
            cluster_permissions:
              - "cluster:admin/plugins/replication/autofollow/update"
            index_permissions:
              - index_patterns:
                  - '*'
                allowed_actions:
                  - "indices:admin/plugins/replication/index/setup/validate"
                  - "indices:data/write/plugins/replication/changes"
                  - "indices:admin/plugins/replication/index/start"
                  - "indices:admin/plugins/replication/index/pause"
                  - "indices:admin/plugins/replication/index/resume"
                  - "indices:admin/plugins/replication/index/stop"
                  - "indices:admin/plugins/replication/index/update"
                  - "indices:admin/plugins/replication/index/status_check"
        config.yml: |-
          _meta:
            type: "config"
            config_version: "2"
          config:
            dynamic:
              http:
                anonymous_auth_enabled: false
              do_not_fail_on_forbidden: true
              do_not_fail_on_forbidden_empty: true  
              authc:
                basic_internal_auth_domain:
                  http_enabled: true
                  transport_enabled: true
                  order: "4"
                  http_authenticator:
                    type: basic
                    challenge: true
                  authentication_backend:
                    type: intern
                openid_auth_domain:
                  http_enabled: true
                  transport_enabled: true
                  order: "0"
                  http_authenticator:
                    type: openid
                    challenge: false
                    config:
                      subject_key: preferred_username
                      roles_key: groups
                      openid_connect_url: "https://id.abc.com/realms/ddp/.well-known/openid-configuration"
                  authentication_backend:
                    type: noop

opensearch dashboard and nodepools:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch-cluster
  namespace: logging
spec:
  general:
    httpPort: 9200
    serviceName: opensearch-cluster
    version: 2.11.1
    drainDataNodes: true 
    setVMMaxMapCount: true 
  security:
    tls:  
      transport:  
        generate: true  
        perNode: true  
      http:
        generate: true
    config:  
      securityConfigSecret:
        name:  securityconfig-secret 
      adminCredentialsSecret:
        name: admin-credentials-secret  
  dashboards:
    env:
      - name: OPENID_CLIENT_ID
        valueFrom:
          secretKeyRef:
            name: openid-client-secret
            key: OPENID_CLIENT_ID
      - name: OPENID_CLIENT_SECRET
        valueFrom:
          secretKeyRef:
            name: openid-client-secret
            key: OPENID_CLIENT_SECRET
    labels:
      app: Opensearch
    additionalConfig:
      opensearch_security.auth.type: "openid"
      opensearch_security.auth.multiple_auth_enabled: "true"
      opensearch_security.openid.connect_url: "xxx"
      opensearch_security.openid.base_redirect_url: "xxx"
      opensearch_security.openid.client_id: "${OPENID_CLIENT_ID}"
      opensearch_security.openid.client_secret: "${OPENID_CLIENT_SECRET}"
      opensearch_security.multitenancy.enabled: "true"
      opensearch_security.multitenancy.tenants.enable_global: "true"
      opensearch_security.multitenancy.tenants.enable_private: "true"
      opensearch_security.multitenancy.enable_filter: "true"  
    version: 2.11.1
    enable: true
    replicas: 1
    opensearchCredentialsSecret:
      name: admin-credentials-secret 
    tls:
      enable: false
      generate: false
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "2Gi"
         cpu: "500m"

  nodePools:
    - component: nodes
      replicas: 3
      diskSize: "200Gi"
      nodeSelector:     
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "1000m"
      roles:
        - "cluster_manager"
        - "data"
      labels:
        app: Opensearch

Hi @conan_kudo,

Is this happening to all users or this behavior is specific to a single or a group of users?

Thanks,
mj

Hey @Mantas ,

thanks a ton for replying. This unfo applies to all users.

In the end, i want different users from different teams to be able to manage their indexes and dashboards, and dashboard objects shouldnt leak across teams.

It seems that the .kibana_hash_xxx indexes seem to not get created :confused:

With my admin user (user that is assigned to all_access) i can see that there is only the .kibana_1 index.

GET /_cat/indices/.kibana*

green open .kibana_1 8J2xSCkuQx682bTf7XrTaQ 1 1 1 0 10.3kb 5.1kb

I must be missing something. Any ideas?

Kind regards

Hi @conan_kudo,

I can’t see your tenants.yml (see sample below), would you mind sharing your tenants.yml?

sample:

        tenants.yml: |-
          _meta:
            type: "tenants"
            config_version: 2

Thanks,
mj

@conan_kudo Are you using a proxy? When proxying requests from Dashboards → OpenSearch backend make sure all headers are forwarded. The security plugin relies on the headers from security/src/main/java/org/opensearch/security/auth/BackendRegistry.java at main · opensearch-project/security · GitHub to determine the tenant a request is destined for.