Failed to resolve if it's a readonly tenant: Error: Not Found error

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.18.0

Describe the issue:
I set up multi-tenancy on opensearch dashboards and created a read-only tenant.
Since then, I have been continuously encountering the following error, and I don’t know why.
Please help.

Failed to resolve if it’s a readonly tenant: Error: Not Found

Configuration:
opensearch dashboards multi-tenancy : true

Relevant Logs or Screenshots:
Dec 13 12: 17: 05 opensearch-dashboards[
2272623
]: {
“type”: “log”,
“@timestamp”: “2024-12-13T03:17:05Z”,
“tags”: [
“error”,
“plugins”,
“securityDashboards”
],
“pid”: 2272623,
“message”: “Failed to resolve if it’s a readonly tenant: Error: Not Found\n
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards-2.18.0/plugins/securityDashboards/server/backend/opensearch_security_client.ts: 130: 13)\n
at processTicksAndRejections (node:internal/process/task_queues: 95: 5)\n
at ReadonlyService.isReadonly (/usr/share/opensearch-dashboards-2.18.0/plugins/securityDashboards/server/readonly/readonly_service.ts: 101: 30)\n
at ReadonlyService.hideForReadonly (/usr/share/opensearch-dashboards-2.18.0/src/core/server/security/readonly_service.js: 18: 13)\n
at /usr/share/opensearch-dashboards-2.18.0/src/plugins/data/server/index_patterns/index_patterns_service.js: 49: 14\n
at /usr/share/opensearch-dashboards-2.18.0/src/core/server/capabilities/resolve_capabilities.js: 52: 21\n
at /usr/share/opensearch-dashboards-2.18.0/src/core/server/capabilities/resolve_capabilities.js: 51: 26\n
at /usr/share/opensearch-dashboards-2.18.0/src/core/server/capabilities/resolve_capabilities.js: 51: 26\n
at /usr/share/opensearch-dashboards-2.18.0/src/core/server/capabilities/resolve_capabilities.js: 51: 26\n
at /usr/share/opensearch-dashboards-2.18.0/src/core/server/capabilities/resolve_capabilities.js: 51: 26\n
at /usr/share/opensearch-dashboards-2.18.0/src/core/server/capabilities/resolve_capabilities.js: 51: 26\n
at /usr/share/opensearch-dashboards-2.18.0/src/core/server/capabilities/resolve_capabilities.js: 51: 26\n
at /usr/share/opensearch-dashboards-2.18.0/src/core/server/capabilities/resolve_capabilities.js: 51: 26\n
at /usr/share/opensearch-dashboards-2.18.0/src/core/server/capabilities/resolve_capabilities.js: 51: 26\n
at /usr/share/opensearch-dashboards-2.18.0/src/core/server/capabilities/routes/resolve_capabilities.js: 53: 26\n
at Router.handle (/usr/share/opensearch-dashboards-2.18.0/src/core/server/http/router/router.js: 174: 44)\n
at handler (/usr/share/opensearch-dashboards-2.18.0/src/core/server/http/router/router.js: 140: 50)\n
at exports.Manager.execute (/usr/share/opensearch-dashboards-2.18.0/node_modules/@hapi/hapi/lib/toolkit.js: 60: 28)\n
at Object.internals.handler (/usr/share/opensearch-dashboards-2.18.0/node_modules/@hapi/hapi/lib/handler.js: 46: 20)\n
at exports.execute (/usr/share/opensearch-dashboards-2.18.0/node_modules/@hapi/hapi/lib/handler.js: 31: 20)\n
at Request._lifecycle (/usr/share/opensearch-dashboards-2.18.0/node_modules/@hapi/hapi/lib/request.js: 371: 32)\n
at Request._execute (/usr/share/opensearch-dashboards-2.18.0/node_modules/@hapi/hapi/lib/request.js: 281: 9)”
}

2

@pizzaman Is the read-only tenant the only one assigned to that user?
Could you share roles config of that user?

3

@pizzaman If this thread is duplicate of Failed to resolve if it's a readonly tenant: Error: Not Found, please close the older one and continue conversation here.

4
  1. role.yml
---
_meta:
  type: "roles"
  config_version: 2


indexes_full_access:
  reserved: false
  index_permissions:
    - index_patterns:
        - "*"
      allowed_actions:
        - "*"
  tenant_permissions:
  - tenant_patterns:
    - "*"
    allowed_actions:
    - "kibana_all_write"
# ----------------------------------------------------
indexes_security_search_full_access:
  reserved: true
  index_permissions:
    - index_patterns:
        - "kube-apiserver-audit-*"
        - "syslog-*"
      allowed_actions:
        - "indices:data/read/search*"
        - "read"
        - "view_index_metadata"
  tenant_permissions:
  - tenant_patterns:
    - "SECURITY"
    allowed_actions:
    - "kibana_all_write"
# ----------------------------------------------------
indexes_web_search_full_access:
  reserved: true
  index_permissions:
    - index_patterns:
        - "ingress-nginx-*"
        - "mywebapp-*"
      allowed_actions:
        - "indices:data/read/search*"
        - "read"
        - "view_index_metadata"
  tenant_permissions:
  - tenant_patterns:
    - "WEB"
    allowed_actions:
    - "kibana_all_write"
# ----------------------------------------------------
# Restrict users so they can only view visualization and dashboard on OpenSearchDashboards
kibana_read_only:
  reserved: true

# ======================== Custom Roles =========================
test_write_access:
  reserved: true
  cluster_permissions:
    - 'indices:data/write/bulk'
    - 'cluster:monitor/main'
    - 'cluster:monitor/health'
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - 'indices:admin/create'
        - 'indices:admin/mapping/put'
        - 'indices:data/write/bulk'
        - 'indices:data/write/bulk*'
        - 'indices:data/write/index'
        - 'indices:data/write/update'
        - 'indices:data/write/delete'
  1. roles_mapping.yml
---

# In this file users, backendroles and hosts can be mapped to Security roles.
# Permissions for OpenSearch roles are configured in roles.yml

_meta:
  type: "rolesmapping"
  config_version: 2

kibana_server:
  reserved: true
  users:
  - "kibanaserver"

# Define your roles mapping here
all_access:
  reserved: false
  backend_roles:
  - "admin"
  - "opensearch_admin"
  - "OPENSEARCH_ADMIN"
  description: "Maps admin to all_access"
# ----------------------------------------------------
indexes_full_access:
  reserved: false
  backend_roles:
  - "opensearch_admin"
  description: "Maps admin to indexes_full_access"
# ----------------------------------------------------
own_index:
  reserved: false
  users:
  - "*"
  description: "Allow full access to an index named like the username"
# ----------------------------------------------------
readall:
  reserved: false
  backend_roles:
  - "opensearch_index_read_all"
  - "OPENSEARCH_USER"
# ----------------------------------------------------
indexes_security_search_full_access:
  reserved: true
  backend_roles:
  - "opensearch_index_read_all"
  - "opensearch_index_read_security"
  description: "Maps users to indexes_security_search_full_access"
# ----------------------------------------------------
indexes_web_search_full_access:
  reserved: true
  backend_roles:
  - "opensearch_index_read_all"
  - "opensearch_index_read_web"
  description: "Maps users to indexes_web_search_full_access"

# ======================== Custom Role Mapping =========================
kibana_user:
  reserved: true
  backend_roles:
  - "kibanauser"
  - "OPENSEARCH_USER"
  description: "General"

kibana_read_only:
  reserved: true
  backend_roles:  
  - "OPENSEARCH_VIEWER"
  description: "Read_Only"

test_write_access:
  reserved: true
  users:
  - "test"
  description: "Public Client User"
  1. internal_users.yml
---

# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

_meta:
  type: "internalusers"
  config_version: 2

# Define your internal users here

## Demo users
kibanaserver:
  hash: {{opensearch_kibanaserver_hash}}
  reserved: true
  description: "Demo OpenSearch Dashboards user"

# ======================== Custom User =========================
admin:
  hash: {{opensearch_admin_hash}}
  reserved: true
  backend_roles:
  - "OPENSEARCH_ADMIN"
  description: "Super User"

kibanauser:
  hash: {{opensearch_kibanauser_hash}}
  reserved: true
  backend_roles:
  - "OPENSEARCH_USER"
  description: "General"

kibanareadonly:
  hash: {{opensearch_kibanareadonly_hash}}
  reserved: true
  backend_roles:  
  - "OPENSEARCH_VIEWER"
  description: "Read_Only"

tester:
  hash: {{opensearch_tester_hash}}
  reserved: true
  backend_roles:
  - "OPENSEARCH_TESTER"
  description: "Tester"

test:
  hash: {{opensearch_test_hash}}
  reserved: true
  backend_roles:
  - "OPENSEARCH_ADMIN"
  description: "Public Client User"
  1. tenants.yml
---
_meta:
  type: "tenants"
  config_version: 2

# Define your tenants here
SECURITY:
  reserved: false
  description: "Tenant for security logs (e.g. kubernetes audit or opensearch audit)"
WEB:
  reserved: false
  description: "Tenant for web-app logs"
5

Yes, this is a duplicate of the previous issue and I will close it and continue the conversation here. It’s a very difficult problem, and I would appreciate your help.

6

Additionally, the error did not occur with the security settings used in 2.11 before upgrading to 2.18.

7

In version 2.15, the error does not occur, but it seems that the error occurs from version 2.16.

8

@pizzaman I don’t see any read-only tenant. All tenant permissions are kibana_all_write.

Could you run the following command with the user that has the issue and share the output?

curl --insecure -u <username>:<passowrd> -XGET https://<OpenSearch_IP_or_FQDN>:9200/_plugs/_security/authinfo?pretty
9

@pablo

The error occurs in version 2.16 or later for any user,
and the error does not occur in version 2.15 or earlier.

{
    "user": "User [name=admin, backend_roles=[OPENSEARCH_ADMIN], requestedTenant=null]",
    "user_name": "admin",
    "user_requested_tenant": null,
    "remote_address": "{{private}}",
    "backend_roles": [
        "OPENSEARCH_ADMIN"
    ],
    "custom_attribute_names": [],
    "roles": [
        "all_access"
    ],
    "tenants": {
        "SECURITY": true,
        "admin": true,
        "raycluster": true,
        "system": true,
        "global_tenant": true,
        "WEB": true
    },
    "principal": null,
    "peer_certificates": "0",
    "sso_logout_url": null
}
10

try run a command with the user

11

@pizzaman I understand the reported user belongs to an external IDP. You’ve mapped the backend role OPENSEARCH_ADMIN to the all_access role.

Do you have an issue with this user? The all_access role will have read/write access to all tenants.
The read/write permission is set for the role, not the tenant.