Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch - 2.11.1 (I upgraded to the last version because I thought it might be the version)
Docker
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.25
GitCommit: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc:
Version: 1.1.10
GitCommit: v1.1.10-0-g18a0cb0
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Hosting Operating System:
Ubuntu 22.04.3 LTS with EPYC cpu
Describe the issue:
(From the original post [weird behaviour] Dashboards doesn't work as intended after a opensearch node fails in a cluster · Issue #1712 · opensearch-project/security-dashboards-plugin · GitHub):
I have docker swarm with 5 nodes running Opensearch on each node. I have followed this guide: Docker - OpenSearch documentation. I have a healthy cluster, all nodes are working fine and I have a process indexing data at the rate of 15GB per day across multiple indices. All nodes are joined in the cluster because I checked it with _cat/nodes.
The problem begins when I shut down a node. At that point Dashboards stops working properly and I get kicked to the login page, and it fails to login every time.
The cluster entered in yellow state. I checked through the CLI and waited until it changed back to green. Then I tried to login again and it still doesn’t let me. But I can issue commands through the CLI with no problem and I confirmed that the documents are being inserted into Opensearch.
Configuration:
Every node has the following environment:
OPENSEARCH_JAVA_OPTS: "-Xms4096m -Xmx4096m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
node.name: opensearch-node05
discovery.seed_hosts: opensearch-node01,opensearch-node02,opensearch-node03,opensearch-node04,opensearch-node05
cluster.initial_master_nodes: opensearch-node01,opensearch-node02,opensearch-node03,opensearch-node04,opensearch-node05
plugins.security.ssl.transport.pemcert_filepath: certs/opensearch-nodeXX/nodeXX.pem
plugins.security.ssl.transport.pemkey_filepath: certs/opensearch-nodeXX/nodeXX-key.pem
plugins.security.ssl.http.pemcert_filepath: certs/opensearch-nodeXX/nodeXX.pem
plugins.security.ssl.http.pemkey_filepath: certs/opensearch-nodeXX/nodeXX-key.pem
DISABLE_INSTALL_DEMO_CONFIG: "true"
bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping
The XX are replaced by the corresponding node number.
Every opensearch node shares the same opensearch.yml:
network.bind_host: "0.0.0.0"
network.host: "0.0.0.0"
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/ca/root-ca.pem
plugins.security.ssl.http.pemtrustedcas_filepath: certs/ca/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
- CN=AAAAAAAAA-ADMIN,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
plugins.security.nodes_dn:
- CN=opensearch-node01,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
- CN=opensearch-node02,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
- CN=opensearch-node03,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
- CN=opensearch-node04,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
- CN=opensearch-node05,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE
Environment of Dashboards:
OPENSEARCH_HOSTS: '["https://opensearch-node01:9200","https://opensearch-node02:9200","https://opensearch-node03:9200","https://opensearch-node04:9200","https://opensearch-node05:9200"]' # must be a string with no spaces when specified as an environment variable
DISABLE_INSTALL_DEMO_CONFIG: "true"
opensearch_dashboards.yml:
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: XXXXXXXXXXXXXXXXXXXXXXXXXX
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: false
opensearch_security.multitenancy.tenants.enable_private: false
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: false
server.host: '0.0.0.0'
I also made changes to the config.yml in the Opensearch-Security:
_meta:
type: "config"
config_version: 2
config:
dynamic:
kibana:
multitenancy_enabled: true
private_tenant_enabled: true
default_tenant: Global
server_username: kibanaserver
index: '.kibana'
do_not_fail_on_forbidden: false
http:
anonymous_auth_enabled: false
xff:
enabled: false
authc:
basic_auth_internal:
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: internal
Relevant Logs or Screenshots:
Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z tags=error,plugins,securityDashboards Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z tags=error,plugins,securityDashboards pid=1
Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log @timestamp=2023-12-27T11:16:18Z Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log @timestamp=2023-12-27T11:16:18Z tags=error,http,server,OpenSearchDashboards Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log @timestamp=2023-12-27T11:16:18Z Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log @timestamp=2023-12-27T11:16:18Z tags=error,http,server,OpenSearchDashboards pid=1
Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z tags=error,plugins,securityDashboards Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z tags=error,plugins,securityDashboards pid=1
Error: Request Timeout after 30000ms
ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= pid=1 ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= pid=1 error=[object Object] ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= pid=1 ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= pid=1 error=[object Object] url=http://XXXXXXXXX/app/home`