Dashboards stops working as intended after a opensearch node stops in a cluster

goncalo.bpedras · January 3, 2024, 9:36am

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch - 2.11.1 (I upgraded to the last version because I thought it might be the version)

Docker
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Context: default

Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.25
GitCommit: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc:
Version: 1.1.10
GitCommit: v1.1.10-0-g18a0cb0
docker-init:
Version: 0.19.0
GitCommit: de40ad0

Hosting Operating System:
Ubuntu 22.04.3 LTS with EPYC cpu

Describe the issue:
(From the original post [weird behaviour] Dashboards doesn't work as intended after a opensearch node fails in a cluster · Issue #1712 · opensearch-project/security-dashboards-plugin · GitHub):
I have docker swarm with 5 nodes running Opensearch on each node. I have followed this guide: Docker - OpenSearch documentation. I have a healthy cluster, all nodes are working fine and I have a process indexing data at the rate of 15GB per day across multiple indices. All nodes are joined in the cluster because I checked it with _cat/nodes.

The problem begins when I shut down a node. At that point Dashboards stops working properly and I get kicked to the login page, and it fails to login every time.
The cluster entered in yellow state. I checked through the CLI and waited until it changed back to green. Then I tried to login again and it still doesn’t let me. But I can issue commands through the CLI with no problem and I confirmed that the documents are being inserted into Opensearch.

Configuration:
Every node has the following environment:

  OPENSEARCH_JAVA_OPTS: "-Xms4096m -Xmx4096m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
  node.name: opensearch-node05
  discovery.seed_hosts: opensearch-node01,opensearch-node02,opensearch-node03,opensearch-node04,opensearch-node05
  cluster.initial_master_nodes: opensearch-node01,opensearch-node02,opensearch-node03,opensearch-node04,opensearch-node05
  plugins.security.ssl.transport.pemcert_filepath: certs/opensearch-nodeXX/nodeXX.pem
  plugins.security.ssl.transport.pemkey_filepath: certs/opensearch-nodeXX/nodeXX-key.pem
  plugins.security.ssl.http.pemcert_filepath: certs/opensearch-nodeXX/nodeXX.pem
  plugins.security.ssl.http.pemkey_filepath: certs/opensearch-nodeXX/nodeXX-key.pem
  DISABLE_INSTALL_DEMO_CONFIG: "true"
  bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping

The XX are replaced by the corresponding node number.

Every opensearch node shares the same opensearch.yml:

network.bind_host: "0.0.0.0"
network.host: "0.0.0.0"
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/ca/root-ca.pem
plugins.security.ssl.http.pemtrustedcas_filepath: certs/ca/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
    - CN=AAAAAAAAA-ADMIN,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
plugins.security.nodes_dn:
    - CN=opensearch-node01,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
    - CN=opensearch-node02,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
    - CN=opensearch-node03,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
    - CN=opensearch-node04,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
    - CN=opensearch-node05,OU=AAAAAAAAA,O=AAAAAAAAA,L=AAAAAAAAA,ST=AAAAAAAAA,C=AAAAAAAAA
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE

Environment of Dashboards:

  OPENSEARCH_HOSTS: '["https://opensearch-node01:9200","https://opensearch-node02:9200","https://opensearch-node03:9200","https://opensearch-node04:9200","https://opensearch-node05:9200"]' # must be a string with no spaces when specified as an environment variable
  DISABLE_INSTALL_DEMO_CONFIG: "true"

opensearch_dashboards.yml:

opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: XXXXXXXXXXXXXXXXXXXXXXXXXX
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: false
opensearch_security.multitenancy.tenants.enable_private: false
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: false
server.host: '0.0.0.0'

I also made changes to the config.yml in the Opensearch-Security:

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    kibana:
      multitenancy_enabled: true
      private_tenant_enabled: true
      default_tenant: Global
      server_username: kibanaserver
      index: '.kibana'
    do_not_fail_on_forbidden: false
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      basic_auth_internal:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: internal

Relevant Logs or Screenshots:

Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z tags=error,plugins,securityDashboards Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z tags=error,plugins,securityDashboards pid=1
Error: Request Timeout after 30000ms


at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log @timestamp=2023-12-27T11:16:18Z Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log @timestamp=2023-12-27T11:16:18Z tags=error,http,server,OpenSearchDashboards Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log @timestamp=2023-12-27T11:16:18Z Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | Error: Request Timeout after 30000ms
at SecurityClient.dashboardsinfo (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:130:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BasicAuthentication.resolveTenant (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:249:28)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:172:24
at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:133:22)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9) | type=log @timestamp=2023-12-27T11:16:18Z tags=error,http,server,OpenSearchDashboards pid=1

Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z tags=error,plugins,securityDashboards Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log Failed to resolve user tenant: Error: Request Timeout after 30000ms | Failed to resolve user tenant: Error: Request Timeout after 30000ms | type=log @timestamp=2023-12-27T11:16:18Z tags=error,plugins,securityDashboards pid=1
Error: Request Timeout after 30000ms

ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= pid=1 ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= pid=1 error=[object Object] ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= pid=1 ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error ERR ERR ERR ERR Internal Server Error | ERR ERR ERR ERR Internal Server Error | type=error @timestamp=2023-12-27T11:28:02Z tags= pid=1 error=[object Object] url=http://XXXXXXXXX/app/home`

Eugene7 · January 3, 2024, 3:11pm

Hi @goncalo.bpedras

Do you have any tenants other than private and global?

I have noticed that you enabled private tenant and set global tenant as the default for OpenSearch. Also, you disabled private and global tenants for OpenSearch Dashboard. I would change it.

Please try to enable global and private tenants for Opensearch and Opensearch Dashboard.

goncalo.bpedras · January 3, 2024, 4:06pm

Hi @Eugene7 , thanks for the reply.

The problem persists, I’m kicked to the login page but the documents are still being inserted. I changed the config and restarted every node including the dashboard instance. Here’s what I’ve done:

The new opensearch_dashboards.yml:

opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: XXXXXXXXXXXXXXXXXXX
opensearch.requestHeadersWhitelist: [authorization, securitytenant]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: false
server.host: '0.0.0.0'

The new config.yml of opensearch-security:

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    kibana:
      multitenancy_enabled: true
      private_tenant_enabled: true
      default_tenant: Global
      server_username: kibanaserver
      index: '.kibana'
    do_not_fail_on_forbidden: false
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false

    authc:
      basic_auth_internal:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: internal

I also enabled the private tenant here:
imagem

To answer your question: Yes, I do have other tenants. Each tenant is for a specific group of users. Users cannot access other tenants. And I do not want users to have a private tenant.

peternied · January 4, 2024, 2:45am

The problem begins when I shut down a node.

Slightly different angle - This sounds like the connection from Dashboards to the OpenSearch backend is ‘stuck’ attempting to connect to the node that was shutdown. Maybe using a load balancer between these hosts would resolve the issue?

Looks like docker-compose has many features built around swarms that seem tailor made for your scenarios.

[1] Swarm mode overview | Docker Docs

goncalo.bpedras · January 4, 2024, 9:53am

Hi @peternied ,

I don’t think that makes sense because after I shutdown a opensearch node, the backend still works, because I can index and issue commands with no problems at all. It’s the dashboards that struggles for some reason. Also, when all opensearch nodes are up, it works fine.

Only one opensearch node is running on each docker swarm host, so it’s pretty balanced. The dashboards is the only one with a exposed port. I’m not talking about when a docker node goes down, but the opensearch node.

Eugene7 · January 8, 2024, 2:00pm

@pperesbr Could you please share your docker pull request?

Eugene7 · January 15, 2024, 9:54am

Hi @pperesbr

Plugins security options can’t be passed as variable and must be placed in the opensearch.yml file

Topic		Replies	Views
Docker Data node crash (?) after invalid number 18446744073709551615 OpenDistro troubleshoot	1	507	February 7, 2023
Issue viewing prometheus metrics OpenSearch troubleshoot	1	71	April 11, 2024
Opensearch dashboard custom branding OpenDistro configure	5	685	March 1, 2022
Opensearch restarts on Openshift On-Premises OpenSearch troubleshoot	2	529	June 2, 2022
I Geeting some vulnerability issue in opensearchproject/opensearch-Dashboard:2.8.0 docker Image OpenSearch releases , cve , security-issue	2	269	June 22, 2023

Dashboards stops working as intended after a opensearch node stops in a cluster

Related Topics