Slow loading, no info in Discover

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch (and Dashboards) - 2.16.0
Mozilla Firefox 129.0.1
Google Chrome 127.0.6533.99

Describe the issue:
After upgrading Opensearch to 2.16.0, Dashboards won’t load and keep timing out.

Configuration:
opensearch.yml

cluster.name: "private-email-logstore"

node.name: "LOGSTORE-MASTER-01"

path.data: /var/lib/opensearch

path.logs: /var/log/opensearch

bootstrap.memory_lock: true

network.host: "10.50.207.11"

http.port: 9200

discovery.seed_hosts: ["LOGSTORE-MASTER-01","LOGSTORE-MASTER-02","LOGSTORE-MASTER-03","LOGSTORE-HOT-01","LOGSTORE-HOT-02","LOGSTORE-HOT-03","LOGSTORE-HOT-04","LOGSTORE-WARM-01","LOGSTORE-WARM-02","LOGSTORE-WARM-03","LOGSTORE-WARM-04","LOGSTORE-WARM-05","LOGSTORE-WARM-06","LOGSTORE-WARM-07","LOGSTORE-WARM-08","LOGSTORE-WARM-09","LOGSTORE-WARM-10","LOGSTORE-COLD-01","LOGSTORE-COLD-02","LOGSTORE-COLD-03","LOGSTORE-COLD-04","LOGSTORE-COLD-05","LOGSTORE-COLD-06","LOGSTORE-COLD-07","LOGSTORE-COLD-08","LOGSTORE-COLD-09","LOGSTORE-COLD-10"]

cluster.initial_cluster_manager_nodes: ["LOGSTORE-MASTER-01","LOGSTORE-MASTER-02","LOGSTORE-MASTER-03"]

plugins.security.ssl.transport.pemcert_filepath: LOGSTORE-MASTER-01-cert.pem
plugins.security.ssl.transport.pemkey_filepath: LOGSTORE-MASTER-01-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca-cert.pem
plugins.security.ssl.transport.enforce_hostname_verification: true

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: LOGSTORE-MASTER-01-cert.pem
plugins.security.ssl.http.pemkey_filepath: LOGSTORE-MASTER-01-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca-cert.pem

plugins.security.ssl.transport.truststore_filepath: truststore.jks
plugins.security.ssl.transport.truststore_password: ***

plugins.security.authcz.admin_dn:
  - 'EMAILADDRESS=pe@namecheap.com,CN=opensearch-admin,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'

plugins.security.nodes_dn:
  - 'CN=LOGSTORE-MASTER-01,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-MASTER-02,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-MASTER-03,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-HOT-01,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-HOT-02,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-HOT-03,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-HOT-04,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-WARM-01,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-WARM-02,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-WARM-03,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-WARM-04,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-WARM-05,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-WARM-06,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-WARM-07,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-WARM-08,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-WARM-09,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-WARM-10,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-COLD-01,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-COLD-02,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-COLD-03,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-COLD-04,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-COLD-05,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-COLD-06,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-COLD-07,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-COLD-08,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-COLD-09,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'
  - 'CN=LOGSTORE-COLD-10,OU=Private Email,O=Namecheap,L=Phoenix,ST=Arizona,C=US'

plugins.security.allow_default_init_securityindex: true

# Put 0 to disable auth information caching
plugins.security.cache.ttl_minutes: 60

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

plugins.security.audit.type: internal_opensearch

plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true

plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

node.max_local_storage_nodes: 3
node.roles: ["master"]

security config

---

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      basic_internal:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 3
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: internal
      ldap_ncad:
        description: "Authenticate via Namecheap Active Directory"
        http_enabled: true
        transport_enabled: true
        order: 2
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: ldap
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: false
            hosts:
            - corp.namecheap.net
            bind_dn: ***
            password: ***
            userbase: "DC=corp,DC=namecheap,DC=net"
            usersearch: "(|(mail={0})(userPrincipalName={0}@corp.namecheap.net)(userPrincipalName={0}))"
            username_attribute: "userPrincipalName"
      ldap_inner:
        description: "Authenticate via inner.directory"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: ldap
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: false
            hosts:
            - haproxy-vip.inner.directory
            bind_dn: ***
            password: ***
            userbase: "DC=inner,DC=directory"
            usersearch: "(|(mail={0})(userPrincipalName={0}@inner.directory)(userPrincipalName={0}))"
            username_attribute: "userPrincipalName"
    authz:
      ldap_ncad_roles:
        description: "Authorize via Namecheap Active Directory"
        http_enabled: true
        transport_enabled: true
        authorization_backend:
          type: ldap
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: false
            hosts:
            - corp.namecheap.net
            bind_dn: ***
            password: ***
            userbase: "DC=corp,DC=namecheap,DC=net"
            usersearch: "(userPrincipalName={0})"
            rolebase: "OU=Teams,OU=Staff Groups,DC=corp,DC=namecheap,DC=net"
            rolesearch: '(member={0})'
            rolename: cn
            resolve_nested_roles: true
            #nested_role_filter:
            #  - "DN of a group to filter OUT (can be regexp)"
            #max_nested_depth: 1
            skip_users:
              - admin
              - kibanaserver
              - logstash
              - zabbix
              - grafana
              - '/CN=.*,DC=inner,DC=directory/'
      ldap_inner_roles:
        description: "Authorize via inner.directory"
        http_enabled: true
        transport_enabled: true
        authorization_backend:
          type: ldap
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: false
            hosts:
            - haproxy-vip.inner.directory
            bind_dn: ***
            password: ***
            userbase: "DC=inner,DC=directory"
            usersearch: "(userPrincipalName={0})"
            rolebase: "OU=UA Administrative,DC=inner,DC=directory"
            rolesearch: '(member={0})'
            rolename: cn
            resolve_nested_roles: true
            #nested_role_filter:
            #  - "DN of a group to filter OUT (can be regexp)"
            max_nested_depth: 1
            skip_users:
              - admin
              - kibanaserver
              - logstash
              - zabbix
              - grafana
              - '/CN=.*,DC=corp,DC=namecheap,DC=net/'

Relevant Logs or Screenshots:

image
image3452×1956 137 KB

I don’t see any relevant logs.

2

There is something wrong with Private tenant. We’ve disabled it and Discover started loading fine in Global tenant.

3

Hi @stas.fastov,

Are there any entries in the OS logs when the privet Tenant is enabled?

Could you share the permission info of your user (output of the below)?

GET /_plugins/_security/api/permissionsinfo

thanks,
mj

4

Hi,

Nothing was seen in logs related to the issue.

Here are the permissions of a test user mapped to the ‘all_access’ role.

{"user":"User [name=me, backend_roles=[], requestedTenant=null]","user_name":"me","has_api_access":true,"disabled_endpoints":{}}
5

Would you mind running the below and sharing the output (just want to inspect how roles are mapped):

curl --insecure -u me:<password> -XGET https://localhost:9200/_plugins/_security/authinfo?pretty

Best,
mj

6

Hello,
Sorry I was away for some time.

Here is the output (some info was removed, like hostnames, IPs, LDAP groups):

{
  "user" : "User [name=stasfastov@<redacted>.net, backend_roles=[... G_<redacted>-Opensearch-Admin, ...], requestedTenant=null]",
  "user_name" : "stasfastov@<redacted>.net",
  "user_requested_tenant" : null,
  "remote_address" : "10.<redacted>.11:55280",
  "backend_roles" : [
    ...
    "G_<redacted>-Opensearch-Admin",
    ...
  ],
  "custom_attribute_names" : [
    "attr.ldap.primaryGroupID",
    "attr.ldap.msTSExpireDate",
    "attr.ldap.logonCount",
    "attr.ldap.lastLogon",
    "attr.ldap.badPwdCount",
    "attr.ldap.userAccountControl",
    "attr.ldap.objectGUID",
    "attr.ldap.msTSLicenseVersion3",
    "attr.ldap.msTSLicenseVersion2",
    "attr.ldap.company",
    "attr.ldap.whenCreated",
    "ldap.original.username",
    "attr.ldap.physicalDeliveryOfficeName",
    "attr.ldap.lastLogoff",
    "attr.ldap.countryCode",
    "attr.ldap.department",
    "attr.ldap.instanceType",
    "attr.ldap.sAMAccountName",
    "attr.ldap.msTSManagingLS",
    "attr.ldap.userPrincipalName",
    "attr.ldap.objectClass",
    "attr.ldap.whenChanged",
    "attr.ldap.givenName",
    "ldap.dn",
    "attr.ldap.sAMAccountType",
    "attr.ldap.co",
    "attr.ldap.cn",
    "attr.ldap.accountExpires",
    "attr.ldap.description",
    "attr.ldap.title",
    "attr.ldap.dSCorePropagationData",
    "attr.ldap.initials",
    "attr.ldap.name",
    "attr.ldap.c",
    "attr.ldap.lockoutTime",
    "attr.ldap.uSNCreated",
    "attr.ldap.uSNChanged",
    "attr.ldap.displayName",
    "attr.ldap.objectSid",
    "attr.ldap.codePage",
    "attr.ldap.pwdLastSet",
    "attr.ldap.division",
    "attr.ldap.sn",
    "attr.ldap.msTSLicenseVersion",
    "attr.ldap.mail",
    "attr.ldap.lastLogonTimestamp"
  ],
  "roles" : [
    "own_index",
    "all_access"
  ],
  "tenants" : {
    "to_shift" : true,
    "pe_cs" : true,
    "global_tenant" : true,
    "dev_logs" : true,
    "stasfastov@<redacted>.net" : true,
    "pe_storages" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}
7

We had basically the same issue. Upgraded from 2.9 to 2.16 on a large stack and had the slow load times. We found a common message in the log was

{“type”:“log”,“@timestamp”:“2024-09-18T01:25:30Z”,“tags”:[“debug”,“opensearch”,“data”,“query”],“pid”:2986479,“message”:“200\nGET /_mapping?pretty=true\n[buffer]”}

Every time it froze up (spinning), it would eventually load and we’d get that log entry. It would happen about every 2 mins.

We finally realized it would happen when we would have dev tools page up. We proved that once they were all closed, we went 30 minutes with no issues. One of our platform engineers finally discovered in the GitHub repo a set of comments that read:

src/plugins/console/public/application/containers/settings.tsx:
// We’ll only retrieve settings if polling is on. The expectation here is that if the user
// disables polling it’s because they want manual control over the fetch request (possibly
// because it’s a very expensive request given their cluster and bandwidth). In that case,
// they would be unhappy with any request that’s sent automatically.

That lead us to realize we need to disable the autocomplete setting on the dev tools page. Once we did that and restarted opensearch dashboards, everything started working better again. With this feature being so expensive, I’m not sure why it’s on by default.

Hope this helps you with your issue.

8

@stas.fastov, (just curious), have you tried the solution suggested by @ecspot?
If yes, did it do the trick?

best,
mj

9

Hi, thanks for the suggestions.

I haven’t tried them, but we’ve upgraded to 2.17.1 and the problem is gone.

1 Like