[Solved] Unable to log in with LDAP after upgrade to 3.3

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.3.1 (dashboards 3.3.0)

Describe the issue:

Today I did a rolling restart of my 17-node OpenSearch cluster.

I started on version 2.19.0 and went straight to 3.3.1. During the rolling upgrade, everything went smoothly and I was still able to log in. However, after the final server had updated, and I also upgraded my Dashboards Docker container, I could no longer log in with LDAP.

Internal users still work, however I did have to manually reset their passwords. I saw a Github issue about this, and it mentioned new hashing methods. Probably related? Issue mentioned there would be a migration guide, but I’ve found no such thing.

When I just do a simple curl basic auth login against /_cluster/health, these errors show up in my server logs:

Cannot retrieve roles for User [name=MYUSER, backend_roles=[], requestedTenant=null] from ldap due to OpenSearchSecurityException[java.lang.NullPointerException: Cannot invoke "org.ldaptive.Connection.getProviderConnection()" because the return value of "org.ldaptive.SearchOperation.getConnection()" is null]; nested: NullPointerException[Cannot invoke "org.ldaptive.Connection.getProviderConnection()" because the return value of "org.ldaptive.SearchOperation.getConnection()" is null];

and afterwards

No cluster-level perm match for User [name=MYUSER, backend_roles=[], requestedTenant=null] Resolved [aliases=[*], allIndices=[*], types=[*], originalRequested=[*], remoteIndices=[]] [Action [cluster:monitor/health]] [RolesChecked []]. No permissions for [cluster:monitor/health]

I did try reapplying my security configs with securityadmin.sh, but to no avail (now I’ve probably broken a bunch of updates since last backup..).

I did try enabling the new features on Dashboards it suggested (data source, workspace, explore), but that didn’t help and I’ve disabled them again.

Any tips? Do I need to migrate something?

2

Hey @rcz ,

Could you share your config please?

Leeroy.

3

Also could you test and confirm that you can get connection using ldapsearch and paste the result here

4

I can connect from the server, like so:

$ ldapwhoami -vvv -D <admin_dn> -x -W
ldap_initialize(  )
Enter LDAP Password:
u:<admin_dn>
Result: Success (0)

Here’s my security config.yml

---
_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    kibana:
      multitenancy_enabled: true
      server_username: kibanaserver
    do_not_fail_on_forbidden: true
    http:
      anonymous_auth_enabled: false
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 2
        http_authenticator:
          type: "basic"
          challenge: true
        authentication_backend:
          type: "intern"
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: false
        transport_enabled: true
        order: 1
        http_authenticator:
          type: "clientcert"
          config:
            username_attribute: "CN"
          challenge: false
        authentication_backend:
          type: "noop"
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: false
        order: 3
        http_authenticator:
          type: "basic"
          challenge: false
        authentication_backend:
          type: "ldap"
          config:
            pemtrustedcas_filepath: "/etc/opensearch/certs/ca.pem"
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
            - "REDACTED:636"
            bind_dn: "CN=REDACTED"
            password: "REDACTED"
            userbase: "OU=Users,DC=int,DC=REDACTED"
            usersearch: "(sAMAccountName={0})"
            username_attribute: "sAMAccountName"
    authz:
      roles_from_my_ldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: false
        authorization_backend:
          type: "ldap"
          config:
            pemtrustedcas_filepath: "/etc/opensearch/certs/ca.pem"
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
            - "REDACTED:636"
            bind_dn: "CN=REDACTED,CN=Users,DC=int,DC=REDACTED"
            password: "REDACTED"
            userrolename: "memberOf"
            rolename: "cn"
            resolve_nested_roles: false
            rolesearch_enabled: false
            userbase: "OU=Users,DC=int,DC=REDACTED"
            usersearch: "(sAMAccountName={0})"
            skip_users:
            - "rpa"
            - "testuser"
            - "monitor"
            - "kibanaserver"

I used this config all the way since Elasticsearch OSS, to ODFE, then OpenSearch, with minor changes. And again, it worked fine until the final server (and Dashboards) was updated.

Any help is greatly appreciated!

EDIT: I should mention, it seems authentication works fine as it reports wrong password. I only reach the Missing Role page after logging in correctly.

5

I changed the config to not use userrolename and userroleattribute, but instead use rolebase and rolesearch, and now it works, for some reason..

Thanks anyway :slight_smile: