OpenSearch Auth Fails to Retrieve Backend Roles from LDAP when Authenticating via Dashboards

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

  • OpenSearch: 2.17.1
  • OpenSearch Dashboards: 2.17.1

Deployed to IBM OpenShift Container Platform on LinuxONE (s390x).

Describe the issue:
My OpenSearch cluster is configured for LDAP Authentication and Authorization – see opensearch-security/config.yml below. Both authentication and authorization work perfectly fine when I use curl to query _plugins/_security/authinfo?pretty … it successfully authenticates my user and populates my backend roles.

However, when a user logs in via the Dashboards Web UI, it performs a successful authentication, but then fails to retrieve the backend roles.

My only work around at the moment is to pre-authenticate using curl prior to logging in to the Dashboards interface, since opensearch then has my backend-roles cached already.

For the logs below I did the following:

  1. Start opensearch cluster
  2. Execute curl -iku 'user1234:mypassword' https://opensearch:9200
  3. Capture opensearch logs showing good authentication and authorization
  4. Restart opensearch cluster to clear cached authc/authz results
  5. Log into the Dashboards Web UI using user1234
  6. Capture opensearch logs showing good authentication but no backend roles retrieved.

Configuration:

config/opensearch-security/config.yml

---
_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    authc:
      # Internal authentication (fallback, e.g. for admin)
      internal_auth_domain:
        description: "Internal authentication"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal

      # LDAP authentication
      my_ldap_domain:
        description: "LDAP authentication "
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: ldap
          config:
            enable_ssl: true  # Change to true if using LDAPS
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: false
            pemtrustedcas_filepath: '/usr/share/opensearch/config/ldap.mycompany.com.pem'
            hosts:
              - 192.168.100.51:636
            connect_timeout: 5000
            response_timeout: 0
            userbase: 'ou=all_users,ou=mycompany,o=com'
            usersearch: '(uid={0})'
            username_attribute: 'uid'

    authz:
      my_ldap_domain:
        description: "LDAP authorization"
        http_enabled: true
        transport_enabled: true
        authorization_backend:
          type: ldap
          config:
            enable_ssl: true  # Change to true if using LDAPS
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: false
            pemtrustedcas_filepath: '/usr/share/opensearch/config/ldap.mycompany.com.pem'
            hosts:
              - 192.168.100.51:636
            connect_timeout: 5000
            response_timeout: 0
            userbase: 'ou=all_users,ou=mycompany,o=com'
            usersearch: '(uid={0})'
            username_attribute: 'uid'
            resolve_nested_roles: false
            rolebase: 'ou=mycompanyGroup,ou=mycompany,o=com'
            rolesearch: '(memberuid={1})'
            rolename: 'cn'
            skip_users:
              - admin
              - logstash
              - dashboards

config/log4j2.properties:

status = error

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

rootLogger.level = warn
rootLogger.appenderRef.console.ref = console

logger.ldap.name = com.amazon.dlic.auth.ldap
logger.ldap.level = trace

Effective opensearch_dashboards.yml (with environment substitutions made manually)

opensearch.hosts: [https://opensearch:9200]
opensearch.ssl.verificationMode: none
opensearch.username: dashboards    # defined in internal-users.yml
opensearch.password: password1244  # defined in internal-users.yml
opensearch.requestHeadersWhitelist: [authorization, securitytenant]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: false
server.host: '0.0.0.0'

Relevant Logs or Screenshots:

When using curl to query _plugins/_security/authinfo, things work greak:

Command Line Output:

$ curl -iku "user1234:mA-RwJm3QWC6P9oLoHqh"  -H 'securitytenant: global_tenant' -X GET "https://opensearch.mycompany.com/opensearch/_plugins/_security/authinfo?pretty"    
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 1300
set-cookie: 3bea95fcf0f60fe0538a2beda7eeef18=45650bcf437c30b95a20327f975c6085; path=/; HttpOnly; Secure; SameSite=None                                                                                                                                                                                                          
{
  "user" : "User [name=user1234, backend_roles=[group1, group2, group3, group4, group5, group6, group7, group8, group9, group10, security, group12, group13], requestedTenant=global_tenant]",
  "user_name" : "user1234",
  "user_requested_tenant" : "global_tenant",
  "remote_address" : "10.128.2.2:44534",
  "backend_roles" : [
    "group1",
    "group2",
    "group3",
    "group4",
    "group5",
    "group6",
    "group7",
    "group8",
    "group9",
    "group10",
    "security",
    "group12",
    "group13"
  ],
  "custom_attribute_names" : [
    "ldap.dn",
    "attr.ldap.cn",
    "attr.ldap.ACLSOURCE",
    "attr.ldap.gidNumber",
    "attr.ldap.subschemaSubentry",
    "attr.ldap.createTimestamp",
    "attr.ldap.host",
    "attr.ldap.modifyTimestamp",
    "ldap.original.username",
    "attr.ldap.creatorsName",
    "attr.ldap.uidNumber",
    "attr.ldap.OWNERSOURCE",
    "attr.ldap.homeDirectory",
    "attr.ldap.loginShell",
    "attr.ldap.objectClass",
    "attr.ldap.IBM-ENTRYUUID",
    "attr.ldap.uid",
    "attr.ldap.modifiersName"
  ],
  "roles" : [
    "own_index",
    "all_access"
  ],
  "tenants" : {
    "global_tenant" : true,
    "admin_tenant" : true,
    "user1234" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}

OpenSearch Logs Generated by curl Command:

[2025-05-30T13:34:14,198][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Connect to 192.168.100.51:636
[2025-05-30T13:34:14,204][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Connect to ldaps://192.168.100.51:636
[2025-05-30T13:34:14,205][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] verifyHostname false:
[2025-05-30T13:34:14,205][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] trustall false:
[2025-05-30T13:34:14,211][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Use PEM to secure communication with LDAP server (client auth is false)
[2025-05-30T13:34:14,212][WARN ][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] In order to disable host name verification for LDAP connections (verify_hostnames: true), you also need to set set the system property com.sun.jndi.ldap.object.disableEndpointIdentification to true when starting the JVM running OpenSearch. This applies for all Java versions released since July 2018.
[2025-05-30T13:34:14,212][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] enabled ssl/tls protocols for ldaps [TLSv1.2, TLSv1.1]
[2025-05-30T13:34:14,212][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Connect timeout: PT5S/ResponseTimeout: PT0S
[2025-05-30T13:34:14,212][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] bindDn null, password <not set>
[2025-05-30T13:34:14,212][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Will perform anonymous bind because no bind dn is given
[2025-05-30T13:34:14,307][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Opened a connection, total count is now 1
[2025-05-30T13:34:14,324][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-1] Results for LDAP search for user1234 in base _legacyConfig:
[]
[2025-05-30T13:34:14,324][TRACE][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-1] Try to authenticate dn cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com
[2025-05-30T13:34:14,325][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Closed a connection, total count is now 0
[2025-05-30T13:34:14,325][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] bindDn cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com, password ****
[2025-05-30T13:34:14,901][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-1] Authenticated username user1234
[2025-05-30T13:34:14,902][DEBUG][c.a.d.a.l.u.Utils        ] [opensearch-1] Multiple values found for objectClass ([objectClass[posixAccount, IBM-NATIVEAUTHENTICATION, top, account, shadowAccount, LDAPPUBLICKEY]])
[2025-05-30T13:34:14,903][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] DBGTRACE (2): username: user1234 -> [<obfuscated ...>]
[2025-05-30T13:34:14,903][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] DBGTRACE (3): authenticatedUser: cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com -> [<obfuscated ...>]
[2025-05-30T13:34:14,903][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Try to get roles for cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com
[2025-05-30T13:34:14,903][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] user class: class com.amazon.dlic.auth.ldap.LdapUser
[2025-05-30T13:34:14,903][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] authenticatedUser: cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com
[2025-05-30T13:34:14,903][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] originalUserName: user1234
[2025-05-30T13:34:14,903][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] entry: [dn=cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com[[gidNumber[4000072]], [cn[FNAME LNAME - user1234]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[posixAccount, IBM-NATIVEAUTHENTICATION, top, account, shadowAccount, LDAPPUBLICKEY]], [loginShell[/bin/bash]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[72CBA000-5833-1839-883E-408562047A88]], [uid[user1234]], [uidNumber[2130252467]], [[homeDirectory[/home/user1234]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.470206Z]], [host[*]], [modifyTimestamp[20250530070317.204018Z]], [entryDN[cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com]]], responseControls=null, messageId=-1]
[2025-05-30T13:34:14,903][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] dn: null
[2025-05-30T13:34:14,903][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Connect to 192.168.100.51:636
[2025-05-30T13:34:14,904][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Connect to ldaps://192.168.100.51:636
[2025-05-30T13:34:14,904][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] verifyHostname false:
[2025-05-30T13:34:14,904][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] trustall false:
[2025-05-30T13:34:14,907][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Use PEM to secure communication with LDAP server (client auth is false)
[2025-05-30T13:34:14,907][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] enabled ssl/tls protocols for ldaps [TLSv1.2, TLSv1.1]
[2025-05-30T13:34:14,907][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Connect timeout: PT5S/ResponseTimeout: PT0S
[2025-05-30T13:34:14,907][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] bindDn null, password <not set>
[2025-05-30T13:34:14,907][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Will perform anonymous bind because no bind dn is given
[2025-05-30T13:34:14,965][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Opened a connection, total count is now 1
[2025-05-30T13:34:14,966][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com is a valid DN
[2025-05-30T13:34:14,966][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] DBGTRACE (4): authenticatedUser=cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com -> [<obfuscated ...>]
[2025-05-30T13:34:14,977][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] User found with DN cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com
[2025-05-30T13:34:14,977][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] DBGTRACE (6): dncn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com -> [<obfuscated ...>]
[2025-05-30T13:34:14,978][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] raw userRoleName(s): memberOf
[2025-05-30T13:34:14,979][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] User attr. ldap roles count: 0
[2025-05-30T13:34:14,979][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] User attr. ldap roles []
[2025-05-30T13:34:14,979][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] User attr. non-ldap roles count: 0
[2025-05-30T13:34:14,979][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] User attr. non-ldap roles []
[2025-05-30T13:34:14,981][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] roleName: cn
[2025-05-30T13:34:14,981][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] userRoleAttribute: null
[2025-05-30T13:34:14,981][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] rolesearch: (memberuid={1})
[2025-05-30T13:34:14,981][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] DBGTRACE (8): escapedDncn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com
[2025-05-30T13:34:15,015][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Results for LDAP group search for cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com in base convertedOldStyleSettings:
[[dn=cn=group4,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[4000814]], [cn[group4]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[CB678000-5833-1839-883E-408562047A88]], [memberUid[user1, user2, user3, user4, user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.833147Z]], [modifyTimestamp[20250530070315.833147Z]], [entryDN[cn=group4,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group2,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[87338991]], [cn[group2]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[BBBFA000-5833-1839-883E-408562047A88]], [memberUid[user1, user2, user3, user4, user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.769020Z]], [modifyTimestamp[20250530070315.769020Z]], [entryDN[cn=group2,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group13,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[789637]], [cn[group13]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[D4F64000-5833-1839-883E-408562047A88]], [memberUid[user1234]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.872299Z]], [modifyTimestamp[20250530070315.872299Z]], [entryDN[cn=group13,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group10,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[4000839]], [cn[group10]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[C42D0000-5833-1839-883E-408562047A88]], [memberUid[user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.803539Z]], [modifyTimestamp[20250530070315.803539Z]], [entryDN[cn=group10,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group12,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[4000855]], [cn[group12]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[D917C000-5833-1839-883E-408562047A88]], [memberUid[user1234, user5, user6, bldusr1 ]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.889214Z]], [modifyTimestamp[20250530070315.889214Z]], [entryDN[cn=group12,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group7,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[877639]], [cn[group7]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[D575D000-5833-1839-883E-408562047A88]], [memberUid[user1, user2, user3, user4, user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.874336Z]], [modifyTimestamp[20250530070315.874336Z]], [entryDN[cn=group7,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group3,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[4000835]], [cn[group3]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[C2F29000-5833-1839-883E-408562047A88]], [memberUid[user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.798509Z]], [modifyTimestamp[20250530070315.798509Z]], [entryDN[cn=group3,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group1,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[4000774]], [cn[group1]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[DF2B5000-5833-1839-883E-408562047A88]], [memberUid[bldusr1, user1, user2, user3, user4, user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.914105Z]], [modifyTimestamp[20250530070315.914105Z]], [entryDN[cn=group1,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group6,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[4000828]], [cn[group6]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[BDB1F000-5833-1839-883E-408562047A88]], [memberUid[user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.776994Z]], [modifyTimestamp[20250530070315.776994Z]], [entryDN[cn=group6,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group9,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[4000072]], [cn[group9]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[E68F1000-5833-1839-883E-408562047A88]], [memberUid[user1, user2, user3, user4, user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.944371Z]], [modifyTimestamp[20250530070315.944371Z]], [entryDN[cn=group9,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=security,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[4001066]], [cn[security]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[E37B6000-5833-1839-883E-408562047A88]], [memberUid[secscan, fsuser, user1, user2, user3, user4, user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.931770Z]], [modifyTimestamp[20250530070315.931770Z]], [entryDN[cn=security,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group5,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[4000779]], [cn[group5]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[C63E9000-5833-1839-883E-408562047A88]], [memberUid[user1, user2, user3, user4, user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.812012Z]], [modifyTimestamp[20250530070315.812012Z]], [entryDN[cn=group5,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1], [dn=cn=group8,ou=mycompanyGroup,ou=mycompany,o=com[[gidNumber[787767]], [cn[group8]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[top, posixGroup]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[E0AB6000-5833-1839-883E-408562047A88]], [memberUid[pzenasru, user1234, user5, user6]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.920248Z]], [modifyTimestamp[20250530070315.920248Z]], [entryDN[cn=group8,ou=mycompanyGroup,ou=mycompany,o=com]]], responseControls=null, messageId=-1]]
[2025-05-30T13:34:15,015][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] roles count total 13
[2025-05-30T13:34:15,015][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Evaluate nested roles
[2025-05-30T13:34:15,018][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] result nested attr count for depth 1 : 0
[2025-05-30T13:34:15,018][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] DBGTRACE (10): escapedDn cn=group4,ou=mycompanyGroup,ou=mycompany,o=com
[2025-05-30T13:34:15,020][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Results for LDAP group search for cn=group4,ou=mycompanyGroup,ou=mycompany,o=com in base convertedOldStyleSettings:
[]
[2025-05-30T13:34:15,183][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] returned user: User [name=user1234, backend_roles=[group1, group2, group3, group4, group5, group6, group7, group8, group9, group10, security, group12, group13], requestedTenant=null]
[2025-05-30T13:34:15,183][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-1] Closed a connection, total count is now 0

However when logging in via Dashboards, the opensearch logs look like this:

Login with Dashboards:

[2025-05-30T13:32:49,538][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Connect to 192.168.100.51:636
[2025-05-30T13:32:49,540][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Connect to ldaps://192.168.100.51:636
[2025-05-30T13:32:49,540][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] verifyHostname false:
[2025-05-30T13:32:49,540][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] trustall false:
[2025-05-30T13:32:49,543][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Use PEM to secure communication with LDAP server (client auth is false)
[2025-05-30T13:32:49,544][WARN ][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] In order to disable host name verification for LDAP connections (verify_hostnames: true), you also need to set set the system property com.sun.jndi.ldap.object.disableEndpointIdentification to true when starting the JVM running OpenSearch. This applies for all Java versions released since July 2018.
[2025-05-30T13:32:49,544][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] enabled ssl/tls protocols for ldaps [TLSv1.2, TLSv1.1]
[2025-05-30T13:32:49,544][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Connect timeout: PT5S/ResponseTimeout: PT0S
[2025-05-30T13:32:49,544][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] bindDn null, password <not set>
[2025-05-30T13:32:49,544][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Will perform anonymous bind because no bind dn is given
[2025-05-30T13:32:49,626][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Opened a connection, total count is now 1
[2025-05-30T13:32:49,644][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-2] Results for LDAP search for user1234 in base _legacyConfig:
[]
[2025-05-30T13:32:49,644][TRACE][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-2] Try to authenticate dn cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com
[2025-05-30T13:32:49,645][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Closed a connection, total count is now 0
[2025-05-30T13:32:49,645][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] bindDn cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com, password ****
[2025-05-30T13:32:50,243][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-2] Authenticated username user1234
[2025-05-30T13:32:50,243][DEBUG][c.a.d.a.l.u.Utils        ] [opensearch-2] Multiple values found for objectClass ([objectClass[posixAccount, IBM-NATIVEAUTHENTICATION, top, account, shadowAccount, LDAPPUBLICKEY]])
[2025-05-30T13:32:50,243][DEBUG][c.a.d.a.l.u.Utils        ] [opensearch-2] Multiple values found for sshpublickey ([)
[2025-05-30T13:32:50,244][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] DBGTRACE (2): username: user1234 -> [<obfuscated ...>]
[2025-05-30T13:32:50,244][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] DBGTRACE (3): authenticatedUser: cn=FNAME LNAME - user1234,ou=all_users,=mycompany,o=com -> [<obfuscated ...>]
[2025-05-30T13:32:50,244][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Try to get roles for cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com
[2025-05-30T13:32:50,244][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] user class: class com.amazon.dlic.auth.ldap.LdapUser
[2025-05-30T13:32:50,244][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] authenticatedUser: cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com
[2025-05-30T13:32:50,244][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] originalUserName: user1234
[2025-05-30T13:32:50,244][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] entry: [dn=cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com[[gidNumber[4000072]], [cn[FNAME LNAME - user1234]], [OWNERSOURCE[o=COM]], [modifiersName[cn=ldapAdmin]], [creatorsName[cn=ldapAdmin]], [objectClass[posixAccount, IBM-NATIVEAUTHENTICATION, top, account, shadowAccount, LDAPPUBLICKEY]], [loginShell[/bin/bash]], [subschemaSubentry[cn=Subschema]], [IBM-ENTRYUUID[72CBA000-5833-1839-883E-408562047A88]], [uid[user1234]], [uidNumber[2130252467]], [[homeDirectory[/home/user1234]], [ACLSOURCE[o=COM]], [createTimestamp[20250530070315.470206Z]], [host[*]], [modifyTimestamp[20250530070317.204018Z]], [entryDN[cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com]]], responseControls=null, messageId=-1]
[2025-05-30T13:32:50,244][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] dn: null
[2025-05-30T13:32:50,244][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Connect to 192.168.100.51:636
[2025-05-30T13:32:50,244][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Connect to ldaps://192.168.100.51:636
[2025-05-30T13:32:50,244][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] verifyHostname false:
[2025-05-30T13:32:50,245][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] trustall false:
[2025-05-30T13:32:50,246][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Use PEM to secure communication with LDAP server (client auth is false)
[2025-05-30T13:32:50,246][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] enabled ssl/tls protocols for ldaps [TLSv1.2, TLSv1.1]
[2025-05-30T13:32:50,246][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Connect timeout: PT5S/ResponseTimeout: PT0S
[2025-05-30T13:32:50,246][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] bindDn null, password <not set>
[2025-05-30T13:32:50,246][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Will perform anonymous bind because no bind dn is given
[2025-05-30T13:32:50,305][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Opened a connection, total count is now 1
[2025-05-30T13:32:50,305][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com is a valid DN
[2025-05-30T13:32:50,306][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] DBGTRACE (4): authenticatedUser=cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com -> [<obfuscated ...>]
[2025-05-30T13:32:50,312][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] User found with DN cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com
[2025-05-30T13:32:50,312][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] DBGTRACE (6): dncn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com -> [<obfuscated ...>]
[2025-05-30T13:32:50,313][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] raw userRoleName(s): memberOf
[2025-05-30T13:32:50,313][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] User attr. ldap roles count: 0
[2025-05-30T13:32:50,313][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] User attr. ldap roles []
[2025-05-30T13:32:50,313][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] User attr. non-ldap roles count: 0
[2025-05-30T13:32:50,313][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] User attr. non-ldap roles []
[2025-05-30T13:32:50,313][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] roleName: cn
[2025-05-30T13:32:50,313][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] userRoleAttribute: null
[2025-05-30T13:32:50,313][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] rolesearch: (memberuid={1})
[2025-05-30T13:32:50,313][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] DBGTRACE (8): escapedDncn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com
[2025-05-30T13:32:50,317][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Results for LDAP group search for cn=FNAME LNAME - user1234,ou=all_users,ou=mycompany,o=com in base convertedOldStyleSettings:
[]
[2025-05-30T13:32:50,317][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] roles count total 0
[2025-05-30T13:32:50,317][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Evaluate nested roles
[2025-05-30T13:32:50,317][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Roles for user1234 -> []
[2025-05-30T13:32:50,317][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] returned user: User [name=user1234, backend_roles=[], requestedTenant=null]
[2025-05-30T13:32:50,317][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-2] Closed a connection, total count is now 0
2

Quick update after a few more hours of pouring through docs. I was so focused on the LDAP side that I neglected my proxy setup, and I believe that’s the root of the issue.

My topology currently has an HAProxy in front of a 3-node Dashboard cluster, which uses another HAProxy as the opensearch.hosts to connect to the 3-node OS Cluster. Both proxies have sticky sessions based on source, but I suspect the forwarded-for and auth headers passing through 2 separate proxies without proxy auth configured is probably what’s causing the issue.

I am going to do the following later today. Will post back with results.

  1. Remove the HAProxy between Dashboards and OS and just list all 3 nodes as opensearch.hosts values
  2. Enable proxy authentication in my security config
4

This turned out to be unrelated to Opensearch all together.

My LDAP backend was performing case insensitive against the uid field during authorization, but case sensitive matches against the memberuid field during authorization.

My password manager was auto completing my username in all caps in the browser, but I was using all lowercase on the command line testing authinfo :/.

I notified my LDAP admin to request they update the memberuid attribute definition.

Meanwhile, I set userrolename to uid and use the {2} substitution in my role filter to work around it. This way it uses the username as it appears in the authenticated user’s directory entry, rather than the original username as the user typed it.