Disable custom attributes for LDAP backend

I’m configuring LDAP (Active Directory) for authorization and facing with the issue that pulling user info (GET /_opendistro/_security/authinfo) takes from 5 to 10 seconds.

The first thing I’ve noticed that it pulls all LDAP attributes of the user. I’m trying to disable all attributes with custom_attr_* settings:

    roles_from_myldap:
        http_enabled: true
        transport_enabled: true
        authorization_backend:
          type: ldap 
          config:
            ...
            custom_attr_whitelist: []
            custom_attr_maxval_len: 0

But it has no effect in my case: all attributes are still shown and query takes 5-10 sec. What could be wrong?
Appreciate any help!

what ODFE version are you on?