LDAP com.amazon.dlic.auth.ldap.LdapUser is not serializable

Security
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

opensearch-dashboards-2.13.0-1.x86_64
opensearch-2.13.0-1.x86_64

ServerOS: Oracle Linux Server 8.9
Browser: Tested in Chrome/Firefox

Describe the issue:
After an update from Opensearch 2.11 to 2.13, the LDAP integration, which had worked well for years until then, no longer works, see log output.
As LDAP server we use freeIPA in the version: 4.9.12 if this should be relevant.
We use backend roles with “-” and “_” included, but this has never been a problem so far.

Configuration:

  • /etc/opensearch/opensearch-security/config.yml
config:
  dynamic:
    do_not_fail_on_forbidden: true
    http:
      anonymous_auth_enabled: false
    authc:
      BOR-IPA-LDAP:
        description: "Authenticate via IPA LDAP"
        http_enabled: true
        transport_enabled: false
        order: 15
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: ldap
          config:
            connect_timeout: 3000
            response_timeout: 3000
            verify_hostnames: true
            enable_start_tls: true
            enabled_ssl_ciphers:
              - "TLS_DHE_RSA_WITH_AES_256_CBC_SHA"
              - "TLS_DHE_DSS_WITH_AES_128_CBC_SHA256"
            enabled_ssl_protocols:
              - "TLSv1.2"
            hosts:
              - xxx1:389
              - xxx2:389
              - xxx3:389
            bind_dn: uid=ldapbind,cn=users,cn=accounts,dc=xxx,dc=xxx,dc=de
            password: <secret>
            userbase: 'cn=users,cn=accounts,dc=xxx,dc=xxx,dc=de'
            usersearch: '(uid={0})'
            username_attribute: uid
    authz:
      BOR-IPA:
        http_enabled: true
        transport_enabled: false
        authorization_backend:
          type: ldap
          config:
            enable_start_tls: true
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
              - xxx1:389
              - xxx2:389
              - xxx3:389
            bind_dn: uid=ldapbind,cn=users,cn=accounts,dc=xxx,dc=xxx,dc=de
            password: <secret>
            userbase: 'cn=users,cn=accounts,dc=xxx,dc=xxx,dc=de'
            usersearch: '(uid={0})'
            username_attribute: uid
            rolesearch: '(member={0})'
            rolebase: 'cn=groups,cn=accounts,dc=xxx,dc=xxx,dc=de'
            rolename: cn
            rolesearch_enabled: true
            resolve_nested_roles: false

Relevant Logs or Screenshots:

[2024-04-30T14:41:50,001][WARN ][r.suppressed             ] [xxx-logs1.xxx.xxxxxx.de] path: /_index_template/*, params: {name=*}
org.opensearch.transport.TransportException: failure to send
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1757) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:844) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.executeOnClusterManager(TransportClusterManagerNodeAction.java:436) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.doStart(TransportClusterManagerNodeAction.java:309) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.tryAction(TransportClusterManagerNodeAction.java:239) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction$1.doRun(RetryableAction.java:139) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction.run(RetryableAction.java:117) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction.doExecute(TransportClusterManagerNodeAction.java:200) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction.doExecute(TransportClusterManagerNodeAction.java:88) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:77) [opensearch-performance-analyzer-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:395) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.rest.action.admin.indices.RestGetComposableIndexTemplateAction.lambda$prepareRequest$0(RestGetComposableIndexTemplateAction.java:87) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.filter.SecurityRestFilter$AuthczRestHandler.handleRequest(SecurityRestFilter.java:190) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.rest.RestController.dispatchRequest(RestController.java:334) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:425) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.rest.RestController.dispatchRequest(RestController.java:263) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:69) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:387) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:468) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:370) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:56) [transport-netty4-client-2.13.0.jar:2.13.0]
        at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-2.13.0.jar:2.13.0]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72) [transport-netty4-client-2.13.0.jar:2.13.0]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) [netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) [netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) [netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) [netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.107.Final.jar:4.1.107.Final]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.OpenSearchException: Instance User [name=rke, backend_roles=[null, xxx, xxx-xxx, xxx_xxx_xxx, xxx_xxx, Xxx-xxx, xxx-xxx], requestedTenant=xxx] of class class com.amazon.dlic.auth.ldap.LdapUser is not serializable
        at org.opensearch.security.support.Base64CustomHelper.serializeObject(Base64CustomHelper.java:104) ~[?:?]
        at org.opensearch.security.support.Base64Helper.serializeObject(Base64Helper.java:34) ~[?:?]
        at org.opensearch.security.transport.SecurityInterceptor.ensureCorrectHeaders(SecurityInterceptor.java:329) ~[?:?]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:242) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) ~[?:?]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) ~[opensearch-2.13.0.jar:2.13.0]
        ... 96 more
Caused by: java.lang.NullPointerException: Cannot invoke "String.length()" because "str" is null
        at org.opensearch.core.common.io.stream.StreamOutput.writeString(StreamOutput.java:444) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1196) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.common.io.stream.StreamOutput.writeStringCollection(StreamOutput.java:1208) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.security.user.User.writeTo(User.java:260) ~[?:?]
        at com.amazon.dlic.auth.ldap.LdapUser.writeTo(LdapUser.java:102) ~[?:?]
        at org.opensearch.security.support.Base64CustomHelper.serializeObject(Base64CustomHelper.java:88) ~[?:?]
        at org.opensearch.security.support.Base64Helper.serializeObject(Base64Helper.java:34) ~[?:?]
        at org.opensearch.security.transport.SecurityInterceptor.ensureCorrectHeaders(SecurityInterceptor.java:329) ~[?:?]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:242) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) ~[?:?]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) ~[opensearch-2.13.0.jar:2.13.0]
        ... 96 more

If we have deleted the .kibana_username index of the user and want to recreate it, it no longer works with the message:

> Blockquote

2

Hi @rkeatboreus.de,

Do you know where the null value comes from, could you please compare the list above with the expected values ( rolesearch: (member={0}), rolename: cn)?

Best,
mj

3

No, actually I can’t explain why it says “null” once, when I run an ldapsearch as follows, I don’t get any “null” values back:

# ldapsearch -Z -LLL -x -W -H ldap://xxx-xxx1.xxx.xxxxxx.de:389 -D "uid=ldapbind,cn=users,cn=accounts,dc=xxxx,dc=xxxxxx,dc=xx" -b "cn=groups,cn=accounts,dc=xxx,dc=xxx,dc=de" member=uid=rke* cn:

dn: cn=xxx,cn=groups,cn=accounts,dc=xxx,dc=xxx,dc=de

dn: cn=xxx_xxxxx,cn=groups,cn=accounts,dc=xxx,dc=xxx,dc=de

dn: cn=xxxx-xxxx,cn=groups,cn=accounts,dc=xxx,dc=xxx,dc=de

whether I enter member={0} or member=uid={0} in the opensearch config apparently makes no difference, at least on my tests…

In addition: If we have deleted the .kibana_username index of the user and want to recreate it, it no longer works with the message:
Request to create index .kibana_1824975290_xxx_1 with aliases [.kibana_1824975290_xxx] failed, failing IndexRequest

4

Hi @rkeatboreus.de,

Do you have access directly to your LDAP, rather than the ldapsearch, to check the membership of your user rke ? To be sure there are no malformed values (group cn) that ldapsearch is ignoring and OpenSearch security is translating to null.

Why would you delete the .kibana_username index?

Best,
mj

5

The question is what is meant by “malformed values” at this point and whether the opensearch really resolves them to “null” in case of doubt.

There are definitely nested groups there, but the opensearch should not resolve them as set in the config, they are not relevant for the authorization either, otherwise I don’t see anything bad about the groups, but as I said, I lack the criterion for this.

We deleted the .kibana_username index because we suspected that it may have been damaged during the update.

What is also interesting is that some users do not have the problem, even though they have the same groups in LDAP.
If you create a new user, different user name, but same groups, then the .kibana_username index is created and everything works, but for exsisting users after deleting the .kibana_username index it doesn’t work.
therefore the assumption that it could possibly be related to the .kibana_username index

6

Currently, for example, I have the condition that I can log in to the dashboard with my user and when I select Tentants, I can select some and not others, then I get the following error in the dashboard:

{"statusCode":500, "error": "Internal Server Error", "message": "An internal server error occurred."}

And the above-mentioned error in the Opensearch log.
Has anything changed in LDAP authentication between versions 2.11 and 2.13?
At least I didn’t find anything in the release nodes

7

Hi @rkeatboreus.de ,

I am not aware of any changes and haven’t seen any increase in reports on issues with it either.

Could you run the below on a failing user and a working correctly one:

curl --insecure -u <ldap_user>:<ldap_password> -XGET https://<OS_node>:9200/_plugins/_security/authinfo?pretty

Thanks,
mj

8

Hi @Mantas,

@rkeatboreus.de will be reachable again by tomorrow.
I had a look on the Systems and created a test user for this purpose.
Currently everyone has the Problem of not being able to log in or select tenants which depends on the kibana index for the user. if it is already present, the login is sometimes possible but when a specific tenant is selected, it tries to perform some action with the index and fails there, see error messages mentioned in previous messages.
If the index is not present, the login will fail because opensearch tries to create the index and fails.

I noticed, that the problem occurs, when a group containing an HBAC-Rule was assigned to the user.
The name of the HBAC -rule also appears in the backend_roles section in the output of the command you provided. The HBAC-Rule was redacted with exclamation marks (!) and general values are redacted using “x”.
When the User does not have a group with an HBAC-Rule, everything works as expected.
It was tested with multiple groups and different HBAC-Rules and the behaviour was identical.

Here the Outputs of the command, one working and without the one group, one broken and with the group:

------
broken (With HBAC-Rule)
------
cle@xxxxxxxxx ~ $ curl --insecure -u testuser:<PASSWORD> -XGET https://xxxxxxxx2.xxx.xxxxxx.xx:9200/_plugins/_security/authinfo?pretty
{
  "user" : "User [name=testuser, backend_roles=[null, xxx_xxxxxx, xxxx_xxxx, xxx-xxxxxxxxxx_xxxxx, !!!!-!!!!, xxx-xxxxxxxxxx_xxxxxxxxxx, xxx-xxxxxxxxxx_xxxxx, xxx-xxxxxxxxxx_xxxxxx, xxxxxxxx], requestedTenant=null]",
  "user_name" : "testuser",
  "user_requested_tenant" : null,
  "remote_address" : "xxx.xxx.xxx.xxx:4xxxx",
  "backend_roles" : [
    null,
    "xxx_xxxxxx",
    "xxxx_xxxx",
    "xxx-xxxxxxxxxx_xxxxx",
    "!!!!-!!!!",
    "xxx-xxxxxxxxxx_xxxxxxxxxx",
    "xxx-xxxxxxxxxx_xxxxx",
    "xxx-xxxxxxxxxx_xxxxxx",
    "xxxxxxxx"
  ],
  "custom_attribute_names" : [
    "ldap.dn",
    "attr.ldap.cn",
    "attr.ldap.gidNumber",
    "attr.ldap.krbCanonicalName",
    "attr.ldap.initials",
    "attr.ldap.createTimestamp",
    "attr.ldap.modifyTimestamp",
    "ldap.original.username",
    "attr.ldap.uidNumber",
    "attr.ldap.gecos",
    "attr.ldap.displayName",
    "attr.ldap.ipaUniqueID",
    "attr.ldap.sn",
    "attr.ldap.krbPrincipalName",
    "attr.ldap.entryusn",
    "attr.ldap.homeDirectory",
    "attr.ldap.krbLastPwdChange",
    "attr.ldap.loginShell",
    "attr.ldap.objectClass",
    "attr.ldap.parentid",
    "attr.ldap.uid",
    "attr.ldap.mail",
    "attr.ldap.givenName"
  ],
  "roles" : [
    null,
    "xxx_xxxxx",
    "xxxx_xxxx",
    "xxxxx",
    "xxxxxx",
    "!!!!-!!!!",
    "xxx-xxxxxxxxxx_xxxxxxxxxx",
    "xxx_xxxxxx",
    "xxx-xxxxxxxxxx_xxxxx",
    "xxx-xxxxxxxxxx_xxxxxx",
    "xxxxxxxxxx",
    "xxxxxxxx",
    "xxx_xxxxxx",
    "xxxxx",
    "xxx-xxxxxxxxxx_xxxxx"
  ],
  "tenants" : {
    "xxxxx" : true,
    "testuser" : true,
    "xxxxxxxxxx" : true,
    "global_tenant" : true,
    "xxxxxxx" : true,
    "xxxxxx" : true,
    "xxxxx" : true,
    "xxx" : true,
    "xxxxxxxxx" : true,
    "xxxxxx" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}

-------
working
-------
cle@xxxxxxxx2 ~ $ curl --insecure -u testuser:<PASSWORD> -XGET https://xxxxxxxx2.xxx.xxxxxx.de:9200/_plugins/_security/authinfo?pretty
{
  "user" : "User [name=testuser, backend_roles=[xxx_xxxxxx, xxx-xxxxxxxxxx_xxxxx, xxx-xxxxxxxxxx_xxxxxxxxxx, xxx-xxxxxxxxxx_xxxxx, xxx-xxxxxxxxxx_xxxxxx, xxxxxxxx], requestedTenant=null]",
  "user_name" : "testuser",
  "user_requested_tenant" : null,
  "remote_address" : "xxx.xxx.xxx.xxx:4xxxx",
  "backend_roles" : [
    "xxx_xxxxxx",
    "xxx-xxxxxxxxxx_xxxxx",
    "xxx-xxxxxxxxxx_xxxxxxxxxx",
    "xxx-xxxxxxxxxx_xxxxx",
    "xxx-xxxxxxxxxx_xxxxxx",
    "xxxxxxxx"
  ],
  "custom_attribute_names" : [
    "ldap.dn",
    "attr.ldap.cn",
    "attr.ldap.gidNumber",
    "attr.ldap.krbCanonicalName",
    "attr.ldap.initials",
    "attr.ldap.createTimestamp",
    "attr.ldap.modifyTimestamp",
    "ldap.original.username",
    "attr.ldap.uidNumber",
    "attr.ldap.gecos",
    "attr.ldap.displayName",
    "attr.ldap.ipaUniqueID",
    "attr.ldap.sn",
    "attr.ldap.krbPrincipalName",
    "attr.ldap.entryusn",
    "attr.ldap.homeDirectory",
    "attr.ldap.krbLastPwdChange",
    "attr.ldap.loginShell",
    "attr.ldap.objectClass",
    "attr.ldap.parentid",
    "attr.ldap.uid",
    "attr.ldap.mail",
    "attr.ldap.givenName"
  ],
  "roles" : [
    "xxx_xxxxxx",
    "xxx_xxxxx",
    "xxxxx",
    "xxxxx",
    "xxxxxx",
    "xxx-xxxxxxxxxx_xxxxx",
    "xxx-xxxxxxxxxx_xxxxxxxxxx",
    "xxx_xxxxxx",
    "xxx-xxxxxxxxxx_xxxxx",
    "xxx-xxxxxxxxxx_xxxxxx",
    "xxxxxxxxxx",
    "xxxxxxxx"
  ],
  "tenants" : {
    "xxxxx" : true,
    "testuser" : true,
    "xxxxxxxxxx" : true,
    "global_tenant" : true,
    "xxxxxxx" : true,
    "xxxxxx" : true,
    "xxxxx" : true,
    "xxx" : true,
    "xxxxxxxxx" : true,
    "xxxxxx" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}
9

Hi @cle,

I suspect that the backend_role null value (as per above) is breaking your permissions, this is coming from your LDAP, can you find out where it is coming from?

best,
mj

10

Hi @Mantas,

I can not see any null values in the response coming from the LDAP when requesting the groups of a user with ldapsearch like @rkeatboreus.de did in one of the previous comments.
I have also looked in the packages directly and do not see a null value in the response there (when running the ldapsearch command).
At the moment we do not have any clue where the null value is coming from.