Getting occasional "Authentication finally failed" for OpenSearch Connector

Security
1

OpenSearch Version: 2.14.0
3-node cluster, hosted on K8s, operated through OpenSearch Operator

We are running a CDC system feeding aggregated data from Kafka via a Kafka Connect OpenSearch connector into OpenSearch.

The connector uses a dedicated account to authenticate against OpenSearch.

roles.yml:

    kafka_sink:
      index_permissions:
        - index_patterns:
            - "<redacted>.*"
          allowed_actions:
            - "indices:admin/exists"
            - "indices:admin/create"
            - "indices:admin/get"
            - "indices:admin/mapping/put"
            - "indices:data/write/index"
            - "indices:data/write/bulk"
            - "indices:data/write/bulk*"
            - "indices:data/write/update"
            - "indices:data/write/delete"

internal_users.yml:

    opensearchconnector:
      hash: "<redacted>"
      reserved: false
      description: "User for Kafka Connect Opensearch Connector"
      opendistro_security_roles:
        - "kafka_sink"

While most of the time this works as expected, we see occasional ‘authentication finally failed’ messages in OpenSearch logs:
[2024-06-21T14:11:28,057][WARN ][o.o.s.a.BackendRegistry ] [opensearch-nodes-0] Authentication finally failed for opensearchconnector from 10.96.21.147:57438

At the same time, the connector logs a series of exceptions and shuts down with an unrecoverable error.

OpenSearchStatusException[Unable to parse response body]; nested: ResponseException[method [POST], host [https://opensearch-nodes.opensearch.svc:9200], URI [/_bulk?timeout=1m], status line [HTTP/1.1 401 Unauthorized]

	Suppressed: java.lang.IllegalStateException: Unsupported Content-Type: text/plain; charset=UTF-8

Caused by: org.opensearch.client.ResponseException: method [POST], host [https://opensearch-nodes.opensearch.svc:9200], URI [/_bulk?timeout=1m], status line [HTTP/1.1 401 Unauthorized]

Security auditlogs show no clue as to why this happens.

{
  "_index": "security-auditlog-2024.06.21",
  "_id": "pyojO5ABUlTnn7vhTnq5",
  "_version": 1,
  "_score": null,
  "_source": {
    "audit_cluster_name": "opensearch",
    "audit_rest_request_params": {
      "timeout": "1m"
    },
    "audit_node_name": "opensearch-nodes-0",
    "audit_rest_request_method": "POST",
    "audit_category": "FAILED_LOGIN",
    "audit_request_origin": "REST",
    "audit_node_id": "pGVu2uoOQ_-_SITOl0grQw",
    "audit_request_layer": "REST",
    "audit_rest_request_path": "/_bulk",
    "@timestamp": "2024-06-21T14:11:28.057+00:00",
    "audit_request_effective_user_is_admin": false,
    "audit_format_version": 4,
    "audit_request_remote_address": "10.96.21.147",
    "audit_node_host_address": "10.96.9.139",
    "audit_rest_request_headers": {
      "Connection": [
        "Keep-Alive"
      ],
      "User-Agent": [
        "Apache-HttpAsyncClient/4.1.5 (Java/17.0.10)"
      ],
      "Host": [
        "opensearch-nodes.opensearch.svc:9200"
      ],
      "Content-Length": [
        "4219"
      ],
      "Content-Type": [
        "application/json"
      ]
    },
    "audit_request_effective_user": "opensearchconnector",
    "audit_node_host_name": "opensearch-nodes-0"
  },
  "fields": {
    "@timestamp": [
      "2024-06-21T14:11:28.057Z"
    ]
  },
  "sort": [
    1718979088057
  ]
}

How can I figure out what is actually going on here? The connector shutting down in fact breaks our CDC chain as it fails to deliver index documents afterwards until manual connector restart.

2

Hi @distant.calls,

Have you tried setting your logging to Debug mode to find any clues?

please see more here:

What are the use patterns on your cluster (bulk part of API calls per time unit)?

best,
mj

3

@Mantas thanks for getting back. I’ve set org.opensearch.rest to DEBUG level logging, but as of yet, only metric-related information has been logged. (As a consequence, I’ve also set org.opensearch.rest.prometheus back to INFO.)

PUT /_cluster/settings
{
  "persistent" : {
    "logger.org.opensearch.rest" : "DEBUG"
  }
}

In fact I’m not even sure which logs could be interesting in order to isolate the issue at hand (package-wise).

Does OpenSearch provide metrics related to _bulk calls? I’m pretty sure the mentioned connector uses the bulk API, but I haven’t found any usage stats yet.

4

@distant.calls, have you tried increasing the log level in log4j2.properties opensearch/config/log4j2.properties to get more clues on the cause.

something like: rootLogger.level = debug

Note the logs will become very chatty.

best,
mj

5

@Mantas Perhaps I’ve managed to capture the source using _root=DEBUG. Around the time when the connector dies with an HTTP 401 error, OpenSearch logs show this lengthy part of information:

...
request: BulkShardRequest [[<redacted>.person.document][0]] containing [update {[<redacted>.person.document][139766999], doc_as_upsert[true], doc[index {[null][null], source[<redacted>],"person":<redacted>,"type":"person"}]}], upsert[index {[null][null], source[<redacted>], scripted_upsert[false], detect_noop[true]}]
evaluatedDlsFlsConfig: EvaluatedDlsFlsConfig [dlsQueriesByIndex={}, flsByIndex={}, fieldMaskingByIndex={}]
resolved: Resolved [aliases=[], allIndices=[<redacted>.person.document], types=[*], originalRequested=[<redacted>.person.document], remoteIndices=[]]
mode: ADAPTIVE
[2024-06-26T06:41:59,913][DEBUG][o.o.t.TransportService   ] [opensearch-nodes-1] Action: indices:data/write/bulk[s]
[2024-06-26T06:41:59,914][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Check authdomain for rest ldap/1 or 2 in total
[2024-06-26T06:41:59,915][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Clear user cache for opensearchconnector due to EXPIRED
[2024-06-26T06:41:59,915][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-nodes-0] Connect timeout: PT5S/ResponseTimeout: PT0S
[2024-06-26T06:41:59,915][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-nodes-0] bindDn cn=readonly,dc=<redacted>,dc=de, password ****
[2024-06-26T06:41:59,915][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-nodes-0] Will perform simple bind with bind dn
[2024-06-26T06:41:59,927][DEBUG][o.l.BindOperation        ] [opensearch-nodes-0] execute request=[org.ldaptive.BindRequest@888246425::bindDn=cn=readonly,dc=<redacted>,dc=de, saslConfig=null, controls=null, referralHandler=null, intermediateResponseHandlers=null] with connection=[org.ldaptive.DefaultConnectionFactory$DefaultConnection@2070013520::config=[org.ldaptive.ConnectionConfig@1605569633::ldapUrl=ldap://ldap.<redacted>.de:10389, connectTimeout=PT5S, responseTimeout=PT0S, sslConfig=null, useSSL=false, useStartTLS=false, connectionInitializer=[org.ldaptive.BindConnectionInitializer@1363518629::bindDn=cn=readonly,dc=<redacted>,dc=de, bindSaslConfig=null, bindControls=null], connectionStrategy=org.ldaptive.DefaultConnectionStrategy@6908834c], providerConnectionFactory=[org.ldaptive.provider.jndi.JndiConnectionFactory@489667419::metadata=[ldapUrl=ldap://ldap.<redacted>.de:10389, count=1], environment={com.sun.jndi.ldap.connect.timeout=5000, java.naming.ldap.version=3, java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory, com.sun.jndi.ldap.read.timeout=0}, classLoader=null, providerConfig=[org.ldaptive.provider.jndi.JndiProviderConfig@1522035867::operationExceptionResultCodes=[PROTOCOL_ERROR, SERVER_DOWN], properties={}, controlProcessor=org.ldaptive.provider.ControlProcessor@26a48be5, environment=null, tracePackets=null, removeDnUrls=true, searchIgnoreResultCodes=[TIME_LIMIT_EXCEEDED, SIZE_LIMIT_EXCEEDED, PARTIAL_RESULTS], classLoader=null, sslSocketFactory=null, hostnameVerifier=null]], providerConnection=org.ldaptive.provider.jndi.JndiConnection@b9b7e9e]
[2024-06-26T06:41:59,929][DEBUG][o.l.BindOperation        ] [opensearch-nodes-0] execute response=[org.ldaptive.Response@1796139477::result=null, resultCode=SUCCESS, message=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1] for request=[org.ldaptive.BindRequest@888246425::bindDn=cn=readonly,dc=<redacted>,dc=de, saslConfig=null, controls=null, referralHandler=null, intermediateResponseHandlers=null] with connection=[org.ldaptive.DefaultConnectionFactory$DefaultConnection@2070013520::config=[org.ldaptive.ConnectionConfig@1605569633::ldapUrl=ldap://ldap.<redacted>.de:10389, connectTimeout=PT5S, responseTimeout=PT0S, sslConfig=null, useSSL=false, useStartTLS=false, connectionInitializer=[org.ldaptive.BindConnectionInitializer@1363518629::bindDn=cn=readonly,dc=<redacted>,dc=de, bindSaslConfig=null, bindControls=null], connectionStrategy=org.ldaptive.DefaultConnectionStrategy@6908834c], providerConnectionFactory=[org.ldaptive.provider.jndi.JndiConnectionFactory@489667419::metadata=[ldapUrl=ldap://ldap.<redacted>.de:10389, count=1], environment={com.sun.jndi.ldap.connect.timeout=5000, java.naming.ldap.version=3, java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory, com.sun.jndi.ldap.read.timeout=0}, classLoader=null, providerConfig=[org.ldaptive.provider.jndi.JndiProviderConfig@1522035867::operationExceptionResultCodes=[PROTOCOL_ERROR, SERVER_DOWN], properties={}, controlProcessor=org.ldaptive.provider.ControlProcessor@26a48be5, environment=null, tracePackets=null, removeDnUrls=true, searchIgnoreResultCodes=[TIME_LIMIT_EXCEEDED, SIZE_LIMIT_EXCEEDED, PARTIAL_RESULTS], classLoader=null, sslSocketFactory=null, hostnameVerifier=null]], providerConnection=org.ldaptive.provider.jndi.JndiConnection@b9b7e9e]
[2024-06-26T06:41:59,929][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-nodes-0] Opened a connection, total count is now 1
[2024-06-26T06:41:59,929][DEBUG][o.l.SearchOperation      ] [opensearch-nodes-0] execute request=[org.ldaptive.SearchRequest@1356983043::baseDn=ou=users,dc=<redacted>,dc=de, searchFilter=[org.ldaptive.SearchFilter@457579389::filter=(cn={0}), parameters={0=opensearchconnector}], returnAttributes=[*, +], searchScope=SUBTREE, timeLimit=PT0S, sizeLimit=0, derefAliases=ALWAYS, typesOnly=false, binaryAttributes=null, sortBehavior=UNORDERED, searchEntryHandlers=null, searchReferenceHandlers=[org.ldaptive.referral.SearchReferralHandler$SearchReferenceHandler@724b8c9b], controls=null, referralHandler=org.ldaptive.referral.SearchReferralHandler@70b2ca94, intermediateResponseHandlers=null] with connection=com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend$4@200d3592
[2024-06-26T06:41:59,932][DEBUG][o.l.SearchOperation      ] [opensearch-nodes-0] execute response=[org.ldaptive.Response@1011953671::result=[org.ldaptive.SearchResult@4303153::entries=[], references=[]], resultCode=SUCCESS, message=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1] for request=[org.ldaptive.SearchRequest@1356983043::baseDn=ou=users,dc=<redacted>,dc=de, searchFilter=[org.ldaptive.SearchFilter@457579389::filter=(cn={0}), parameters={0=opensearchconnector}], returnAttributes=[*, +], searchScope=SUBTREE, timeLimit=PT0S, sizeLimit=0, derefAliases=ALWAYS, typesOnly=false, binaryAttributes=null, sortBehavior=UNORDERED, searchEntryHandlers=null, searchReferenceHandlers=[org.ldaptive.referral.SearchReferralHandler$SearchReferenceHandler@724b8c9b], controls=null, referralHandler=org.ldaptive.referral.SearchReferralHandler@70b2ca94, intermediateResponseHandlers=null] with connection=com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend$4@200d3592
[2024-06-26T06:41:59,932][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-nodes-0] Results for LDAP search for opensearchconnector in base _legacyConfig:
[2024-06-26T06:41:59,932][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-nodes-0] No user opensearchconnector found
[2024-06-26T06:41:59,932][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-nodes-0] Closed a connection, total count is now 0
[2024-06-26T06:41:59,932][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-nodes-0] Unable to authenticate user due to 
org.opensearch.OpenSearchSecurityException: No user opensearchconnector found
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:111) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:589) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:579) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:4019) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933) [guava-32.1.3-jre.jar:?]
	at org.opensearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:579) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:331) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:309) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	...
[2024-06-26T06:41:59,934][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Check authdomain for rest ldap/1 or 2 in total
[2024-06-26T06:41:59,935][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-nodes-0] Connect timeout: PT5S/ResponseTimeout: PT0S
[2024-06-26T06:41:59,935][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-nodes-0] bindDn cn=readonly,dc=<redacted>,dc=de, password ****
[2024-06-26T06:41:59,935][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-nodes-0] Will perform simple bind with bind dn
[2024-06-26T06:41:59,934][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Can not authenticate opensearchconnector due to exception
com.google.common.util.concurrent.UncheckedExecutionException: OpenSearchSecurityException[OpenSearchSecurityException[No user opensearchconnector found]]; nested: OpenSearchSecurityException[No user opensearchconnector found];
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2087) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:4019) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933) ~[guava-32.1.3-jre.jar:?]
	at org.opensearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:579) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:331) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:309) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.ssl.http.netty.Netty4HttpRequestHeaderVerifier.channelRead0(Netty4HttpRequestHeaderVerifier.java:91) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.ssl.http.netty.Netty4HttpRequestHeaderVerifier.channelRead0(Netty4HttpRequestHeaderVerifier.java:38) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
	...
Caused by: org.opensearch.OpenSearchSecurityException: OpenSearchSecurityException[No user opensearchconnector found]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:148) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:589) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:579) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081) ~[guava-32.1.3-jre.jar:?]
	... 47 more
Caused by: org.opensearch.OpenSearchSecurityException: No user opensearchconnector found
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:111) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:589) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:579) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081) ~[guava-32.1.3-jre.jar:?]
	... 47 more
[2024-06-26T06:41:59,935][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Cannot authenticate rest user opensearchconnector (or add roles) with authdomain ldap/1 of [AuthDomain [backend=com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend@13ddbde4, httpAuthenticator=org.opensearch.security.http.HTTPBasicAuthenticator@2e90ee51, order=1, challenge=true], AuthDomain [backend=org.opensearch.security.auth.internal.InternalAuthenticationBackend@7d9587b3, httpAuthenticator=org.opensearch.security.http.HTTPBasicAuthenticator@1e7e201e, order=4, challenge=true]], try next
[2024-06-26T06:41:59,935][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Check authdomain for rest internal/4 or 2 in total
[2024-06-26T06:41:59,937][DEBUG][o.l.BindOperation        ] [opensearch-nodes-0] execute request=[org.ldaptive.BindRequest@1794154177::bindDn=cn=readonly,dc=<redacted>,dc=de, saslConfig=null, controls=null, referralHandler=null, intermediateResponseHandlers=null] with connection=[org.ldaptive.DefaultConnectionFactory$DefaultConnection@1073711515::config=[org.ldaptive.ConnectionConfig@1158474574::ldapUrl=ldap://ldap.<redacted>.de:10389, connectTimeout=PT5S, responseTimeout=PT0S, sslConfig=null, useSSL=false, useStartTLS=false, connectionInitializer=[org.ldaptive.BindConnectionInitializer@367246557::bindDn=cn=readonly,dc=<redacted>,dc=de, bindSaslConfig=null, bindControls=null], connectionStrategy=org.ldaptive.DefaultConnectionStrategy@7b8b3e36], providerConnectionFactory=[org.ldaptive.provider.jndi.JndiConnectionFactory@947635085::metadata=[ldapUrl=ldap://ldap.<redacted>.de:10389, count=1], environment={com.sun.jndi.ldap.connect.timeout=5000, java.naming.ldap.version=3, java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory, com.sun.jndi.ldap.read.timeout=0}, classLoader=null, providerConfig=[org.ldaptive.provider.jndi.JndiProviderConfig@445585751::operationExceptionResultCodes=[PROTOCOL_ERROR, SERVER_DOWN], properties={}, controlProcessor=org.ldaptive.provider.ControlProcessor@12024b8f, environment=null, tracePackets=null, removeDnUrls=true, searchIgnoreResultCodes=[TIME_LIMIT_EXCEEDED, SIZE_LIMIT_EXCEEDED, PARTIAL_RESULTS], classLoader=null, sslSocketFactory=null, hostnameVerifier=null]], providerConnection=org.ldaptive.provider.jndi.JndiConnection@30daad77]
[2024-06-26T06:41:59,939][DEBUG][o.l.BindOperation        ] [opensearch-nodes-0] execute response=[org.ldaptive.Response@1457219402::result=null, resultCode=SUCCESS, message=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1] for request=[org.ldaptive.BindRequest@1794154177::bindDn=cn=readonly,dc=<redacted>,dc=de, saslConfig=null, controls=null, referralHandler=null, intermediateResponseHandlers=null] with connection=[org.ldaptive.DefaultConnectionFactory$DefaultConnection@1073711515::config=[org.ldaptive.ConnectionConfig@1158474574::ldapUrl=ldap://ldap.<redacted>.de:10389, connectTimeout=PT5S, responseTimeout=PT0S, sslConfig=null, useSSL=false, useStartTLS=false, connectionInitializer=[org.ldaptive.BindConnectionInitializer@367246557::bindDn=cn=readonly,dc=<redacted>,dc=de, bindSaslConfig=null, bindControls=null], connectionStrategy=org.ldaptive.DefaultConnectionStrategy@7b8b3e36], providerConnectionFactory=[org.ldaptive.provider.jndi.JndiConnectionFactory@947635085::metadata=[ldapUrl=ldap://ldap.<redacted>.de:10389, count=1], environment={com.sun.jndi.ldap.connect.timeout=5000, java.naming.ldap.version=3, java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory, com.sun.jndi.ldap.read.timeout=0}, classLoader=null, providerConfig=[org.ldaptive.provider.jndi.JndiProviderConfig@445585751::operationExceptionResultCodes=[PROTOCOL_ERROR, SERVER_DOWN], properties={}, controlProcessor=org.ldaptive.provider.ControlProcessor@12024b8f, environment=null, tracePackets=null, removeDnUrls=true, searchIgnoreResultCodes=[TIME_LIMIT_EXCEEDED, SIZE_LIMIT_EXCEEDED, PARTIAL_RESULTS], classLoader=null, sslSocketFactory=null, hostnameVerifier=null]], providerConnection=org.ldaptive.provider.jndi.JndiConnection@30daad77]
[2024-06-26T06:41:59,939][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-nodes-0] Opened a connection, total count is now 1
[2024-06-26T06:41:59,939][DEBUG][o.l.SearchOperation      ] [opensearch-nodes-0] execute request=[org.ldaptive.SearchRequest@-350849793::baseDn=ou=users,dc=<redacted>,dc=de, searchFilter=[org.ldaptive.SearchFilter@457579389::filter=(cn={0}), parameters={0=opensearchconnector}], returnAttributes=[*, +], searchScope=SUBTREE, timeLimit=PT0S, sizeLimit=0, derefAliases=ALWAYS, typesOnly=false, binaryAttributes=null, sortBehavior=UNORDERED, searchEntryHandlers=null, searchReferenceHandlers=[org.ldaptive.referral.SearchReferralHandler$SearchReferenceHandler@34fc8967], controls=null, referralHandler=org.ldaptive.referral.SearchReferralHandler@3422e244, intermediateResponseHandlers=null] with connection=com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend$4@517650a2
[2024-06-26T06:41:59,941][DEBUG][o.l.SearchOperation      ] [opensearch-nodes-0] execute response=[org.ldaptive.Response@1545243889::result=[org.ldaptive.SearchResult@4303153::entries=[], references=[]], resultCode=SUCCESS, message=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1] for request=[org.ldaptive.SearchRequest@-350849793::baseDn=ou=users,dc=<redacted>,dc=de, searchFilter=[org.ldaptive.SearchFilter@457579389::filter=(cn={0}), parameters={0=opensearchconnector}], returnAttributes=[*, +], searchScope=SUBTREE, timeLimit=PT0S, sizeLimit=0, derefAliases=ALWAYS, typesOnly=false, binaryAttributes=null, sortBehavior=UNORDERED, searchEntryHandlers=null, searchReferenceHandlers=[org.ldaptive.referral.SearchReferralHandler$SearchReferenceHandler@34fc8967], controls=null, referralHandler=org.ldaptive.referral.SearchReferralHandler@3422e244, intermediateResponseHandlers=null] with connection=com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend$4@517650a2
[2024-06-26T06:41:59,941][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-nodes-0] Results for LDAP search for opensearchconnector in base _legacyConfig:
[2024-06-26T06:41:59,941][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-nodes-0] No user opensearchconnector found
[2024-06-26T06:41:59,941][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [opensearch-nodes-0] Closed a connection, total count is now 0
[2024-06-26T06:41:59,941][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [opensearch-nodes-0] Unable to authenticate user due to 
org.opensearch.OpenSearchSecurityException: No user opensearchconnector found
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:111) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:589) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:579) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:4019) [guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933) [guava-32.1.3-jre.jar:?]
	at org.opensearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:579) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:331) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:309) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.ssl.http.netty.Netty4HttpRequestHeaderVerifier.channelRead0(Netty4HttpRequestHeaderVerifier.java:91) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.ssl.http.netty.Netty4HttpRequestHeaderVerifier.channelRead0(Netty4HttpRequestHeaderVerifier.java:38) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
	...
[2024-06-26T06:41:59,942][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Can not authenticate opensearchconnector due to exception
com.google.common.util.concurrent.UncheckedExecutionException: OpenSearchSecurityException[OpenSearchSecurityException[No user opensearchconnector found]]; nested: OpenSearchSecurityException[No user opensearchconnector found];
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2087) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:4019) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933) ~[guava-32.1.3-jre.jar:?]
	at org.opensearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:579) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:331) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:309) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.ssl.http.netty.Netty4HttpRequestHeaderVerifier.channelRead0(Netty4HttpRequestHeaderVerifier.java:91) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.ssl.http.netty.Netty4HttpRequestHeaderVerifier.channelRead0(Netty4HttpRequestHeaderVerifier.java:38) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
    ...
Caused by: org.opensearch.OpenSearchSecurityException: OpenSearchSecurityException[No user opensearchconnector found]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:148) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:589) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:579) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081) ~[guava-32.1.3-jre.jar:?]
	... 47 more
Caused by: org.opensearch.OpenSearchSecurityException: No user opensearchconnector found
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:111) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:589) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:579) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081) ~[guava-32.1.3-jre.jar:?]
	... 47 more
[2024-06-26T06:41:59,942][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Can not authenticate opensearchconnector due to exception
com.google.common.util.concurrent.UncheckedExecutionException: OpenSearchSecurityException[OpenSearchSecurityException[No user opensearchconnector found]]; nested: OpenSearchSecurityException[No user opensearchconnector found];
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2087) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:4019) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933) ~[guava-32.1.3-jre.jar:?]
	at org.opensearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:579) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:331) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:309) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.ssl.http.netty.Netty4HttpRequestHeaderVerifier.channelRead0(Netty4HttpRequestHeaderVerifier.java:91) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.ssl.http.netty.Netty4HttpRequestHeaderVerifier.channelRead0(Netty4HttpRequestHeaderVerifier.java:38) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
	...
Caused by: org.opensearch.OpenSearchSecurityException: OpenSearchSecurityException[No user opensearchconnector found]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:148) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:589) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:579) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081) ~[guava-32.1.3-jre.jar:?]
	... 47 more
Caused by: org.opensearch.OpenSearchSecurityException: No user opensearchconnector found
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:111) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:589) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:579) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081) ~[guava-32.1.3-jre.jar:?]
	... 47 more
[2024-06-26T06:41:59,943][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Cannot authenticate rest user opensearchconnector (or add roles) with authdomain internal/4 of [AuthDomain [backend=com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend@13ddbde4, httpAuthenticator=org.opensearch.security.http.HTTPBasicAuthenticator@2e90ee51, order=1, challenge=true], AuthDomain [backend=org.opensearch.security.auth.internal.InternalAuthenticationBackend@7d9587b3, httpAuthenticator=org.opensearch.security.http.HTTPBasicAuthenticator@1e7e201e, order=4, challenge=true]], try next
[2024-06-26T06:41:59,943][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Cannot authenticate rest user opensearchconnector (or add roles) with authdomain ldap/1 of [AuthDomain [backend=com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend@13ddbde4, httpAuthenticator=org.opensearch.security.http.HTTPBasicAuthenticator@2e90ee51, order=1, challenge=true], AuthDomain [backend=org.opensearch.security.auth.internal.InternalAuthenticationBackend@7d9587b3, httpAuthenticator=org.opensearch.security.http.HTTPBasicAuthenticator@1e7e201e, order=4, challenge=true]], try next
[2024-06-26T06:41:59,943][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] User still not authenticated after checking 2 auth domains
[2024-06-26T06:41:59,943][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Check authdomain for rest internal/4 or 2 in total
[2024-06-26T06:41:59,943][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Rerequest with class org.opensearch.security.http.HTTPBasicAuthenticator
[2024-06-26T06:41:59,944][DEBUG][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Rerequest class org.opensearch.security.http.HTTPBasicAuthenticator failed
[2024-06-26T06:41:59,944][WARN ][o.o.s.a.BackendRegistry  ] [opensearch-nodes-0] Authentication finally failed for opensearchconnector from 10.96.21.147:60530
...

(I’ve redacted parts of the call stacks to have this text fit into the response…)

As stated initially, the referenced user opensearchconnector is defined in internal_users.yml and roles.yml (role kafka_sink), and should be excluded from LDAP search as per:

...
        authz:
          ldap_roles:
            authorization_backend:
              config:
                skip_users:
                  - ...
                  - opensearchconnector
                  - ...
...
6

@distant.calls, very interesting - at first glance, it looks like it is ignoring the list of users it should skip skip_users:.

Would you mind sharing the output of:

curl --insecure -u <admin_username>:<admin_password> -XGET https://<OS_node>:9200/_plugins/_security/api/securityconfig?pretty

Please make sure any sensitive info is blanked.

Thanks,
mj

7

@Mantas Sure, output looks like this:

{
  "config": {
    "dynamic": {
      "filtered_alias_mode": "warn",
      "disable_rest_auth": false,
      "disable_intertransport_auth": false,
      "respect_request_indices_options": false,
      "kibana": {
        "multitenancy_enabled": true,
        "private_tenant_enabled": true,
        "default_tenant": "",
        "server_username": "kibanaserver",
        "index": ".kibana",
        "sign_in_options": [
          "BASIC"
        ]
      },
      "http": {
        "anonymous_auth_enabled": false,
        "xff": {
          "enabled": false,
          "internalProxies": """10\.\d{1,3}\.\d{1,3}\.\d{1,3}|192\.168\.\d{1,3}\.\d{1,3}|169\.254\.\d{1,3}\.\d{1,3}|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|172\.1[6-9]{1}\.\d{1,3}\.\d{1,3}|172\.2[0-9]{1}\.\d{1,3}\.\d{1,3}|172\.3[0-1]{1}\.\d{1,3}\.\d{1,3}""",
          "remoteIpHeader": "X-Forwarded-For"
        }
      },
      "authc": {
        "ldap": {
          "http_enabled": true,
          "order": 1,
          "http_authenticator": {
            "challenge": true,
            "type": "basic",
            "config": {}
          },
          "authentication_backend": {
            "type": "ldap",
            "config": {
              "enable_ssl": false,
              "enable_start_tls": false,
              "enable_ssl_client_auth": false,
              "verify_hostnames": false,
              "hosts": [
                "<redacted>"
              ],
              "bind_dn": "cn=readonly,dc=<redacted>,dc=de",
              "password": "******",
              "userbase": "ou=users,dc=<redacted>,dc=de",
              "usersearch": "(cn={0})",
              "username_attribute": "cn"
            }
          }
        },
        "basic_internal_auth_domain": {
          "http_enabled": true,
          "order": 4,
          "http_authenticator": {
            "challenge": true,
            "type": "basic",
            "config": {}
          },
          "authentication_backend": {
            "type": "intern",
            "config": {}
          }
        }
      },
      "authz": {
        "ldap_roles": {
          "http_enabled": true,
          "authorization_backend": {
            "type": "ldap",
            "config": {
              "enable_ssl": false,
              "enable_start_tls": false,
              "enable_ssl_client_auth": false,
              "verify_hostnames": false,
              "hosts": [
                "<redacted>"
              ],
              "bind_dn": "cn=readonly,dc=<redacted>,dc=de",
              "password": "******",
              "userbase": "ou=users,dc=<redacted>,dc=de",
              "usersearch": "(cn={0})",
              "username_attribute": "cn",
              "skip_users": [
                "admin",
                "<redacted>",
                "opensearchconnector",
                "dashboard-ro"
              ],
              "rolebase": "ou=groups,dc=<redacted>,dc=de",
              "rolesearch": "(uniqueMember={0})",
              "userrolename": "disabled",
              "rolename": "cn",
              "resolve_nested_roles": false
            }
          },
          "description": "Authorize using LDAP"
        }
      },
      "auth_failure_listeners": {},
      "do_not_fail_on_forbidden": false,
      "multi_rolespan_enabled": true,
      "hosts_resolver_mode": "ip-only",
      "do_not_fail_on_forbidden_empty": false,
      "on_behalf_of": {
        "enabled": false
      }
    }
  }
}
8

@distant.calls, this might be the cause as this will retry the authentication, and as the LDAP has "order": 1 this will eventually fail.

Try setting it to false, generally, you want to challenge only the last in order (in your case "basic_internal_auth_domain").

    "authc": {
        "ldap": {
          "http_enabled": true,
          "order": 1,
          "http_authenticator": {
            "challenge": false,

best,
mj

9

@Mantas Thanks for the hint. I’ve reconfigured LDAP authentication and am closely monitoring connector behaviour now.

1 Like
10

@Mantas The issue has just re-appeared…

I’ll try changing the auth domains now, setting LDAP last with "challenge": true because users / roles for ‘headless’ clients are stored in the internal auth domain. Although I’m not quite sure this will help.

To my understanding, "challenge": true is meant for browsers (or any clients) that can present any form of authentication prompt in case there is no auth header in the request, is that correct?

11

You can see here more: HTTP basic authentication - OpenSearch Documentation

If my understanding is correct, the "challenge": true will trigger the authc to retire. To avoid the failures you can also create opensearchconnector in your LDAP this will authenticate your user with LDAP but the authorization will be still done internally to OpenSearch because:

Hope that makes sense.
Best,
mj