LDAP Connection Not Working

vikramaddagulla · February 14, 2021, 4:37am

Hello,
I am trying to set up LDAP for authc
I have configured the config.yml file

I ran the securityadmin.sh tool and it was success…

However, when I try to login to kibana, it does not work…I see the below errors in the elastic logs :

[2021-02-13T23:35:32,598][WARN ][c.a.d.a.l.b.LDAPAuthorizationBackend] [*****] Unable to connect to ldapserver **********************:636 due to ElasticsearchException[Empty file path for opendistro_security.ssl.transport.truststore_filepath]. Try next.
[2021-02-13T23:35:32,599][WARN ][c.a.o.s.a.BackendRegistry] [usncx441] Authentication finally failed for ******* from ***************

What am I missing out here ?

vikramaddagulla · February 14, 2021, 5:21am

It is working if I change SSL to false…

oscark · February 15, 2021, 8:22am

Your variable opendistro_security.ssl.transport.truststore_filepath in elasticserach.yml seems to be wrong .

pablo · February 17, 2021, 10:34am

@vikramaddagulla

You can also configure “pemtrustedcas_filepath: <DC_root_CA_cert>” with DC root CA cert in config.yml.

anonsens · April 7, 2021, 6:37am

@pablo is the pemtrusted_content configuration supposed to work if I add it to config.yml (i.e. not elasticsearch.yml?

config:
  pemtrustedcas_content: |-
    MIID/jCCAuagAwIBAgIBATANBgkqhkiG9w0BAQUFADCBjzETMBEGCgmSJomT8ixk
    ARkWA2NvbTEXMBUGCgmSJomT8ixkARkWB2V4YW1wbGUxGTAXBgNVBAoMEEV4YW1w
    bGUgQ29tIEluYy4xITAfBgNVBAsMGEV4YW1wbGUgQ29tIEluYy4gUm9vdCBDQTEh
    ...

anonsens · April 7, 2021, 7:26am

When I added the CA certificate to my config.yml like this:

_meta:
  type: "config"
  config_version: 2

config:
  pemtrustedcas_content: |-
    MIIHMUaudjdgKASSGHdgnf+C8Dr16AAAAAACzDANBgkqhkiG9w0BAQUF
    ADBiMRIwEAYKCZImiZPyLGQBGRYCc2UxFTATBgoJkiaJk/IsZAEZFgV0ZWxpYTEU
    MBIGCgmSJo....

  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false

    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      ldap:
        description: "Authenticate via LDAP"
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: true

            # enable start tls, enable_ssl should be false
            enable_start_tls: false

            # send client certificate
            enable_ssl_client_auth: false

            # verify ldap hostname
            verify_hostnames: true

            hosts:
            - XXXXXX:636

            bind_dn: 'USERNAME'
            password: 'PWDPWD'

            userbase: 'ou=Users,dc=ldap,dc=secretservice,dc=com'


            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username

            usersearch: '(&(objectClass=person)(objectClass=user)(sAMAccountName={0}))'

            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: cn

    authz:
      roles_from_ldap:
        description: "Authorize via LDAP"
        http_enabled: true
        transport_enabled: false
        authorization_backend: 

          type: ldap
          config:
            # enable ldaps
            enable_ssl: false

            # enable start tls, enable_ssl should be false
            enable_start_tls: false

            # send client certificate
            enable_ssl_client_auth: false

            # verify ldap hostname
            verify_hostnames: false

            hosts:
            - XXXXXX:389
            bind_dn: 'USERNAME'
            password: 'PWDPWDPWD'

            # Userbase, search and attribute also needs to be in the authz section
            userbase: 'OU=Users,DC=ldap,DC=secretservice,DC=com'

            usersearch: '(&(objectClass=person)(objectClass=user)(sAMAccountName={0}))'
            username_attribute: 'cn'
            rolesearch_enabled: false
            userroleattribute: 'cn'
            userrolename: 'memberOf'
            skip_users:
            - 'kibanaserver'
            - 'admin'

I get the following error in the log when trying to log in:

[2021-04-07T07:09:16,420][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [odfe-opendistro-es-master-0] Connect to XXXXXX:636
[2021-04-07T07:09:16,421][TRACE][c.a.d.a.l.b.LDAPAuthorizationBackend] [odfe-opendistro-es-master-0] Connect to ldaps://XXXXXX:636
[2021-04-07T07:09:16,422][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [odfe-opendistro-es-master-0] verifyHostname true:
[2021-04-07T07:09:16,422][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [odfe-opendistro-es-master-0] trustall false:
[2021-04-07T07:09:16,423][WARN ][c.a.d.a.l.b.LDAPAuthorizationBackend] [odfe-opendistro-es-master-0] Unable to connect to ldapserver XXXXXX:636 due to java.security.cert.CertificateException: No certificate data found. Try next.
[2021-04-07T07:09:16,423][DEBUG][c.a.d.a.l.b.LDAPAuthorizationBackend] [odfe-opendistro-es-master-0] Unable to connect to ldapserver due to
java.security.cert.CertificateException: No certificate data found
	at sun.security.provider.X509Factory.parseX509orPKCS7Cert(X509Factory.java:461) ~[?:?]
	at sun.security.provider.X509Factory.engineGenerateCertificates(X509Factory.java:361) ~[?:?]
	at java.security.cert.CertificateFactory.generateCertificates(CertificateFactory.java:478) ~[?:?]
	at com.amazon.opendistroforelasticsearch.security.support.PemKeyReader.loadCertificatesFromStream(PemKeyReader.java:299) ~[opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.configureSSL(LDAPAuthorizationBackend.java:528) ~[opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:285) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.access$100(LDAPAuthorizationBackend.java:86) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend$2.run(LDAPAuthorizationBackend.java:165) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend$2.run(LDAPAuthorizationBackend.java:155) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at java.security.AccessController.doPrivileged(AccessController.java:554) [?:?]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection(LDAPAuthorizationBackend.java:155) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:87) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry$9.call(BackendRegistry.java:671) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry$9.call(BackendRegistry.java:664) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4875) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3951) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4870) [guava-25.1-jre.jar:?]
	at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:664) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:470) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter.checkAndAuthenticateRequest(OpenDistroSecurityRestFilter.java:177) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter.access$000(OpenDistroSecurityRestFilter.java:66) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter$1.handleRequest(OpenDistroSecurityRestFilter.java:113) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:340) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:191) [elasticsearch-7.10.2.jar:7.10.2]
	at com.amazon.opendistroforelasticsearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:319) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:384) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:309) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-7.10.2.jar:7.10.2]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) [transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-04-07T07:09:16,426][DEBUG][c.a.d.a.l.b.LDAPAuthenticationBackend] [odfe-opendistro-es-master-0] Unable to authenticate user due to
org.ldaptive.LdapException: Unable to connect to any of those ldap servers [XXXXXX:636] due to java.security.cert.CertificateException: No certificate data found
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:364) ~[opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.access$100(LDAPAuthorizationBackend.java:86) ~[opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend$2.run(LDAPAuthorizationBackend.java:165) ~[opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend$2.run(LDAPAuthorizationBackend.java:155) ~[opendistro_security-1.13.0.0.jar:1.13.0.0]
	at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection(LDAPAuthorizationBackend.java:155) ~[opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthenticationBackend.authenticate(LDAPAuthenticationBackend.java:87) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry$9.call(BackendRegistry.java:671) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry$9.call(BackendRegistry.java:664) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4875) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3951) [guava-25.1-jre.jar:?]
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4870) [guava-25.1-jre.jar:?]
	at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:664) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:470) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter.checkAndAuthenticateRequest(OpenDistroSecurityRestFilter.java:177) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter.access$000(OpenDistroSecurityRestFilter.java:66) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter$1.handleRequest(OpenDistroSecurityRestFilter.java:113) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:340) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:191) [elasticsearch-7.10.2.jar:7.10.2]
	at com.amazon.opendistroforelasticsearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opendistro_security-1.13.0.0.jar:1.13.0.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:319) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:384) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:309) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-7.10.2.jar:7.10.2]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) [transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.security.cert.CertificateException: No certificate data found
	at sun.security.provider.X509Factory.parseX509orPKCS7Cert(X509Factory.java:461) ~[?:?]
	at sun.security.provider.X509Factory.engineGenerateCertificates(X509Factory.java:361) ~[?:?]
	at java.security.cert.CertificateFactory.generateCertificates(CertificateFactory.java:478) ~[?:?]
	at com.amazon.opendistroforelasticsearch.security.support.PemKeyReader.loadCertificatesFromStream(PemKeyReader.java:299) ~[opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.configureSSL(LDAPAuthorizationBackend.java:528) ~[opendistro_security-1.13.0.0.jar:1.13.0.0]
	at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:285) ~[opendistro_security-1.13.0.0.jar:1.13.0.0]
	... 83 more
[2021-04-07T07:09:16,428][WARN ][c.a.o.s.a.BackendRegistry] [odfe-opendistro-es-master-0] Authentication finally failed for USERNAME from 172.16.7.204:40926

pablo · April 7, 2021, 9:31am

@anonsens the config part has to be placed in the authentication method. In your case in ldap section as per documentation.

anonsens · April 7, 2021, 10:55am

Thanks @pablo
However I have changed to this in config.yml but I still get the same error message. I don’t think it was clear rom the documentation exactly how to do this.

UPDATE 1: The indentation looks wrong below, but pemtrustedcas_content is on the same level as “enable_ssl” and so on.

UPDATE 2: Should the certificate be serialized to its own file on the storage or should the pem entry just remain in the config.yml?

Is this also wrong?

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      ldap:
        description: "Authenticate via LDAP"
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap
          config:
              pemtrustedcas_content: |-
                MIIHMDCCBRigAwIBAgITHAAAAsxihf+C8Dr16AAAAAACzDANBgkqhkiG9w0BAQUF
                ADBiMRIwEAYKCZImiZPyLGQBGRYCc2UxFTATBgoJkiaJk/IsZAEZFgV0ZWxpYTEU
                MBIGCgmSJomT8ixkARkWBHRjYWQxHzAdBgNVBAMTFldvcmtJVC1Jc3N1aW5nLVRD
                QUQtMDEwHhc....
            # enable ldaps
            enable_ssl: true
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - XXXXXX:636
            bind_dn: 'XXXXXX'
            password: 'XXXXXX'
            userbase: 'ou=XXXXX,ou=XXXXXX,ou=XXXXX,ou=XXXXXX,dc=XXXXXX,dc=XXXXXX,dc=XX'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(&(objectClass=person)(objectClass=user)(sAMAccountName={0}))'
            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: cn
    authz:
      roles_from_ldap:
        description: "Authorize via LDAP"
        http_enabled: true
        transport_enabled: false
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          config:
              pemtrustedcas_content: |-
                MIIHMUTYHSFKFSFVsgITHAAAAsxihf+C8Dr16AAAAAACzDANBgkqhkiG9w0BAQUF
                ADBiMRIwEAYKCZImiZPyLGQBGRYCc2UxFTATBgoJkiaJk/IsZAEZFgV0ZWxpYTEU
                MBIGCgmSJomT8ixkARkWBHRjYWQxHzAdBgNVBAMTFldvcmtJVC1Jc3N1aW5nLVRD
                QUQtMDEw...
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: false
            hosts:
            - XXXXX:389
            bind_dn: 'XXXXX'
            password: 'XXXXXX'

            # Userbase, search and attribute also needs to be in the authz section
            userbase: 'OU=XXXXX,OU=XXXXX,DC=XXXXX,DC=XXXXX,DC=XX'
            usersearch: '(&(objectClass=person)(objectClass=user)(sAMAccountName={0}))'
            username_attribute: 'cn'
            rolesearch_enabled: false
            userroleattribute: 'cn'
            userrolename: 'memberOf'
            skip_users:
            - 'kibanaserver'
            - 'admin'

pablo · April 7, 2021, 11:54am

@anonsens, if you copy ca content to text file does come as one line or you have it broken into multiple?

Update1
You can use either the content or file. I rather to use a file as you don’t have to worry about the content format in the config file.

Update2
I did some testing and it worked for me with pemtrustedcas_filepath. I’ve placed DC cert in the config folder of the elasticsearch and added following line in both authc and authz

pemtrustedcas_filepath: “dc.crt”

pablo · April 7, 2021, 2:03pm

@anonsens, found solution. You need to add Begin Certificate and End certificate in pem content.

        pemtrustedcas_content: |-
          -----BEGIN CERTIFICATE-----
          MIIC4DCCAcigAwIBAgIQc55NSzcyFbRPPgStGrjDQjANBgkqhkiG9w0BAQUFADAZ
          MRcwFQYDVQQDEw5kYy5wYWJsby5sb2NhbDAeFw0xNzExMDQxNzQ0MjJaFw0xODEx
          ........
          -----END CERTIFICATE-----

anonsens · April 7, 2021, 4:23pm

Wow, really nice!
Now it seem to pick up the cert. However, I still get an error.

Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I wonder if it has something to do with me having two certificates in the chain. I added both to the pemtrustedcas_content.

Like this:
-----BEGIN CERTIFICATE-----

          MIIF3DCCA8SgAwIBAgIKHX4WEAAAAAAAEzANBgkqhkiG9w0BAQUFADAYMRYwFAYD

          VQQDEw1Xb3JrSVQgUm9vdENBMB4XDTE1MDIyNjA5MzIzN1oXDTI0MDkxNzEyMzU0

          AgzweWoTkZRz45l200dYWNNui....

          -----END CERTIFICATE-----

          -----BEGIN CERTIFICATE-----

          MIIHMDCCBRigAwIBAgITHAAAAsxihf+C8Dr16AAAAAACzDANBgkqhkiG9w0BAQUF

          ADBiMRIwEAYKCZImiZPyLGQBGRYCc2UxFTATBgoJkiaJk/IsZAEZFgV0ZWxpYTEU

          c3N1aW5nLVRDQUQtMDEsQ049Y....

          -----END CERTIFICATE-----

pablo · April 7, 2021, 6:09pm

@anonsens as per documentation pemtrustedcas_content require only CAs

“The root CA content of your Active Directory/LDAP server.”

In my case, I have DC and CA on the same server. I’ve got cert with the below command.
openssl s_client -connect DC_IP_or_FQDN:636 -showcerts

If you use PEM format, be sure that you’ll keep following format.

-----BEGIN CERTIFICATE-----
Intermediate CAs
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Root CA
-----END CERTIFICATE-----

I’m not sure if that will work here as I only had self-signed Root CA.

anonsens · April 8, 2021, 8:06am

Thanks a lot for the help. Finally got it to work!

After adding the root CA base64 content along with the BEGIN/END strings everything is now working.

Topic		Replies	Views
Error: Empty file path for opendistro_security.ssl.transport.truststore_filepath Security	9	4744	August 16, 2021
Ldap & AD SSL config and truststore Security	2	1314	January 20, 2021
LDAP dont work over SSL Open Source Elasticsearch and Kibana	2	651	July 27, 2021
Cant manage to setup ldap Open Source Elasticsearch and Kibana	3	424	March 13, 2021
Initial configuration LDAP without certificates opendistro-security-1.12.0.0 Security	4	617	May 20, 2021

LDAP Connection Not Working

Related topics