Use Kafka as a Source with KeyTab authentication

Data Prepper
1

Describe the issue:
We are trying to use Data Prepper with Kafka as a source and OpenSearch as a Sink. But the Kafka cluster authenticates only using KeyTab. In the documentation, it says SASL but it supports only username and password. But we have KeyTab and Krb5.conf file.

How can we authenticate using KeyTab instead of Username and password?

This is the container docker run command I am using:

docker run -e JAVA_OPTS="-Djava.security.auth.login.config=/usr/share/data-prepper/jaas.conf -Djava.security.krb5.conf=/usr/share/data-prepper/krb5.conf 
 -Djava.ssl.truststore=/usr/share/data-prepper/Services.jks -Djava.ssl.truststoreType=jks -Djava.ssl.keystore=/usr/share/data-prepper/Services.jks 
 -Djava.ssl.keyStoreType=jks -Djava.ssl.keyStorePassword=passwordtemplate -Djava.ssl.trustStorePassword=passwordtemplate" \
--name data-prepper \
-v /OpenSearch/krb5.conf:/usr/share/data-prepper/krb5.conf \
-v //OpenSearch/TestPipeline-Kafka.yml:/usr/share/data-prepper/pipelines/pipelines.yml \
-v /OpenSearch/CORPDEV_svc_vd_streams.keytab:/usr/share/data-prepper/CORPDEV_svc_vd_streams.keytab \
-v /OpenSearch/jaas.conf:/usr/share/data-prepper/jaas.conf \
-v /OpenSearch:/usr/share/data-prepper/Services.jks \
opensearchproject/data-prepper:latest

I am trying to pass the KeyTab file and the .jks file in the JAVA_OPTS in the env params.

The error that I am facing:

Authentication error while doing poll(). Will retry after 10 seconds
2024-07-18 10:17:12 org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
2024-07-18 10:17:12 Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Not sure which CERTS I am missing. The path of all the files are correct and verified. They are all getting mounted to the Docker container as well.

Configuration:
Pipeline Config:

prepper-pipeline:
  source: 
    kafka:
      bootstrap_servers: 
        - sl73kfkdbq001.testinternal.com:9093
        - sl73kfkdbq002.testinternal.com:9093 
        - sl73kfkdbq003.testinternal.com:9093
      topics: 
        - name: topic_name
          group_id: data-prepper-1
      authentication:
          sasl:
  processor:
    - parse_json:
    - delete_entries:
        with_keys: ["message"]
  sink: 
    - opensearch:
        hosts: ["https://opensearch-np.testinternal.com:9243"]
        username: username
        password: password
        index: vd-payouts-transactions-${/version}-${/transactionType}-{/timestamp}
        document_id: ${/transactionId}
        insecure: false
        max_retries: 2
2

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.