How to use kafka-connect-opensearch with Opensearch

OpenSearch
1

How can I use the kafka-connect-OpenSearch connector to directly send data from Kafka topic to OpenSearch? Earlier I was using FluentD but now the OpenSearch connector is just launched, so I want to leverage that connector to eliminate the need for fluentD(I was using that for just connecting Kafka and OpenSearch).
I followed this page from the official doc.

I did the following things till now,

  1. Configured Kafka and zookeeper using docker
  2. Schema registry to use connectors
    docker run -d --net=host --name=schema-registry
    -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=localhost:2181
    -e SCHEMA_REGISTRY_HOST_NAME=localhost
    -e SCHEMA_REGISTRY_LISTENERS=http://localhost:8081 confluentinc/cp-schema-registry:5.0.0
  3. Created kafka topic to store config, status, and offsets of the connectors in Kafka topics

everything is up and running without errors
now I want to use OpenSearch connector - GitHub - dmathieu/kafka-connect-opensearch: Kafka Connect plugin for sending events to OpenSearch
What will be the next steps to use this particular connector?

2

hey @spzala!

Looks like that connector is pretty new (11 days!) so I’m not sure how working/stable it is. You might want to touch base with the author on the github repo.

1 Like
3

Have you installed the connector plugin in your Kafka Connect worker?

Once you’ve done that you need to create an instance of the connector. If you’re using Kafka Connect in distributed mode (which you generally should) then this is done using a REST call. There’s an example of what the JSON payload might look like in this post.

To get a general understanding of Kafka Connect and how to use it check out this Kafka Connect 101: https://developer.confluent.io/learn-kafka/kafka-connect/

1 Like