Changestream-kafka 3.7.0.0 - stream OpenSearch index changes out to Kafka

OpenSearch
1

Hi all,

OpenSearch 3.x has a pull-based ingestion framework and the ingestion-kafka plugin for getting data from Kafka into OpenSearch. I kept wanting the other direction and couldn’t find it, so I wrote it: changestream-kafka publishes every index, update and delete on a matching index to a Kafka topic as a change event.

It’s useful if you want to keep a cache or a downstream datastore in sync, feed a stream processor, drive webhooks off index activity, or replicate into something that speaks Kafka but not OpenSearch.

Repo: GitHub - AdityaTeltia/opensearch-changestream-kafka · GitHub
Release: Release changestream-kafka 3.7.0.0 · AdityaTeltia/opensearch-changestream-kafka · GitHub

changestream-kafka demo

How it works

A cluster-wide coordinator task reads stream definitions from a system index and, on each poll, dispatches work per primary shard. Each shard task reads translog history from its last confirmed checkpoint, publishes the operations, and advances the checkpoint only after the broker has acknowledged them. A retention lease is held per (stream, shard) at the current checkpoint - the same mechanism cross-cluster replication uses - so segment merges can’t discard history the stream hasn’t consumed yet.

Streams are declared over the REST API rather than in opensearch.yml, so adding or removing one never needs a node restart:

PUT _plugins/_changestream_kafka/_start
{"index": "orders-*", "bootstrap_servers": "kafka:9092", "topic": "orders-cdc"}

Index patterns are globs, so orders-* covers indices that don’t exist yet.

Each record is keyed by the document _id, so every change to one document lands on one partition in order. Delivery is at-least-once: consumers may see duplicates after a broker failure or a shard relocation, but not gaps. Deduplicate on (index, shard, seq_no).

Status, honestly

This is alpha. The delivery path is tested end to end against a real broker, but it hasn’t been run at scale in production, and I’d rather say so than have someone find out through their own cluster. The full list is in the README; the ones worth knowing before you try it:

  • A retention lease is not removed when you _stop a stream, so stopped streams keep pinning history until the index is deleted.
  • The full _source is published - there’s no field include/exclude yet, so don’t stream an index whose contents you wouldn’t want on the topic.
  • No dead-letter handling. A permanently failing shard stays FAILED and retries.

Built against OpenSearch 3.7.0. OpenSearch requires plugins to match the exact version they run on, so this build installs on 3.7.0 clusters only. The bundled kafka-clients is kept on the version OpenSearch pins for ingestion-kafka, so both can sit on the same node.

Feedback I’d find most useful

  • Does the at-least-once contract and the (index, shard, seq_no) dedup key fit how you’d actually build a consumer, or do you need exactly-once?
  • Is per-stream source filtering the next thing you’d want, or is per-document topic routing more valuable?
  • Has anyone solved retention lease cleanup on stop in a way that survives a node going away mid-stop? That’s the limitation I’m least happy about.

Issues and PRs welcome. Thanks for reading.

1 Like