# Opensearch\_api source doesn't work with kafka buffer

**URL:** <https://forum.opensearch.org/t/opensearch-api-source-doesnt-work-with-kafka-buffer/28059>\
**Category:** Data Prepper\
**Tags:** troubleshoot\
**Created:** [May 12, 2026, 3:39am UTC](https://forum.opensearch.org/t/opensearch-api-source-doesnt-work-with-kafka-buffer/28059 "2026-05-12T03:39:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dpsmails](https://avatars.discourse-cdn.com/v4/letter/d/ba8739/32.png) [@dpsmails](https://forum.opensearch.org/u/dpsmails)\
**Post date:** [May 12, 2026, 3:39am UTC](https://forum.opensearch.org/t/opensearch-api-source-doesnt-work-with-kafka-buffer/28059/1 "2026-05-12T03:39:53Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser): Latest

**Describe the issue** :

1. opensearch\_api is not explained in the document, although it was mentioned in some 2 years old blog as an available feature.
2. traffic doesn’t flow when we use kafka buffer with opensearch\_api. it works fine with bounded blocking.

**Configuration** : (see snippet below, refer to screenshot as well for formatted yaml)

`version: “2”`  
`dual-write-pipeline:`  
`source:`  
`opensearch_api:`  
`port: 9200`  
`buffer:`  
`kafka:`  
`bootstrap_servers: [“kafka:9092”]`  
`authentication:`  
`sasl:`  
`plaintext:`  
`username: “kafka-user”`  
`password: “kafka-pass”`  
`encryption:`  
`type: ssl`  
`topics:`  
`- name: “opensearch-writes”`  
`group_id: “dp-dual-write”`  
`create_topic: true`  
`sink:`  
`- opensearch:`  
`hosts: [“``https://cluster-a-direct:9200``”]`  
`username: “admin”`  
`password: “admin”`  
`index: “${getMetadata("opensearch_index")}”`  
`action: “${getMetadata("opensearch_action")}”`  
`document_id: “${getMetadata("opensearch_id")}”`  
`routing: “${getMetadata("opensearch_routing")}”`  
`- opensearch:`  
`hosts: [“``https://cluster-b:9200``”]`  
`username: “admin”`  
`password: “admin”`  
`index: “${getMetadata("opensearch_index")}”`  
`action: “${getMetadata("opensearch_action")}”`  
`document_id: “${getMetadata("opensearch_id")}”`  
`document_version_type: “external”`  
`document_version: “${getMetadata("opensearch_document_version")}”`  
`routing: "${getMetadata("opensearch_routing")}`

 ![Screenshot_1](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/309b1943649c18caa85ee164b544c1548d4cf5f9.jpeg)

**Relevant Logs or Screenshots** :

---

<div class="post-metadata">

**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\
**Post date:** [May 12, 2026, 12:45pm UTC](https://forum.opensearch.org/t/opensearch-api-source-doesnt-work-with-kafka-buffer/28059/2 "2026-05-12T12:45:01Z")

</div>

@dpsmails Welcome to the forum!

Based on my understanding with **kafka buffer** , `isByteBuffer() = true` tells the source to hand off raw bytes and let the buffer’s consumer decode them on the other side. This round-trip only works if the source’s decoder (`JsonDecoder`) can reconstruct the original events from those bytes, and it can’t, because `JsonDecoder` expects `{"records":[...]}` while `opensearch_api` produces NDJSON bulk format.

The bounded blocking buffer never needs a decoder at all, it stores `Record<Event>` objects directly in a Java `BlockingQueue`, so nothing is ever serialized or re-parsed.

This would seem like a good candidate for a feature request. Which can be submitted [here](https://github.com/opensearch-project/data-prepper/issues).

---

<div class="post-metadata">

**Author:** ![dpsmails](https://avatars.discourse-cdn.com/v4/letter/d/ba8739/32.png) [@dpsmails](https://forum.opensearch.org/u/dpsmails)\
**Post date:** [May 21, 2026, 6:18am UTC](https://forum.opensearch.org/t/opensearch-api-source-doesnt-work-with-kafka-buffer/28059/3 "2026-05-21T06:18:52Z")

</div>

@Anthony I think this is a bug, if a newly added source isn’t working with an existing buffer (more because Kafka buffer being unsupported is not mentioned in any documentation for this source).

So I went ahead and created this issue. I will provide my fix/contribution also for this which i already have: [[BUG] opensearch\_api source does not work with Kafka buffer - no ByteDecoder registered · Issue #6876 · opensearch-project/data-prepper · GitHub](https://github.com/opensearch-project/data-prepper/issues/6876)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [July 20, 2026, 6:18am UTC](https://forum.opensearch.org/t/opensearch-api-source-doesnt-work-with-kafka-buffer/28059/4 "2026-07-20T06:18:59Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.