# Opensearch Cluster

**URL:** <https://forum.opensearch.org/t/opensearch-cluster/24851>\
**Category:** Data Prepper\
**Tags:** troubleshoot\
**Created:** [June 23, 2025, 10:24am UTC](https://forum.opensearch.org/t/opensearch-cluster/24851 "2025-06-23T10:24:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sanjayraja1211](https://avatars.discourse-cdn.com/v4/letter/s/a4c791/32.png) [@Sanjayraja1211](https://forum.opensearch.org/u/Sanjayraja1211)\
**Post date:** [June 23, 2025, 10:24am UTC](https://forum.opensearch.org/t/opensearch-cluster/24851/1 "2025-06-23T10:24:54Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.0

**Describe the issue** :  
I am using **OpenSearch Pipeline version 2**. My data source is an **OCI Streaming topic** , and I have configured the pipeline to sink data into an **OpenSearch index**. However, I am observing that **millions of duplicate records** are being ingested into the OpenSearch index.

In my setup, I build multiple **OpenSearch dashboards** that rely on **SQL queries** executed via **Metricbeat**. The data flow is as follows: **Metricbeat queries Oracle using SQL, pushes results to Kafka (OCI Streaming), and the OpenSearch pipeline consumes these streams to index the data**.

Currently, I have designed the architecture such that **each pipeline handles one stream and writes to one corresponding OpenSearch index**. I am wondering if this is the right approach or if there is a more efficient design. Specifically, I want to know whether I can have **one pipeline that consumes multiple streams and routes the data to multiple OpenSearch indices** , instead of maintaining separate pipelines for each stream.

**Configuration** :

**Relevant Logs or Screenshots** :

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [August 22, 2025, 10:25am UTC](https://forum.opensearch.org/t/opensearch-cluster/24851/2 "2025-08-22T10:25:52Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.