Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.0
Describe the issue:
I am using OpenSearch Pipeline version 2. My data source is an OCI Streaming topic, and I have configured the pipeline to sink data into an OpenSearch index. However, I am observing that millions of duplicate records are being ingested into the OpenSearch index.
In my setup, I build multiple OpenSearch dashboards that rely on SQL queries executed via Metricbeat. The data flow is as follows: Metricbeat queries Oracle using SQL, pushes results to Kafka (OCI Streaming), and the OpenSearch pipeline consumes these streams to index the data.
Currently, I have designed the architecture such that each pipeline handles one stream and writes to one corresponding OpenSearch index. I am wondering if this is the right approach or if there is a more efficient design. Specifically, I want to know whether I can have one pipeline that consumes multiple streams and routes the data to multiple OpenSearch indices, instead of maintaining separate pipelines for each stream.
Configuration:
Relevant Logs or Screenshots: