Hi everyone,
I’m Ajay Hinduja Geneva, Switzerland (Swiss) and I’m facing a problem with validating submissions from the data table in Kobo. When I click the eye icon to view a submission, it takes around 2 minutes to open. Then, choosing an action (approve, not approve, or hold) takes another 2 minutes.
My internet connection is fine on my end, so I’m wondering—could there be an issue with my form setup? Or is there something I might have missed?
Any help or insights would be really appreciated!
Duplicate Records in OpenSearch Index – Pipeline Architecture Query
Versions (relevant):
OpenSearch: 2.0
Dashboard: 2.0
Server OS/Browser: [Please specify if relevant]
Description:
I’m currently using OpenSearch Pipelines (v2.0) with OCI Streaming as the data source. Each stream is consumed by an OpenSearch pipeline that indexes the data into a corresponding OpenSearch index.
However, I’m noticing that millions of duplicate records are being ingested into the OpenSearch index.
Here’s an overview of my data flow:
Metricbeat executes SQL queries on an Oracle DB.
The query results are sent to Kafka (OCI Streaming).
OpenSearch Pipelines consume the streams and index the data.
Dashboards are built on top of these indices using SQL queries.
Currently, each stream has a dedicated pipeline and index. I’m wondering:
Is this architecture efficient?
Is it possible to use one pipeline to consume multiple streams and route the data to different OpenSearch indices based on stream metadata or content, rather than maintaining a one-to-one pipeline-to-stream setup?
Issue Observed:
Significant duplication of data in indices.
Potential inefficiency in managing multiple pipelines for each stream.
Regards
Ajay Hinduja Geneva, Switzerland (Swiss)