Read incremental logs from opensearch index as input

Data Prepper
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:

Hello, I’m trying to use Data Prepper to read logs from opensearch index (as input) and send to an external server. Just for test, I’m reading from opensearch index and writing to local file instead of external server. The problem (?) is that for each interval, the full content of the index is written into the file, leading to lot of duplicates into the file. I would like to have an “incremental read” from the index to put in the file only new logs which have not been written in the file already. Is there any configuration to make it work?

Configuration:

  source:
    opensearch:
      hosts: ["https://<my-host>:9200"]
      indices:
        include:
          - index_name_regex: "<my-index>_2026-01-16"
      username: "admin"
      password: "<password>"
      connection:
        insecure: true
      scheduling:
        interval: "30s"
      acknowledgments: true

Relevant Logs or Screenshots: