# Data prepper does not take changes in file source at runtime

**URL:** <https://forum.opensearch.org/t/data-prepper-does-not-take-changes-in-file-source-at-runtime/20516>\
**Category:** Data Prepper\
**Created:** [July 25, 2024, 6:09am UTC](https://forum.opensearch.org/t/data-prepper-does-not-take-changes-in-file-source-at-runtime/20516 "2024-07-25T06:09:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sinchana](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@Sinchana](https://forum.opensearch.org/u/Sinchana)\
**Post date:** [July 25, 2024, 6:09am UTC](https://forum.opensearch.org/t/data-prepper-does-not-take-changes-in-file-source-at-runtime/20516/1 "2024-07-25T06:09:08Z")

</div>

data prepper version: opensearch-data-prepper-jdk-2.8.0-linux-x64

**Describe the issue** : I have a pipeline configured with file source and opensearch sink. Data prepper starts and reads the file source. Later if any changes are made in file say adding new line with some text, data prepper does not take that and pass it to opensearch.

The new change in file is taken only after data prepper is restarted.  
Is this how it is supposed to work?

restarting the data prepper also duplicates the data as it reads the whole file again.

Please let know if the events are taken at runtime or restart of dataprepper is required for every change made to file source.

**Configuration** :  
pipelines.yaml is placed in \<home\_dir\>/pipelines:

sample-pipeline-2:  
delay: “1”  
source:  
file:  
path: “/scratch/Data\_prepper/source.txt”  
record\_type: event  
sink:  
- opensearch:  
hosts: [“[http://host:9200](http://host:9200)”]  
index: “sample-logs-2”  
bulk\_size: 10  
max\_retries: 5  
insecure: true

---

<div class="post-metadata">

**Author:** ![Sinchana](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@Sinchana](https://forum.opensearch.org/u/Sinchana)\
**Post date:** [July 30, 2024, 5:36am UTC](https://forum.opensearch.org/t/data-prepper-does-not-take-changes-in-file-source-at-runtime/20516/2 "2024-07-30T05:36:18Z")

</div>

Hi, any updates on this?

---

<div class="post-metadata">

**Author:** ![kitkit](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kitkit](https://forum.opensearch.org/u/kitkit)\
**Post date:** [August 10, 2024, 10:45am UTC](https://forum.opensearch.org/t/data-prepper-does-not-take-changes-in-file-source-at-runtime/20516/3 "2024-08-10T10:45:45Z")

</div>

Hi

From my experience with data prepper (ingestion pipeline in AWS), the moment the data prepper activated, it will take a snapshot from your source and migrate over.

Any new or updated documents will not migrate over and you need to start again the data prepper for the prepper to snapshot the latest state and migrate

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [October 9, 2024, 10:46am UTC](https://forum.opensearch.org/t/data-prepper-does-not-take-changes-in-file-source-at-runtime/20516/4 "2024-10-09T10:46:39Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.