# Data duplication in Opensearch Dashboard

**URL:** <https://forum.opensearch.org/t/data-duplication-in-opensearch-dashboard/11192>\
**Category:** OpenSearch\
**Created:** [October 12, 2022, 7:36am UTC](https://forum.opensearch.org/t/data-duplication-in-opensearch-dashboard/11192 "2022-10-12T07:36:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lynow](https://avatars.discourse-cdn.com/v4/letter/l/dfb087/32.png) [@Lynow](https://forum.opensearch.org/u/Lynow)\
**Post date:** [October 12, 2022, 7:36am UTC](https://forum.opensearch.org/t/data-duplication-in-opensearch-dashboard/11192/1 "2022-10-12T07:36:23Z")

</div>

Hello,

I have a question about updating data on Opensearch Dashboards.

In fact, I retrieve with a python script data from a CTI tool, which I then display on OD. Every day, I want to update this data, however if I run the script again the data will be added to the old ones. Namely that I filter on the Dashboard over 1 year, so I would have the same data 3 times. For now, I’ve created a second script that deletes data from the corresponding index before new data arrives.

Is there another solution to this?

I have another tool from which I retrieve its data, however it is even more complex: some data correspond to logs and others come from a script which retrieves the status of HIDS agents continuously. In this case, I cannot delete the index because I would delete the logs, however the information on the status will be duplicated progressively… How to do?

---

<div class="post-metadata">

**Author:** ![tony](https://avatars.discourse-cdn.com/v4/letter/t/ee7513/32.png) [@tony](https://forum.opensearch.org/u/tony)\
**Post date:** [October 18, 2022, 2:14pm UTC](https://forum.opensearch.org/t/data-duplication-in-opensearch-dashboard/11192/2 "2022-10-18T14:14:54Z")

</div>

You could use a fingerprint as the doc id if you can generate a unique one from your data.