Data duplication in Opensearch Dashboard


I have a question about updating data on Opensearch Dashboards.

In fact, I retrieve with a python script data from a CTI tool, which I then display on OD. Every day, I want to update this data, however if I run the script again the data will be added to the old ones. Namely that I filter on the Dashboard over 1 year, so I would have the same data 3 times. For now, I’ve created a second script that deletes data from the corresponding index before new data arrives.

Is there another solution to this?

I have another tool from which I retrieve its data, however it is even more complex: some data correspond to logs and others come from a script which retrieves the status of HIDS agents continuously. In this case, I cannot delete the index because I would delete the logs, however the information on the status will be duplicated progressively… How to do?

You could use a fingerprint as the doc id if you can generate a unique one from your data.