I’m new to OpenSearch, I was given a task to configure four data streams and also rollover the data streams using ISM policy if the size reaches 50 GB and delete old data to manage storage. please help with the configuration part.
Data comes from fluent bit, with each index reaching around 200 GB daily.
Thanks @Mantas, I have followed the documentation and set the initial ISM to rollover and delete the index. Is it possible to rollover the Index and then delete the data in indexes older than 2 days ?
@Mantas Thanks for the response, the size of the incoming logs is very large, so I can’t delete them based on the days, I want to preserve the latest 50GB and delete all logs older than that, So at every instance, the last 50GB of data will be available.
This looks good, but the 50GB will be reached within 4-8 hours based on the traffic. Can’t store rolledover data for 1 day. also what’s the use of min_index_count ?
Thanks, I will look into it, just help me with one more thing,
My ISM policy on reaching 50GB trigger rollover, but it fails with error “missing alias or not write index”. I have the rollover alias setting in my index template index.plugins.index_state_management.rollover_alias:"prod-data"
prod-data is the alias name of my index and prod-data-000094 is the current index which is also write index
ism policy was not able to change the write index from current index to the new index.