Describe the issue:
We are using data stream to ingest log data and we are also using ISM policies to rollover the data stream. In this ISM policy once the index is rollover, we would like to reduce shards using ISM shrink policy.
Shrink action creates new index with _shrunken suffix. After successful shrink action, there exists 2 indices with same data
backing index in data stream
other newly created index with _shrunken suffix with lesser shards.
We want delete this existing backing index and redirect search queries to shrunken index. Is there any way to include this shrunken index as a part of data stream ?
UI does not allow to shrink data stream backing indices but we can achieve this via APIs or ISM.
The problem is if we shrink data stream via ISM, it creates new shruken index which is hidden and it is not a part of original data stream. So we cant apply proper retention to this shrunken index and also there exists 2 copies of same index in the cluster.