# Newly created index via shrink ISM action not added to data stream

**URL:** <https://forum.opensearch.org/t/newly-created-index-via-shrink-ism-action-not-added-to-data-stream/14466>\
**Category:** OpenSearch\
**Created:** [May 26, 2023, 4:41am UTC](https://forum.opensearch.org/t/newly-created-index-via-shrink-ism-action-not-added-to-data-stream/14466 "2023-05-26T04:41:08Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Markant](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@Markant](https://forum.opensearch.org/u/Markant)\
**Post date:** [May 26, 2023, 4:41am UTC](https://forum.opensearch.org/t/newly-created-index-via-shrink-ism-action-not-added-to-data-stream/14466/1 "2023-05-26T04:41:08Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):  
2.5

**Describe the issue** :  
We are using data stream to ingest log data and we are also using ISM policies to rollover the data stream. In this ISM policy once the index is rollover, we would like to reduce shards using ISM shrink policy.  
Shrink action creates new index with \_shrunken suffix. After successful shrink action, there exists 2 indices with same data

1. backing index in data stream
2. other newly created index with \_shrunken suffix with lesser shards.

We want delete this existing backing index and redirect search queries to shrunken index. Is there any way to include this shrunken index as a part of data stream ?

**Configuration** :

**Relevant Logs or Screenshots** :

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [May 26, 2023, 8:10am UTC](https://forum.opensearch.org/t/newly-created-index-via-shrink-ism-action-not-added-to-data-stream/14466/2 "2023-05-26T08:10:24Z")

</div>

@Markant Could you share your ISM policy?

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [May 26, 2023, 8:17am UTC](https://forum.opensearch.org/t/newly-created-index-via-shrink-ism-action-not-added-to-data-stream/14466/3 "2023-05-26T08:17:36Z")

</div>

@Markant Can you take a look at this OpenSearch blog?

> **[Index Management UI enhancements](https://opensearch.org/blog/admin-panels-for-index-operations/)**
>
> OpenSearch is a community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/d/d25afef09791915e4e6085b1586e690acda4fd7d.png)

According to that blog, shrink operation doesn’t support data stream indices.

---

<div class="post-metadata">

**Author:** ![Markant](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@Markant](https://forum.opensearch.org/u/Markant)\
**Post date:** [May 26, 2023, 8:18am UTC](https://forum.opensearch.org/t/newly-created-index-via-shrink-ism-action-not-added-to-data-stream/14466/4 "2023-05-26T08:18:37Z")

</div>

test\_data\_stream\_every\_day\_rollover\_policy  
{  
“id”: “test\_data\_stream\_every\_day\_rollover\_policy”,  
“policy”: {  
“policy\_id”: “test\_data\_stream\_every\_day\_rollover\_policy”,  
“description”: “Policy to rollover datastream’s and shrink them on daily basis and delete after 90 days”,  
“last\_updated\_time”: 1684995037220,  
“schema\_version”: 17,  
“error\_notification”: null,  
“default\_state”: “active\_state”,  
“states”: [  
{  
“name”: “active\_state”,  
“actions”: [  
{  
“retry”: {  
“count”: 3,  
“backoff”: “exponential”,  
“delay”: “10m”  
},  
“rollover”: {  
“min\_index\_age”: “15m”  
}  
}  
],  
“transitions”: [  
{  
“state\_name”: “shrink\_state”,  
“conditions”: {  
“min\_index\_age”: “15m”  
}  
}  
]  
},  
{  
“name”: “shrink\_state”,  
“actions”: [  
{  
“retry”: {  
“count”: 3,  
“backoff”: “exponential”,  
“delay”: “10m”  
},  
“shrink”: {  
“num\_new\_shards”: 1,  
“target\_index\_name\_template”: {  
“source”: “{{ctx.index}}\_shrunken”,  
“lang”: “mustache”  
},  
“force\_unsafe”: false  
}  
}  
],  
“transitions”: [  
{  
“state\_name”: “delete\_state”,  
“conditions”: {  
“min\_index\_age”: “30m”  
}  
}  
]  
},  
{  
“name”: “delete\_state”,  
“actions”: [  
{  
“retry”: {  
“count”: 3,  
“backoff”: “exponential”,  
“delay”: “10m”  
},  
“delete”: {}  
}  
],  
“transitions”:   
}  
],  
“ism\_template”: [  
{  
“index\_patterns”: [  
“apim\_data\_stream”  
],  
“priority”: 1  
}  
]  
}  
}

Close

---

<div class="post-metadata">

**Author:** ![Markant](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@Markant](https://forum.opensearch.org/u/Markant)\
**Post date:** [May 26, 2023, 8:39am UTC](https://forum.opensearch.org/t/newly-created-index-via-shrink-ism-action-not-added-to-data-stream/14466/5 "2023-05-26T08:39:06Z")

</div>

UI does not allow to shrink data stream backing indices but we can achieve this via APIs or ISM.

The problem is if we shrink data stream via ISM, it creates new shruken index which is hidden and it is not a part of original data stream. So we cant apply proper retention to this shrunken index and also there exists 2 copies of same index in the cluster.

---

<div class="post-metadata">

**Author:** ![Kruzerson](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/kruzerson/32/5937_2.png) [@Kruzerson](https://forum.opensearch.org/u/Kruzerson)\
**Post date:** [July 31, 2023, 8:35am UTC](https://forum.opensearch.org/t/newly-created-index-via-shrink-ism-action-not-added-to-data-stream/14466/6 "2023-07-31T08:35:47Z")

</div>

Hi @Markant. I faced the same problem. Did you manage to find a solution?