Newly created index via shrink ISM action not added to data stream

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.5

Describe the issue:
We are using data stream to ingest log data and we are also using ISM policies to rollover the data stream. In this ISM policy once the index is rollover, we would like to reduce shards using ISM shrink policy.
Shrink action creates new index with _shrunken suffix. After successful shrink action, there exists 2 indices with same data

  1. backing index in data stream
  2. other newly created index with _shrunken suffix with lesser shards.

We want delete this existing backing index and redirect search queries to shrunken index. Is there any way to include this shrunken index as a part of data stream ?

Configuration:

Relevant Logs or Screenshots:

2

@Markant Could you share your ISM policy?

3

@Markant Can you take a look at this OpenSearch blog?

image
image940×240 22.9 KB

According to that blog, shrink operation doesn’t support data stream indices.

4

test_data_stream_every_day_rollover_policy
{
“id”: “test_data_stream_every_day_rollover_policy”,
“policy”: {
“policy_id”: “test_data_stream_every_day_rollover_policy”,
“description”: “Policy to rollover datastream’s and shrink them on daily basis and delete after 90 days”,
“last_updated_time”: 1684995037220,
“schema_version”: 17,
“error_notification”: null,
“default_state”: “active_state”,
“states”: [
{
“name”: “active_state”,
“actions”: [
{
“retry”: {
“count”: 3,
“backoff”: “exponential”,
“delay”: “10m”
},
“rollover”: {
“min_index_age”: “15m”
}
}
],
“transitions”: [
{
“state_name”: “shrink_state”,
“conditions”: {
“min_index_age”: “15m”
}
}
]
},
{
“name”: “shrink_state”,
“actions”: [
{
“retry”: {
“count”: 3,
“backoff”: “exponential”,
“delay”: “10m”
},
“shrink”: {
“num_new_shards”: 1,
“target_index_name_template”: {
“source”: “{{ctx.index}}_shrunken”,
“lang”: “mustache”
},
“force_unsafe”: false
}
}
],
“transitions”: [
{
“state_name”: “delete_state”,
“conditions”: {
“min_index_age”: “30m”
}
}
]
},
{
“name”: “delete_state”,
“actions”: [
{
“retry”: {
“count”: 3,
“backoff”: “exponential”,
“delay”: “10m”
},
“delete”: {}
}
],
“transitions”:
}
],
“ism_template”: [
{
“index_patterns”: [
“apim_data_stream”
],
“priority”: 1
}
]
}
}

Close

5

UI does not allow to shrink data stream backing indices but we can achieve this via APIs or ISM.

The problem is if we shrink data stream via ISM, it creates new shruken index which is hidden and it is not a part of original data stream. So we cant apply proper retention to this shrunken index and also there exists 2 copies of same index in the cluster.

6

Hi @Markant. I faced the same problem. Did you manage to find a solution?