ISM Policy and Searching restored indices

Open Source Elasticsearch and Kibana
1

Hi there,

I have an ISM policy that takes a snapshot and restores the index using the convert-remote-to-index action. Here is the complete ISM policy

PUT _plugins/_ism/policies/vz-tsgi-hvuv-osdev-30m-ism-policy
{
  "policy": {
    "policy_id": "vz-tsgi-hvuv-hot-warm-s3-14d-mmb-ism-policy",
    "description": "Hot 2d → Warm 1m → S3 Snapshot → Delete at 15m",
    "error_notification": null,
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "index_priority": {
              "priority": 100
            }
          },
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_index_age": "1d",
              "min_primary_shard_size": "50gb",
              "min_doc_count": 2000000,
              "copy_alias": false
            }
          },
          {
            "force_merge": {
              "max_num_segments": 1
            }
          }
        ],
        "transitions": [
          {
            "state_name": "searchable_snapshot",
            "conditions": {
              "min_index_age": "30s"
            }
          }
        ]
      },
      {
        "name": "searchable_snapshot",
        "actions": [
          
          {
            "retry": {
              "count": 2,
              "backoff": "exponential",
              "delay": "1m"
            },
            "replica_count": {
              "number_of_replicas": 0
            }
          },
          {
            "retry": {
              "count": 1,
              "backoff": "exponential",
              "delay": "1m"
            },
            "snapshot": {
              "repository": "osdev-ism",
              "snapshot": "osdev-ism-{{ctx.index}}"
            }
          },
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "convert_index_to_remote": {
              "repository": "osdev-hvuv-ism",
              "snapshot": "osdev-ism-{{ctx.index}}",
              "rename_pattern": "remote-$1"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "30m"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "delete": {}
          }
        ],
        "transitions": []
      }        
    ],
    "ism_template": [
      {
        "index_patterns": [
          "logs-*"
        ],
        "priority": 400
      }
    ]
  }
}

I do see the restored index in UI when I toggle “show hidden indices”. When I try to create an index pattern for all restored indices, it does not allow me to pick the timestamp field. If I try to create an index pattern for each restored index, I see the timestamp field.

2

I have only data streams in my cluster and I am seeing a ton of MISSING aliases in the cluster logs. Example.

Index to privilege matrix:\\n                                                    | indices:admin/aliases/get |\\nremote-.ds-logs-ec2.pm2-pr_k2ov-000001              | MISSING                   |

I am not able to add an alias from DEV Tools, or I see any reference of this being done using an ISM policy.

Thanks in advance for your help.

Thanks

Murali

3

I had to define the alias in the index template and change the flag “copy_alias”: true. This is the solution I came up with.

4

The alias does not get copied over to remote indices all the time. I tried the following in the convert_to_remote_index and it still does not work as I had hoped for. What am I doing wrong?

{
  "_id": "vz-tsgi-hvuv-osprod-30d-ism-policy",
  "_version": 15,
  "_seq_no": 275861,
  "_primary_term": 2,
  "policy": {
    "policy_id": "osprod-30d-ism-policy",
    "description": "Hot 30d TO  Warm 1m TO S3 Snapshot TO Delete at 15m",
    "last_updated_time": 1782482239051,
    "schema_version": 29,
    "error_notification": {
      "channel": {
        "id": "g0rz0Z4BqgNUcgV82pc1"
      },
      "message_template": {
        "source": """ISM policy execution failed.

==============
Policy: {{ctx.policy_id}}
Index: {{ctx.index}}
State: {{ctx.state.name}}
Action: {{ctx.action.name}}
Reason: {{ctx.info.message}}
==============""",
        "lang": "mustache"
      }
    },
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "index_priority": {
              "priority": 100
            }
          },
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_doc_count": 200000000,
              "min_index_age": "30d",
              "min_primary_shard_size": "50gb",
              "copy_alias": false
            }
          },
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "force_merge": {
              "max_num_segments": 1
            }
          }
        ],
        "transitions": [
          {
            "state_name": "searchable_snapshot",
            "conditions": {
              "min_rollover_age": "1h"
            }
          }
        ]
      },
      {
        "name": "searchable_snapshot",
        "actions": [
          {
            "retry": {
              "count": 2,
              "backoff": "exponential",
              "delay": "1m"
            },
            "replica_count": {
              "number_of_replicas": 0
            }
          },
          {
            "retry": {
              "count": 1,
              "backoff": "exponential",
              "delay": "1m"
            },
            "snapshot": {
              "repository": "nonprod-ism",
              "snapshot": "nonprod-30d-ism-{{ctx.index}}"
            }
          },
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "convert_index_to_remote": {
              "repository": "nonprod-ism",
              "snapshot": "nonprod-30d-ism-{{ctx.index}}",
              "include_aliases": true,
              "ignore_index_settings": "index.hidden",
              "delete_original_index": true,
              "rename_pattern": "remote_$1"
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": [
      {
        "index_patterns": [
          "logs-*"
        ],
        "priority": 400,
        "last_updated_time": 1782482239051
      }
    ]
  }
}
5

@muraliv I tested this locally and not able to reproduce the issue.

Can you confirm that alias is defined in the index template before the data stream is created.

Additional point to make: template.aliases defined alongside data_stream: {} is applied as a classical index-level alias on each backing index (not at the data-stream level as you might expect). So the same approach works for data stream backing indices. You cannot add classical aliases to a backing index manually via the _aliases API after the fact, OpenSearch rejects that with a 400 error, therefore the template is the only route.