ISM Policy and Searching restored indices

Open Source Elasticsearch and Kibana
1

Hi there,

I have an ISM policy that takes a snapshot and restores the index using the convert-remote-to-index action. Here is the complete ISM policy

PUT _plugins/_ism/policies/vz-tsgi-hvuv-osdev-30m-ism-policy
{
  "policy": {
    "policy_id": "vz-tsgi-hvuv-hot-warm-s3-14d-mmb-ism-policy",
    "description": "Hot 2d → Warm 1m → S3 Snapshot → Delete at 15m",
    "error_notification": null,
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "index_priority": {
              "priority": 100
            }
          },
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_index_age": "1d",
              "min_primary_shard_size": "50gb",
              "min_doc_count": 2000000,
              "copy_alias": false
            }
          },
          {
            "force_merge": {
              "max_num_segments": 1
            }
          }
        ],
        "transitions": [
          {
            "state_name": "searchable_snapshot",
            "conditions": {
              "min_index_age": "30s"
            }
          }
        ]
      },
      {
        "name": "searchable_snapshot",
        "actions": [
          
          {
            "retry": {
              "count": 2,
              "backoff": "exponential",
              "delay": "1m"
            },
            "replica_count": {
              "number_of_replicas": 0
            }
          },
          {
            "retry": {
              "count": 1,
              "backoff": "exponential",
              "delay": "1m"
            },
            "snapshot": {
              "repository": "osdev-ism",
              "snapshot": "osdev-ism-{{ctx.index}}"
            }
          },
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "convert_index_to_remote": {
              "repository": "osdev-hvuv-ism",
              "snapshot": "osdev-ism-{{ctx.index}}",
              "rename_pattern": "remote-$1"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "30m"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "delete": {}
          }
        ],
        "transitions": []
      }        
    ],
    "ism_template": [
      {
        "index_patterns": [
          "logs-*"
        ],
        "priority": 400
      }
    ]
  }
}

I do see the restored index in UI when I toggle “show hidden indices”. When I try to create an index pattern for all restored indices, it does not allow me to pick the timestamp field. If I try to create an index pattern for each restored index, I see the timestamp field.

2

I have only data streams in my cluster and I am seeing a ton of MISSING aliases in the cluster logs. Example.

Index to privilege matrix:\\n                                                    | indices:admin/aliases/get |\\nremote-.ds-logs-ec2.pm2-pr_k2ov-000001              | MISSING                   |

I am not able to add an alias from DEV Tools, or I see any reference of this being done using an ISM policy.

Thanks in advance for your help.

Thanks

Murali

3

I had to define the alias in the index template and change the flag “copy_alias”: true. This is the solution I came up with.