Invalid field: [ism_template] found in Policy

Plugins Index Management
1

Attempting to follow resource Index State Management in Amazon Elasticsearch Service - Amazon Elasticsearch Service to add an ISM to transition my data from Hot to UltraWarm.

Policy can be created but struggling to add this policy to an index.

  • Attempt one was to create a template
_template/hot_warm_delete
{
  "index_patterns": [
    "log-collector*",
    "aws-log-collector*"
  ],
  "template": {
    "settings": {
      "opendistro.index_state_management.policy_id": "hot_cold_workflow"
    }
  }
}

This did not add any indices to be managed by the policy.

  • Attempt two was to create an ism_template as per policy below
{
    "policy": {
        "policy_id": "hot_cold_workflow",
        "description": "Demonstrate a hot-warm-delete workflow.",
        "last_updated_time": 1620283047106,
        "schema_version": 1,
        "error_notification": null,
        "default_state": "hot",
        "ism_template": {
            "index_patterns": [
                "log-collector*",
                "aws-log-collector*"
            ]
        },
        "states": [
            {
                "name": "hot",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "warm",
                        "conditions": {
                            "min_index_age": "10d"
                        }
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "timeout": "24h",
                        "retry": {
                            "count": 5,
                            "backoff": "exponential",
                            "delay": "1h"
                        },
                        "warm_migration": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "20d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ]
    }
}

which throws error:

[illegal_argument_exception] Invalid field: [ism_template] found in Policy.

This is being attempted on an AWS Managed ES Cluster in Ireland.
Service software release - R20210331
Elasticsearch version - 7.10

Please advise as I cannot currently add indices automatically which is harming the cluster stability

2

Hey @thameezbo,

The ism_template support is currently being rolled out and you should start seeing support of it on the service in the next 1-2 weeks. In the mean time in your index template example it seems like you have an extra “template” key wrapped around the settings?

3

I think AWS ES uses ODFE prior to 1.13.0 so you need to use policy_id instead of ism_template as per Index State Management - Open Distro Documentation

extract: “For ODFE versions prior to 1.13.0, include the policy_id in an index template so when an index is created that matches the index template pattern…”

4

Ah great catch.
Note that the template block was copied from the AWS Docs
Index State Management in Amazon Elasticsearch Service - Amazon Elasticsearch Service

image
image1203Ă—320 11.3 KB

I don’t suppose this is the place to submit a PR for their docs?

After removing the extra, template, polices are being added as expected.
Thank you

5

Copy copy, thank you.

Do you perhaps know how I can check the version of ODFE running?

6

Have submitted a feedback request to AWS to update.

Thanks again for the super swift response

7

Unfortunately not. No matter how hard I tried I couldn’t find anywhere a mapping between AWS ES releases and ODFE versions. The best I could do is to cross-check the features available here Document history for Amazon OpenSearch Service - Amazon OpenSearch Service against the opendistro releases features.

8

Pretty much the same conclusion I came to.

Thank you again for the assistance

1 Like
9

Fixed the AWS docs, sorry for the confusion.

10

As an update, AWS ES Service Software R20210426 enables ism_templates to be used in the managed policy.

Does not mention it anywhere in the release notes lol, but it works.

11

I added it to the release notes: Document history for Amazon OpenSearch Service - Amazon OpenSearch Service