Applied a new template to my indices, but new indices are created with the wrong shard/replica count

OpenSearch
1

Versions ES 7.10 AWS

AWS OpenSearch, running 7.10 ElasticSearch version.

I have my current template as this:

{
  "ism_rollover" : {
    "order" : 100,
    "index_patterns" : [
      "default-logs-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "2",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

It’s the only template I have, it also has the highest possible priority.

My indices are rolled over with the following policy:

{
    "policy_id": "default-logs-policy",
    "description": "Combined Policy for Retention and Rollover",
    "last_updated_time": 1709720050484,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [
                {
                    "rollover": {
                        "min_size": "3gb",
                        "min_index_age": "7d"
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "60d"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": [
        {
            "index_patterns": [
                "default-logs-*"
            ],
            "priority": 100,
            "last_updated_time": 1709720050484
        }
    ]
}

And rollovers work just fine, no issues there. According to my template, new indices are supposed to be started with only 2 shards. However, all of my indices including new ones, look like this:

{
  "default-logs-000017" : {
    "settings" : {
      "index" : {
        "opendistro" : {
          "index_state_management" : {
            "rollover_alias" : "default-logs-current"
          }
        },
        "number_of_shards" : "5",
        "provided_name" : "default-logs-000017",
        "creation_date" : "1718371146144",
        "number_of_replicas" : "1",
        "uuid" : "dR2OCLXpR7q_N8QLAUjq2Q",
        "version" : {
          "created" : "7100299"
        }
      }
    }
  }
}

This is obviously not what I wanted. 5 shards is an overkill for 3gb worth of data, even 2 possibly, but that’s another topic. I do have memory issues so if 2 is a lot as well, please let me know.

I’ve tried recreating the template, double checked its applied and its the only one running. Went through a ton of “solutions” with GPT and none of them worked. I’m out of ideas. I wouldn’t want to nuke everything and start from scratch - maybe the policy is enforcing some long deleted template back when I started it. Any suggestions welcome. Thank you.

2

Could you try creating an index with prefix default-logs- manually and check the number of the shards in that index?