Dashboard tenant indices avoid specific nodes

OpenDistro
1

Our OpenSearch usage is very heavy on percolate queries, and as such we have optimized our cluster to rapidly scale the nodes hosting our percolate indexes with the indexes set to auto-expand replicas. This works very well for an important use case and allows us to have appropriately scalable performance.

However, we need to keep system indexes off of those nodes, because they scale in rapidly, which has caused us to lose tenant indexes and other important internal indexes.

We are currently preventing the indexes we don’t want to go onto the percolate nodes with a very simple component index template:

{
      "name" : "no_perc",
      "component_template" : {
        "template" : {
          "settings" : {
            "index" : {
              "routing" : {
                "allocation" : {
                  "exclude" : {
                    "data_type" : "percolate"
                  }
                }
              }
            }
          }
        }
      }
    }

Then for the .kibana indexes we apply this with a simple index template:

PUT _index_template/avoid_perc_nodes
{
  "index_patterns" : [
          ".ds*", ".open*", ".kibana_*"
        ],
        "template" : {
         
        },
        "composed_of" : ["no_perc" ],
        "priority" : 0
}

However, for the Dashboards generated indexes such as .kibana_-283538891_user_1 we have the appropriate settings applied, but not the index mapping.
Per the documentation, the index templates should be combined, however that does not line up with what I am seeing on my cluster. Our templates are only specifying settings, never any mappings, so the tenant_template (which I can’t view or edit), should be unchanged in that regard, but the mappings are never set in the tenant generated indexes.

Is there a better way to provide settings to system indices to get them to avoid certain nodes? Is there another approach I could take to have a node reject all shards except those matching a specific name pattern? I am looking for any solution to keep those indexes off of some nodes that I know will be joining and leaving the cluster rapidly.

Thank you.

2

Hi, how I understood your questions:
You have two “index types” one that should auto expand to the whole cluster and a second that should just stay on (or be locked to) specific nodes.

I have a similar thing for ssd and hhd-nodes.

"template": {
    "settings" : {
      "number_of_shards" : "1",
      "number_of_replicas" : "2",
      "refresh_interval" : "5s",
      "routing" : {
        "allocation" : {
          "require" : {
            "data" : "ssd"
            }
          }
        }
      },

and then I have I have on in opensearch.yml on each node defined node.attr.data: ssd or node.attr.data: hhd

I would lock the non-expanding indices to require to be a on nodes with a specific node attribute. While the auto-expanding indices would be allowed to be anywhere. So when I would expand my cluster I would make sure that the new nodes does not have the attribute defined in openserach.yml

I have not thought about this in a long time so I might be a bit rusty.

3

Thank you for the response.
I’ve already done that and that is how I manage all of my user-generated indexes on the cluster. However, I need the system indexes to stay off certain nodes because they will disappear rapidly as load changes.

The specific challenge is applying a template such as that to system generated indexes (such as the .kibana* indexes generated by Dashboards), without overwriting the system generated template and also without the ability to query OpenSearch for the generated tenant_template in order to modify it to include the necessary index routing allocation settings. Additionally, I don’t want to have to monitor the cluster for newly generated system indexes to then apply the necessary setting updates. This should be something that can be managed with templates and component templates, but the system generated indexes seem to behave differently than how the documentation indicates when multiple templates match.

4

IDK what I’m talking about, but have you tried setting the template priority? It seems to me that the default “system templates” have default priority of zero. Index templates - OpenSearch documentation

It was news to me that these gives different results:

_cat/templates?v

_index_template/
5

Yes, I tried setting the template priority.

I think I’ll file this as a bug to try to get some dev attention.

The issue I have is that the docs say that the templates are applied in an overlapping fashion, but that isn’t working here, because my template provides no mapping and yet it overwrites the mapping of the tenant_template provided by Dashboards.

From the index template docs:

You can create multiple index templates for your indices. If the index name matches more than one template, OpenSearch merges all mappings and settings from all matching templates and applies them to the index.

Yup, you can’t see or edit the system templates. I don’t do anything with filebeat or logstash but I would assume their templates are in the same category.