Describe the issue:
We define two index policies with over-lapping index patterns and expected that the most-specific index would be applied to some of our indexes. However, we’re seeing the more generic policy being linked to these indexes. I am certain that this worked when we used Open Distro for Elasticsearch 1.7.0 but it looks like things may have changed after that.
Details:
Here are the ism_template section of the two policies:
So there is actually an issue open to modify the docs for this. When there are two matches priority is used to define which applies. Here you have 100 as the priority for the more generic policy so it is getting prioritized (higher numbers as I understand makes it a greater priority).
Try flipping the priorities and if that doesn’t work then we should submit a bug for this.
You are correct: swapping the priorities resulted in the “right” index policy being assigned to the indexes.
But I want to point out that the documentation explains that the priority value is used to determine the order in which indexes were restored/recovered if there was a problem. That may mean the two uses of priority are conflicting.
In our case, the documents in the viya_logs-openshift-* (i.e. more specific) indexes are less important and we prefer that they are loaded after the other indexes matching the viya_logs-* pattern, so we assigned it the lower priority value. For context: OpenShift uses a large number (40+) of namespaces (all of which start with “openshift”) for internal/overhead infrastructure. Our log monitoring solution is primarily focused on application logs which can be generated by pods in any namespace. We want to keep the application logs for a longer period of time than the overhead/infrastructure logs. In addition, if there is a failure and a need to restore indexes, we want the application logs to be brought up first (i.e. we don’t want our users to have to wait for the huge number of infrastructure logs to be restored). The priority values I showed above achieve that objective. However, if that same priority is now being used to determine which index policy is applied, it doesn’t (since the indexes containing log messages from the overhead namespaces are now assigned the wrong/other index policy). But I suppose someone else might be able to offer a counter-example where the reverse is true.
I guess my point is that re-use of the existing priority setting for a 2nd purpose may not have been the best option here. It might have been better to add a new setting to control which policy is applied when multiple policies match. In our case, we will probably decide getting the right retention period is more important than worrying about the order indexes are restored.
Yeah I completely agree. It seems that the index_priority only applies when priority is a subfield under index_priority. It feels like we could have just used index_priority: rather than index_priority: { priority: }.
It seems like we should create some top level documentation on what priority is and how its handled and then mention that priority may be used as a sub-field that applies to different parts of an ISM policy.
My concern is less about the naming/nesting of the property; it’s more about re-using an existing property with a documented purpose and using it for a completely different purpose. I think it would have been “cleaner” to keep the current property focused on its documented purpose (i.e. controlling order indexes are restored) and creating a new property, perhaps index_template_priority, that controls which index template is assigned when multiple templates match.
Reusing or expanding how an existing property is used risks negative impacts on existing users. If the new use is very close to the original use or represents some logical extension of the original use, such a change is probably fine. I would argue that in this particular case, the new use doesn’t meet that test (the concerns of index pattern assignment and index restoration order are too different).