ILM explaination

Plugins Index Management
1

Hello,
I have been having difficulties to explain how OpenDistro ILM works to my peers, I came out with this small document but I would like to have feedback from who knows more than me on this topic (I have been working with ILM only for 2/3 weeks trying to fix an uncontrolled behavior that was bringing down a big cluster on it’s knees).

SM is the OpenDistro implementation of the ILM found in the proprietary x-pack from Elasticsearch B.V. , released with proprietary license and available only for the ElasticSearch distribution from the previously mentioned company.

ISM is different from ILM, it does not have concept of global state and it is applied to a single index at a time. It can be applied to multiple indices but state is always attached to the managed index .

ISM is built on top of a State Machine logic. Each step has actions and transitions . Each action or transition has one or more conditions .

When an action has a matching condition it will perform the said action, the machine will continue to check for actions and their conditions of the current state.

An index can change state when is moved to a new state using transitions , the first matching condition of a transition triggers the move .

Key points:

  • State is executed once every 20~40 minutes, it’s an asyncronous operation
  • Actions are always checked until they fail or are completely fulfilled
  • some actions can be performed multiple times if the condition apply
  • Without a transition, any remaining action will still apply

Example

Given an ISM with 4 states: current , repacked , pre-storage and storage .

  • current state
    • actions
      • rollover on condition: index_size >= 1tb
    • transitions
      • repacked on condition: index_size >= 1tb
  • repacked
    • actions
      • forcemerge on condition: none (always applied)
    • transitions
      • pre-storage on condition: none (always applied)
  • pre-storage
    • actions
    • transitions
      • storage on condition: index_age >= 10d
  • storage
    • actions
      • ultrawarm on condition: none (always applied)
      • delete on condition: index_age >= 30d
    • transitions

When an index is first created it will stay in the current state until the action rollover is performed and the index is transitioned to repacked .

During this time, between the first creation and it’s transition, the index is checked for actions first and transitions second. It will perform the action and then move the index when the condition apply (when the indexsize is greather or equal to 1TB)

An index repacked has an action forcemerge applied the first time the ISM state machine is executed because it’s condition is always true (or empty), the affected index then will then transition to the pre-storage state.

Any index with state pre-storage will wait until the condition for a transition is met: storage , index_age >= 10d (index older than 10 days).

On transition storage the action ultrawarm will be executed moving the index to the ultrawarm storage tier, it will then try to execute the delete action when it’s condition becomes true.

1 Like
2

Hi. Can you explain in more detail about rollover. This is the topic of what I expect. ISM Policy rollover