General and simple questions regarding index management polcies

Plugins Index Management
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.18

I have read the docs but still have questions on general behavior of policies with regards to indexes.

It seems that an index can have only one policy. My setup uses data-prepper “raw-pipeline” trace policy which generates indexes in the format: otel-v1-apm-span-######.

That policy has a single action called ‘Rollover’ which has a condition where it rolls the index either when it reaches 50GB or passes age of 24hours (whichever comes first).

How do I delete indexes older than 30 days for example?

Do I update this policy to add new states or do I create a new policy?

I tried adding a new policy but it seems that a new policy only applies to the indexes created from that point on and even then the otel span indexes continue to use the raw-span-policy only. How do I apply the policy to all indexes so that the older indexes are deleted?

I tried added a new policy with first a transition state based on index age and the second being the delete action state but this doesn’t seem to be triggered.

I tried adding older indexes manually to be under a policy but nothing happens. How often are the conditions checked? When a policy is created for an index pattern shouldn’t I see all indexes under this pattern in the ‘Policy managed indexes’ section? What does it mean that a policy is in ‘Running’ state?

Sorry for so many questions but there just isn’t enough documentation on these concepts.

2

Hi @dmossakowski! I’ll do my best with some of these but I might need further help from the community. I agree with you though. I tried answering some of these by clicking around on the UI and viewing the documentation. It seemed insufficient. You should definitely provide some feedback via the tiny form on the documentation page. It’s the only way they can know.

It would seem logical to me that an index can’t be controlled by more than one policy at the same time. If one of those policies transitioned it to a state that had the delete action while some other policy tried to roll it up, the results would be undefined.

Regarding new policies only seeming to apply to indices created from that point on, I’m not entirely sure. This might be by design. I created a policy that didn’t apply to any indices via the ISM template pattern I provided. I went back and edited it so that the template pattern matched a whole bunch of indices, and the list of “Policy managed indexes” didn’t change.

What did work was going to "indexes’ in the Index Management page, selecting an index, and then going to the “Actions” dropdown and clicking “Apply Policy” like so:

image
image3014×1112 401 KB

When I created a new index by hand that matched the ISM index pattern, it was automagically added to the list of policy managed indices, whereas existing ones had to be added manually.

To delete an index older than 30 days you’d have to add another state to your existing policy. Since you already have a transition based on age of 24 hours, you’d want to transition to this new state after the conditional transfer to the rollup/24hour conditional transition.

I’m confused about why manually adding indices to be managed by a policy didn’t work - the only thing I can think of is that perhaps the index in question was currently being worked on by an existing policy. This is what puts a policy into the “running” state.

I hope that helps you. Btw - have you set up any notifications for your policies? If something is failing or taking too long in the ‘running’ state it might give you more helpful information. And of course it always helps to check the logs as well (/usr/share/opensearch/logs/opensearch.log)

3

Oddly enough, when I checked my “Policy managed indexes” screen, the job status said “running” and when I clicked on the policy I got this:

image
image1444×578 34.5 KB

Are you getting a different response when you click on your policy names that are still “Running” ?

4
  1. A single policy can manage multiple actions for an index – rollover, delete, notification.
  2. I have never tried to assign more than one policy to an index. I do not think it is supported. As noted above, it is not necessary.
  3. A policy gets applied to an index in the following way:
  • At the time of index creation if the index is created based on a configured index template.
  • Manually using the API.
  1. There use to be on older version of OpenS where the policy would only work if the creation of the index was later than the policy. Not sure the reason. I came across it like you.
  2. For a policy example, see below. In addition to this, you will need to setup your index templates. See the link to example below.
  3. For the existing indexes you can update, delete policies with the api: ISM API - OpenSearch Documentation
  4. An example for setting up ISM is in the docs: Policies - OpenSearch Documentation
  5. A few things about ISM in general.
  • It runs on scheduled task. I don’t know the internals but it seems to start running a fair amount of time after you start OpenSearch. I get the feeling it has a lower priority overall. It runs but the intervals seem to vary. The default is every 5 min but that does not always happen. As noted in the docs, do not rely on this for precision in your conditions.
  • Once a policy it added it seems to stay in the initial state for a while – really long while.
  • It is a bit complex to setup but I have run it in production and it works.

For rollover:
{
“policy”: {
“description”: " rollover_delete",
“default_state”: “rollover”,
“states”: [
{
“name”: “rollover”,
“actions”: [
{
“rollover”: {
“min_size”: “50GB”,
“min_index_age”: “24h”
}
}
],
“transitions”: [
{
“state_name”: “delete”,
“conditions”: {
“min_index_age”: “30d”
}
}
]
},
{
“name”: “delete”,
“actions”: [
{
“delete”: {}
}
],
“transitions”:
}
],
“ism_template”: {
“index_patterns”: [
“raven_data*”
],
“priority”: 50
}
}
}

5

As explained by @nateynate I’ll just clarify some things according to my own understanding because as you guys both said, I had to experiment to understand because documentation is scarce.

#1: Yes only one policy per index, at least that’s what the GUI allows, maybe possible to circumvent with CLI/scripting but I’d avoid that.

#2: To delete indexes older than 30 days, you will have to add another state to your current policy. I’m not sure why you are using rollover, but if you can use a data stream instead it might make your policy setup easier. For example, in my case my policy with a datastream is simple: 1. Hot, condition wait until 30 days old, transition in Cold, which put read-only, then wait until 1 year old, transition in Delete, which triggers Delete index.

#3: From my understanding, a policy, once applied to an index and usually at the creation of the index, is unique and static to that index. Each index has their own copy of the original policy that was applied, that they will abide to. Even if you modify the original policy, it will not go through all indexes to check if their policy is updated or not, you only modified the original policy. That original policy, which is now modified, will now apply to new indexes that are created after the current original policy modification.

#4: As stated above, even if you apply policies using an index pattern, they will be applied when a new index is created after the policy was modified. Older indexes already have been created and already have a policy assigned, they won’t be updated. The mecanism to apply policies only triggers at index creation, so it won’t trigger for old ones, they already exists.

#5 As noted by nateynate, you can manually force to apply an updated policy to an existing index. You need to go in “Indexes” (not in Policy managed indexes), then you select all existing indexes by using the checkboxes you want the policy to apply to, and you do Action, Apply Policy. Please note that if you have several pages of indexes, you will need to do it at least once per page to be sure you put the updated policy on every index.

#6 In policy managed indexes, there’s a refresh / retry policy button, that can be helpful in some cases, but in all cases it takes time to apply policies, it’s not instantaneous.

Here’s why it is my understanding that every index as an unique policy, look carefully at the text when you try to manually apply a policy in Indexes:

Capture d’écran 2025-02-06 133411
Capture d’écran 2025-02-06 133411530×290 11.6 KB

“A copy will be created and applied to that index”.

Hoping everything is OK, if there’s anything, do not hesitate to ask!

Konnan

2 Likes
6

Thank you for the replies. The root of my confusion is the idea of a state that an index can be in. If we take the specific example of raw-span-policy, it has a single state called ‘current_write_index’ and so I see that the newest index has in fact this state and the rolled over indexes don’t have any state:

image
image1339×191 24.5 KB

This made me think that I can apply a new policy on this pattern that starts with an empty state (is ‘-’ empty/none state?).

In this case yes there would be two policies on a given index pattern but they would work from different initial state and leave the index in another state after finishing therefore it would be possible to have multiple policies that activate based on state of an index.

With further testing I can see that after fresh restart, the ‘otel-v1-apm-span-000001’ index has a state ‘current_write_index’. I have removed the raw-span-policy from this index and then reapplied it. At first the state was empty (‘-’) but after few minutes the state was ‘current_write_index’ and the Info said ‘Successfully initialized policy: raw-span-policy’.

How did that happen? Was the cause of this that the policy was applied and so the index is put on the initial state that is inside the policy?

If that is the case then yes having multiple policies on an index might cause a problem. Which policy would set the intial state. There is a ‘Add policy’ api call which explicitly states the ability to add a policy to an index and cautions against using * as the pattern as a given policy might delete the system .opendistro-security index. It says: “This operation does not change the policy if the index already has one.” so that may suggest that there can be only ONE policy. There is no mention of an error being thrown if you try to add a policy to an index that already has a policy.

Furthermore regarding the raw-span-policy. This policy seems to belong to data-prepper and until it is not started, this policy does not exist in OpenSearch Dashboards. Can I customize this policy in data prepper configuration to include the deletion action?

This issue is helpful: raw-span-policy delete action · Issue #1751 · opensearch-project/data-prepper · GitHub