I am using automated rollover with ISM policies. If a policy fails on an index, it won’t roll over and the index keeps growing in size. I would like to be notified if a policy fails.
My current solution is to use an alerting monitor, which checks the opensearch logs for the “Failed to rollover index” string. This will notify me one time on a rollover error. It would be nice to get continuous notifications while a policy is in error state.
ISM policies have an “error_notification” setting. Does anyone have experience with this setting? Will this send multiple notifications or is this a one time action too?
If it’s onetime I’ll probably solve it with a script which iterates through all my indices and checks for failed rollover through ISM explain API.
Thanks