I have an alert set up to look for one specific message in the past 10 minutes with the trigger condition: ctx.results[0].hits.total > 0
When the elasticsearch cluster is in good health, everything works well and we get alerts. However if the cluster is not responding for whatever reason, the alert will trigger even if the message I am looking for is not actually being returned. This makes me think that the query is timing out and that results in an alert when the condition isn’t actually being matched.
Is it possible to avoid triggering an alert on the condition that no data is returned (i.e. search timeout)?
Hi @gizmonicus,
If there is an error during the execution of the monitor the alert is put into an ERROR state which will cause the notification to be sent out. It’s possible your monitor query is timing out which is causing your alert for that trigger to enter the above ERROR state and automatically notify you.
We do not currently have a way to disable ERROR notifications as we wanted to default to warning the user about potential issues nor do we have a way to allow skipping execution because of a query timeout.
If this is something you’d like to see you can make a feature request on our GitHub issues so we can look into it more.
1 Like
OK, thanks, this is helpful. I do think it would be nice to disable these types of notifications in certain circumstances. In particular, I’m already monitoring cluster health so I don’t need alerts informing me that things dependent on that are broken. It’s not a huge problem though and I would rather tackle the cluster health problem first so I’ll just leave it at that.