How to store and retrieve full alert history for OpenSearch alerting monitors?"

Plugins Alerting
,
1

Describe the issue:

Hi everyone,
I’m working on a system that uses OpenSearch alerting monitors (from the _plugins/_alerting/monitors endpoint) to detect conditions and send alerts via Slack and PagerDuty destinations. Everything works fine in terms of triggering alerts — users get messages as expected.
However, I’m trying to visualize multiple historical alerts on a timeline in my frontend UI. Right now, I’m querying the .opendistro-alerting-alert* indices, but I only see a limited number of recent alerts (10 by default) and not the complete history of every alert that has ever been triggered for a specific monitor (rule).

Configuration:

  1. Does OpenSearch store all historical alert events by default?
  • Or are they overwritten/limited by retention policies?
  1. If not, what’s the best practice to persist every triggered alert?
  • I heard that using a webhook action in the monitor that stores events into a custom index could work — is that the recommended approach?
  1. How can I structure my monitor’s action to store every alert execution (with trigger metadata) into a custom index for long-term reporting?
  • Any examples or best practices would be very helpful.

Relevant Logs or Screenshots:

:wrench: Current Setup:

  • OpenSearch 2.x
  • Using monitors with query-level triggers
  • Alerts go to Slack/PagerDuty
  • Want to build a timeline chart in UI showing all times a trigger was fired (not just the latest alert)

Thanks in advance to anyone who can clarify this! :folded_hands:
Let me know if you need more context or code.

2

@abhikumar18 Could you share your query against .opendistro-alerting-alert*?

3

This is opensearch query right now

const alertQuery = {
    index: OS_ALERTS_INDEX,
    size: 5000,
    _source: [
      "id",
      "monitor_id",
      "monitor_name",
      "trigger_id",
      "trigger_name",
      "state",
      "severity",
      "start_time",
      "end_time",
      "error_message",
      "alert_history",
      "action_execution_results",
    ],
    body: {
      query: {
        bool: {
          filter: [
            {
              term: {
                monitor_id: id,
              },
            },
            {
              range: {
                start_time: {
                  gte: start?.slice(0, -1),
                  lte: end?.slice(0, -1),
                  format: "yyyy-MM-dd'T'HH:mm:ss.SSS",
                },
              },
            },
          ],
        },
      },
      aggs: {
        alerts_over_time: {
          date_histogram: {
            field: "start_time",
            fixed_interval: interval,
            min_doc_count: 1, // Skip empty buckets
            extended_bounds: {
              min: startTimeMillis,
              max: endTimeMillis,
            },
          },
          aggs: {
            severity_info: {
              terms: {
                field: "severity",
                size: 5,
              },
              aggs: {
                trigger_info: {
                  terms: {
                    field: "trigger_id",
                    size: 5,
                  },
                  aggs: {
                    trigger_name: {
                      terms: {
                        field: "trigger_name.keyword",
                        size: 1,
                      },
                    },
                  },
                },
              },
            },
          },
        },
        max_alerts: {
          max_bucket: {
            buckets_path: "alerts_over_time._count",
          },
        },
      },
    },
  }