Clarification about security analytics cluster configuration

Plugins Security Analytics
1

Hi everyone,

I am currently tuning the cluster settings for the Security Analytics plugin to better manage the lifecycle and quantity of internal indices created for findings and alerts.

Based on my testing of the finding settings, I have confirmed the following behavior:

  • plugins.security_analytics.alert_finding_enabled: When true, it generates the .opensearch-sap-<detector_type>-findings-<date> indices.

  • plugins.security_analytics.alert_finding_max_docs: Controls the document count threshold for rolling over finding indices.

  • plugins.security_analytics.alert_finding_rollover_period: Sets the frequency for the background job to check rollover conditions.

  • plugins.security_analytics.finding_history_max_age: Defines the retention/timeframe for data within the findings history.

The Issue: While the documentation references plugins.security_analytics.alert_history_max_age and plugins.security_analytics.alert_history_max_docs, I am having difficulty locating the specific configuration syntax or confirming the relationship between them. Specifically:

  1. Dependency Logic: The documentation suggests that if the alert count doesn’t exceed alert_history_max_docs, a new index is created based on the alert_history_max_age period. How do these two settings interact if both thresholds are reached simultaneously?

  2. Configuration Availability: Are these settings specifically for the Alert history indices (separate from Findings)? If so, what is the exact setting path to update them via the _cluster/settings API, as I have found some inconsistencies in the naming conventions between the docs and the actual plugin behavior.

Goal: My objective is to strictly control the number of indices generated for both Findings and Alerts to prevent index explosion in our cluster.

Suggestions for better understanding the **settings related to Findings and their use cases, as well as the settings related to Alerts and their use cases, are welcome. For reference, I will attach the document that I referred to Security Analytics settings - OpenSearch Documentation

2

@Aravinth Thank you for the question.

alert_history_max_docs and alert_history_max_age use OR logic, rollover triggers when either max_docs OR max_age is met at the next rollover_period check. Default is 12 hours.

Regarding the configuration, you should be able to use the following:

Configure Alert History:

PUT _cluster/settings
{
  "persistent": {
    "plugins.security_analytics.alert_history_enabled": true,
    "plugins.security_analytics.alert_history_rollover_period": "1h",
    "plugins.security_analytics.alert_history_max_docs": 10000,
    "plugins.security_analytics.alert_history_max_age": "7d",
    "plugins.security_analytics.alert_history_retention_period": "30d"
  }
}

Configure Findings History:

PUT _cluster/settings
{
  "persistent": {
    "plugins.security_analytics.alert_finding_enabled": true,
    "plugins.security_analytics.alert_finding_rollover_period": "1h",
    "plugins.security_analytics.finding_history_max_age": "7d",
    "plugins.security_analytics.finding_history_retention_period": "30d"
  }
}

Looking at the code, it would appear that alert_finding_max_docs is deprecated. You can use finding_history_max_age for controlling rollover by time instead.

Configure Correlation and IOC Findings History:

PUT _cluster/settings
{
  "persistent": {
    "plugins.security_analytics.correlation_history_rollover_period": "1h",
    "plugins.security_analytics.correlation_history_max_docs": 10000,
    "plugins.security_analytics.correlation_history_max_age": "7d",
    "plugins.security_analytics.correlation_history_retention_period": "30d",

    "plugins.security_analytics.ioc_finding_enabled": true,
    "plugins.security_analytics.ioc_finding_history_rollover_period": "1h",
    "plugins.security_analytics.ioc_finding_history_max_docs": 10000,
    "plugins.security_analytics.ioc_finding_history_max_age": "7d",
    "plugins.security_analytics.ioc_finding_history_retention_period": "30d"
  }
}

Hope this helps