Issue Creating Detectors – Timeout and Continuous Index Creation/Deletion (Security Analytics)

Plugins Security Analytics
1

Dear Support Team,

We are currently experiencing an issue with the Security Analytics feature in our OpenSearch cluster.

Specifically, we are unable to create new detectors. Every attempt to create a detector results in a timeout.

OpenSearch version: 3.5.0

Observed Behavior:

  • Detector creation requests consistently timeout.

  • The cluster shows a large number of pending tasks, primarily related to:

    • index creation

    • index deletion

  • These operations appear to be continuously triggered and do not stabilize.

image
image1711×601 43.2 KB

2

@loi Do you have any other SA detectors, or is this the first one?
Do you see any errors in the OpenSearch logs? How big is the cluster?

3

Hi @pablo,

Please find the requested details below:

  1. Detectors:
    This is the first Security Analytics detector we are trying to create.

  2. OpenSearch logs:
    Please review the attached logs. The issue occurs during detector creation, where the request eventually times out. We are also observing continuous pending tasks, mainly related to index operations.

  3. Cluster information:
    Our cluster consists of 3 nodes:

  • 1 master node

  • 2 data nodes

Resources:

  • RAM: 128 GB

  • CPU: 20 cores

  • Disk: 8 TB

Given the available resources, we do not expect this behavior to be caused by capacity limitations.

If you need any additional information (e.g., specific logs, cluster stats, or configurations) to further investigate the issue, please let us know and we will provide it.

Log
[EbXxK7A2R2Ouh8jolK6hWg], primary term [1], message [after peer recovery]}[StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-751a37e9-a5b1-4599-aa3c-73372cd011c9-000001][0]], allocationId [EbXxK7A2R2Ouh8jolK6hWg], primary term [1], message [after peer recovery]}], shard-started StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-45006d7b-02c5-442c-ad82-890f8a005bcc-000001][0]], allocationId [6OD8p4SLRT64JK1Ta-lg1w], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}[StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-45006d7b-02c5-442c-ad82-890f8a005bcc-000001][0]], allocationId [6OD8p4SLRT64JK1Ta-lg1w], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}, StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-45006d7b-02c5-442c-ad82-890f8a005bcc-000001][0]], allocationId [6OD8p4SLRT64JK1Ta-lg1w], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}], shard-started StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-751a37e9-a5b1-4599-aa3c-73372cd011c9-000001][0]], allocationId [EbXxK7A2R2Ouh8jolK6hWg], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}[StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-751a37e9-a5b1-4599-aa3c-73372cd011c9-000001][0]], allocationId [EbXxK7A2R2Ouh8jolK6hWg], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}, StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-751a37e9-a5b1-4599-aa3c-73372cd011c9-000001][0]], allocationId [EbXxK7A2R2Ouh8jolK6hWg], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}]]
[2026-04-22T03:40:20,683][DEBUG][o.o.c.s.ClusterManagerService] [opensearch-node3] cluster state updated, version [1505945], source [Tasks batched with key: org.opensearch.cluster.action.shard.ShardStateAction, count:6 and sample tasks: shard-started StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-45006d7b-02c5-442c-ad82-890f8a005bcc-000001][0]], allocationId [6OD8p4SLRT64JK1Ta-lg1w], primary term [1], message [after new shard recovery]}[StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-45006d7b-02c5-442c-ad82-890f8a005bcc-000001][0]], allocationId [6OD8p4SLRT64JK1Ta-lg1w], primary term [1], message [after new shard recovery]}], shard-started StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-751a37e9-a5b1-4599-aa3c-73372cd011c9-000001][0]], allocationId [EbXxK7A2R2Ouh8jolK6hWg], primary term [1], message [after peer recovery]}[StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-751a37e9-a5b1-4599-aa3c-73372cd011c9-000001][0]], allocationId [EbXxK7A2R2Ouh8jolK6hWg], primary term [1], message [after peer recovery]}], shard-started StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-45006d7b-02c5-442c-ad82-890f8a005bcc-000001][0]], allocationId [6OD8p4SLRT64JK1Ta-lg1w], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}[StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-45006d7b-02c5-442c-ad82-890f8a005bcc-000001][0]], allocationId [6OD8p4SLRT64JK1Ta-lg1w], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}, StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-45006d7b-02c5-442c-ad82-890f8a005bcc-000001][0]], allocationId [6OD8p4SLRT64JK1Ta-lg1w], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}], shard-started StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-751a37e9-a5b1-4599-aa3c-73372cd011c9-000001][0]], allocationId [EbXxK7A2R2Ouh8jolK6hWg], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}[StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-751a37e9-a5b1-4599-aa3c-73372cd011c9-000001][0]], allocationId [EbXxK7A2R2Ouh8jolK6hWg], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}, StartedShardEntry{shardId [[.opensearch-sap-ad_ldap-detectors-queries-optimized-751a37e9-a5b1-4599-aa3c-73372cd011c9-000001][0]], allocationId [EbXxK7A2R2Ouh8jolK6hWg], primary term [1], message [master {opensearch-node3}{8o1zpAWqTDKvUeToK4CmCA}{6ick7nj_R5mlVUWyJR-vIw}{opensearch-node3}{172.18.0.4:9300}{m}{shard_indexing_pressure_enabled=true} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]}]]
[2026-04-22T03:40:20,683][DEBUG][o.o.c.s.ClusterManagerService] [opensearch-node3] publishing cluster state version [1505945]
[2026-04-22T03:40:20,684][DEBUG][o.o.c.c.Coordinator ] [opensearch-node3] initialized PublicationContext using class: class org.opensearch.cluster.coordination.PublicationTransportHandler$PublicationContext

Summary

This text will be hidden

4

Are these related to the Security Analytics creation detector, or did they exist before?

5

@loi I’ve just tested 3.5.0 and had no issues creating detectors.

image
image1680×777 64.4 KB

What user did you use for detector creation? Did you have the same timeout error with the default admin user?