Security Analytics only alerting at 12am UTC

vger · May 22, 2024, 6:54am

Hi there,

we are having trouble with correctly setting up alertings in the security analytics plugin. Our OpenSearch instance only does findings and corresponding alertings in the first seconds after 12am UTC. We don’t have any alert throttling enabled in our detectors or alert triggers. The detector is configured to run every minute. The events to which the detector should respond occur permanently throughout the day and are correctly detected in the given timeframe (first seconds after 12am UTC every day). See the dashboard screenshot for a few details.

Has anyone else seen this behavior or any hint on what might be the issue here? We are on OpenSearch version 2.13.0.

Any help is appreciated.

vger · May 23, 2024, 7:20am

I’ve found something in the logs. It seems like the Security Analytics plugin is crashing at the point the alerts are happening:

[2024-05-22T00:00:49,071][ERROR][o.o.s.u.SecurityAnalyticsException] [siem-2.au.kim.akquinet.nx2.dev] Security Analytics error:
java.lang.IllegalStateException: existing codec service factory already overridden in: org.opensearch.index.codec.customcodecs.CustomCodecPlugin attempting to override again by: org.opensearch.securityanalytics.SecurityAnalyticsPlugin
        at org.opensearch.index.engine.EngineConfigFactory.<init>(EngineConfigFactory.java:109) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.index.engine.EngineConfigFactory.<init>(EngineConfigFactory.java:65) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.IndicesService.getEngineConfigFactory(IndicesService.java:907) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.IndicesService.createIndexService(IndicesService.java:868) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.IndicesService.withTempIndexService(IndicesService.java:823) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:483) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV2Template(MetadataCreateIndexService.java:653) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:426) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:452) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:358) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:67) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:882) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:434) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:301) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:212) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246) ~[opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

The only hit on Google was this:

github.com/opensearch-project/OpenSearch

[BUG] Zstd new codec is a breaking change for kNN plugin

opened 04:49PM - 05 Apr 23 UTC

closed 07:44PM - 13 Apr 23 UTC

martin-gaievski

bug

**Describe the bug** Plugins cannot continue to use existing approach and regis…ter CodecServiceFactory and cluster is crushing in runtime (compile time is ok). Example is kNN plugin, error on cluster start: https://github.com/opensearch-project/k-NN/actions/runs/4614564115/jobs/8170344936#step:9:1265 ``` » java.lang.IllegalStateException: existing codec service factory already overridden in: org.opensearch.index.codec.customcodecs.CustomCodecPlugin attempting to override again by: org.opensearch.knn.plugin.KNNPlugin » at org.opensearch.index.engine.EngineConfigFactory.<init>(EngineConfigFactory.java:107) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] » at org.opensearch.index.engine.EngineConfigFactory.<init>(EngineConfigFactory.java:63) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] » at org.opensearch.indices.IndicesService.getEngineConfigFactory(IndicesService.java:913) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] » at org.opensearch.indices.IndicesService.createIndexService(IndicesService.java:874) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] » at org.opensearch.indices.IndicesService.createIndex(IndicesService.java:775) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] » at org.opensearch.indices.IndicesService.createIndex(IndicesService.java:209) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] » at org.opensearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:539) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] ``` Code on kNN side that is registering CustomServiceFactory: https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/plugin/KNNPlugin.java#L271-L276 **To Reproduce** Steps to reproduce the behavior: 1. Get main branch from k-NN repo 2. Do ```./gradlew run``` **Expected behavior** There should be alternative way of either enabling/disabling plugins or for plugin to register their CustomServiceFactory. **Plugins** distribution build, k-NN is included **Additional context** Core change introduced in https://github.com/opensearch-project/OpenSearch/pull/3577 Part of PR that is registering new factory: https://github.com/opensearch-project/OpenSearch/pull/3577/files#diff-629964cdbeae1039851179519fc3e3d3cc45e8329d208dd9f2b7034537675262R38 Core code where the check for only one factory is happening: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/engine/EngineConfigFactory.java#L79

I can’t really find a correlation on those issues, yet. Anyone have an idea?

Gsmitt · May 24, 2024, 3:18am

Hey @vger

By chance is this a fresh install?

vger · May 24, 2024, 4:46am

Hi @Gsmitt ,

Yes, this is a fresh install.

vger · June 4, 2024, 7:29am

I’ve opened a bug report and there’s already a pull request for this issue: [BUG] Custom Codec Plugin breaking Security Analytics Plugin Alerts · Issue #1050 · opensearch-project/security-analytics · GitHub

So this problem should be resolved shortly.

system · August 3, 2024, 7:29am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Feedback: Experimental Feature - Security Analytics General Feedback	10	810	January 3, 2023
Can't view any alerts or findings ( Security Analytics ) Security Analytics	1	61	October 6, 2024
Opensearch alerting plugin is throwing exception Filter by user backend roles is enabled by security disabled Alerting	7	833	June 8, 2023
Security analytics - not able create detector Security Analytics	3	296	July 10, 2024
Issue with Scheduled job for detectors Security Analytics troubleshoot	5	470	August 1, 2023

Security Analytics only alerting at 12am UTC

Related topics