# Security Analytics only alerting at 12am UTC

**URL:** <https://forum.opensearch.org/t/security-analytics-only-alerting-at-12am-utc/19502>\
**Category:** Security Analytics\
**Tags:** alerting\
**Created:** [May 22, 2024, 6:54am UTC](https://forum.opensearch.org/t/security-analytics-only-alerting-at-12am-utc/19502 "2024-05-22T06:54:54Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vger](https://avatars.discourse-cdn.com/v4/letter/v/87869e/32.png) [@vger](https://forum.opensearch.org/u/vger)\
**Post date:** [May 22, 2024, 6:54am UTC](https://forum.opensearch.org/t/security-analytics-only-alerting-at-12am-utc/19502/1 "2024-05-22T06:54:54Z")

</div>

Hi there,

we are having trouble with correctly setting up alertings in the security analytics plugin. Our OpenSearch instance only does findings and corresponding alertings in the first seconds after 12am UTC. We don’t have any alert throttling enabled in our detectors or alert triggers. The detector is configured to run every minute. The events to which the detector should respond occur permanently throughout the day and are correctly detected in the given timeframe (first seconds after 12am UTC every day). See the dashboard screenshot for a few details.

Has anyone else seen this behavior or any hint on what might be the issue here? We are on OpenSearch version 2.13.0.

Any help is appreciated.

 ![Bildschirmfoto 2024-05-22 um 08.49.37](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/e/e25921d8f1b7b91ad39ff7d3a0f4eb834acbb2b8.png)

---

<div class="post-metadata">

**Author:** ![vger](https://avatars.discourse-cdn.com/v4/letter/v/87869e/32.png) [@vger](https://forum.opensearch.org/u/vger)\
**Post date:** [May 23, 2024, 7:20am UTC](https://forum.opensearch.org/t/security-analytics-only-alerting-at-12am-utc/19502/2 "2024-05-23T07:20:48Z")

</div>

I’ve found something in the logs. It seems like the Security Analytics plugin is crashing at the point the alerts are happening:

```auto
[2024-05-22T00:00:49,071][ERROR][o.o.s.u.SecurityAnalyticsException] [siem-2.au.kim.akquinet.nx2.dev] Security Analytics error:
java.lang.IllegalStateException: existing codec service factory already overridden in: org.opensearch.index.codec.customcodecs.CustomCodecPlugin attempting to override again by: org.opensearch.securityanalytics.SecurityAnalyticsPlugin
        at org.opensearch.index.engine.EngineConfigFactory.<init>(EngineConfigFactory.java:109) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.index.engine.EngineConfigFactory.<init>(EngineConfigFactory.java:65) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.IndicesService.getEngineConfigFactory(IndicesService.java:907) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.IndicesService.createIndexService(IndicesService.java:868) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.IndicesService.withTempIndexService(IndicesService.java:823) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:483) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV2Template(MetadataCreateIndexService.java:653) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:426) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:452) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:358) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:67) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:882) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:434) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:301) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:212) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246) ~[opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

```

The only hit on Google was this:

> <https://github.com/opensearch-project/OpenSearch/issues/7012>
>
> \*\*Describe the bug\*\*
> Plugins cannot continue to use existing approach and regis…ter CodecServiceFactory and cluster is crushing in runtime (compile time is ok). Example is kNN plugin, error on cluster start:
> 
> https://github.com/opensearch-project/k-NN/actions/runs/4614564115/jobs/8170344936#step:9:1265
> \`\`\`
> » java.lang.IllegalStateException: existing codec service factory already overridden in: org.opensearch.index.codec.customcodecs.CustomCodecPlugin attempting to override again by: org.opensearch.knn.plugin.KNNPlugin
> » at org.opensearch.index.engine.EngineConfigFactory.\<init\>(EngineConfigFactory.java:107) ~\[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT\]
> » at org.opensearch.index.engine.EngineConfigFactory.\<init\>(EngineConfigFactory.java:63) ~\[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT\]
> » at org.opensearch.indices.IndicesService.getEngineConfigFactory(IndicesService.java:913) ~\[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT\]
> » at org.opensearch.indices.IndicesService.createIndexService(IndicesService.java:874) ~\[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT\]
> » at org.opensearch.indices.IndicesService.createIndex(IndicesService.java:775) ~\[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT\]
> » at org.opensearch.indices.IndicesService.createIndex(IndicesService.java:209) ~\[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT\]
> » at org.opensearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:539) \[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT\]
> \`\`\`
> 
> Code on kNN side that is registering CustomServiceFactory:
> https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/plugin/KNNPlugin.java#L271-L276
> 
> \*\*To Reproduce\*\*
> Steps to reproduce the behavior:
> 1. Get main branch from k-NN repo
> 2. Do \`\`\`./gradlew run\`\`\`
> 
> \*\*Expected behavior\*\*
> There should be alternative way of either enabling/disabling plugins or for plugin to register their CustomServiceFactory.
> 
> \*\*Plugins\*\*
> distribution build, k-NN is included
> 
> \*\*Additional context\*\*
> Core change introduced in https://github.com/opensearch-project/OpenSearch/pull/3577 
> 
> Part of PR that is registering new factory: https://github.com/opensearch-project/OpenSearch/pull/3577/files#diff-629964cdbeae1039851179519fc3e3d3cc45e8329d208dd9f2b7034537675262R38
> 
> Core code where the check for only one factory is happening: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/engine/EngineConfigFactory.java#L79

I can’t really find a correlation on those issues, yet. Anyone have an idea?

---

<div class="post-metadata">

**Author:** ![Gsmitt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gsmitt/32/1718_2.png) [@Gsmitt](https://forum.opensearch.org/u/Gsmitt)\
**Post date:** [May 24, 2024, 3:18am UTC](https://forum.opensearch.org/t/security-analytics-only-alerting-at-12am-utc/19502/3 "2024-05-24T03:18:52Z")

</div>

Hey @vger

By chance is this a fresh install?

---

<div class="post-metadata">

**Author:** ![vger](https://avatars.discourse-cdn.com/v4/letter/v/87869e/32.png) [@vger](https://forum.opensearch.org/u/vger)\
**Post date:** [May 24, 2024, 4:46am UTC](https://forum.opensearch.org/t/security-analytics-only-alerting-at-12am-utc/19502/4 "2024-05-24T04:46:43Z")

</div>

Hi @Gsmitt ,

Yes, this is a fresh install.

---

<div class="post-metadata">

**Author:** ![vger](https://avatars.discourse-cdn.com/v4/letter/v/87869e/32.png) [@vger](https://forum.opensearch.org/u/vger)\
**Post date:** [June 4, 2024, 7:29am UTC](https://forum.opensearch.org/t/security-analytics-only-alerting-at-12am-utc/19502/5 "2024-06-04T07:29:11Z")

</div>

I’ve opened a bug report and there’s already a pull request for this issue: [[BUG] Custom Codec Plugin breaking Security Analytics Plugin Alerts · Issue #1050 · opensearch-project/security-analytics · GitHub](https://github.com/opensearch-project/security-analytics/issues/1050)

So this problem should be resolved shortly.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [August 3, 2024, 7:29am UTC](https://forum.opensearch.org/t/security-analytics-only-alerting-at-12am-utc/19502/6 "2024-08-03T07:29:31Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.