Anomaly Detection 1.8 with ELK7.7 "Stopped detector: limit_exceeded_exception: AD memory circuit is broken"

We have upgraded our Stack to ELK7.7 with AD 1.8 but our current AD detectors fail to start. We are starting the detector and after some period of time it fails with “Initialization failure” info message and these two additional exception message detail. I could not find any clear documentation about what to do or where to look at for solution.

Exception details:

“state” : “DISABLED”,
“error” : “Stopped detector: limit_exceeded_exception: AD memory circuit is broken.”

“state” : “DISABLED”,
“error” : “Stopped detector: AD memory circuit is broken.”

hi, @ulasyuksel

To avoid AD bring too much load, we have MemoryCircuitBreaker (Github code) to block AD running if JVM heap used percentage exceeds 85%. Can you check the JVM usage?

BTW, we are working to let user enable/disable circuit breaker (#169) and user can change threshold with dynamic setting(#175).

We have checked the percentage of jvm usage and it seems that we are exceeding %85 ratio a few times in an hour. We are trying to limit jvm heap size and giving remaining memory to filesystem cache for queries.
I’m not sure if we increase the jvm heap size(we should think twice because of long gc side effects) everything will be ok, because our system seems very stable for a long time.
Anyway, we are looking forword the new circuit breaker switch features to test.