Securing resources for Anomaly Detection

Machine Learning
#1

Hello, I have a question regarding resources management when there are Anomaly Detection jobs on the cluster.
We have standard scenario:

  • we have a cluster used for monitoring (ETL jobs deliver the data, users are consuming dashboards)
  • some users are doing additionall ad-hoc data analysis
  • some users are defining new ML jobs using Anomaly Detection module.

As I understand by default AD jobs are using datanodes, which can result in all cluster nodes to have problems if somebody will use too much resources for AD.
On “standard elastic” there is a way of dedicating nodes for ML jobs by assigning them ml role.

So is there a way of assigning AD nodes like ML role?
Or how you guys are managing limitation or making sure there is a resource pool for core cluster actions so the AD jobs will not cause problems like too high load on data nodes or Out Of Memory errors?

BR
TD

#2

Currently AD plugin will run on data nodes. We are planning to add ML role, but still researching. Welcome any suggestion.

AD has some protection mechanism to avoid using too much resource

  1. Circuit breaker: AD will stop running if JVM heap usage exceeds 85%
  2. Run AD in dedicated thread pool
  3. Track memory usage of AD model, and limit the memory usage
  4. Limit how many detectors can run per cluster via dynamic setting, default is 1000