Anomaly Detection Error after upgrade from 1.11 to 1.12

Plugins Machine Learning
1

“Unable to get all detectors”
We had previously created Anomaly detectors while we where are version 1.11 and this was working fine.
After the upgrade from 1.11 to 1.12 we can’t access our pre defined detectors …

Any help would be much appreciated

2

You mean the detector list page doesn’t show any old detectors ?

3

That is correct.
But while a check with API it lists out status Green and also lists the total number of Detectors (detector_count)

4

Can you show which API you are using? Can you try get detector API and profile API to check if they can return correct result?

5

GET _opendistro/_anomaly_detection/stats

Which shows Status Green and the Total Count of the detectors available

6

If you don’t mind, can you share the commands and the response, maybe some screenshots? So we can understand the problem quickly.

7

Anomaly Error
Anomaly Error1919×899 57.4 KB

Under Dev Tools if i run the following command >> GET _opendistro/_anomaly_detection/stats
I get the following response {truncated for easy viz }
{
“anomaly_detectors_index_status” : “green”,
“anomaly_detection_state_status” : “green”,
"detector_count" : 15,
“anomaly_detection_job_index_status” : “green”,
“models_checkpoint_index_status” : “green”,
“anomaly_results_index_status” : “green”,
“nodes” : {
“G3VywNOURtuwWj49c6_npg” : {
“ad_execute_request_count” : 0,
“ad_execute_failure_count” : 0,
“models” : [
{
“detector_id” : “NySLrHQBUMfGiwP1Y1yC”,
“model_type” : “threshold”,
“model_id” : “NySLrHQBUMfGiwP1Y1yC_model_threshold”
},
{
“detector_id” : “ZhbstngBLc5aSMTuST5l”,
“model_type” : “threshold”,
“model_id” : “ZhbstngBLc5aSMTuST5l_model_threshold”
}
],
…

But the issue is after the upgrade from 1.11 to 1.12, I cant view or edit any off my Anomaly Detectors.
The Dashbaord and the Detectors Window is empty. [see picture above && give me that error msg]

8

Got it. Looks like some bug. Can you share more details so we can reproduce this error and debug, like on which OS you are running, the upgrade steps,

9

They are running in CentOS7 and the upgrade path has been always the same as prescribed by this post

10

Can’t reproduce the problem. What I have done:

Install ODFE 1.11 on a new EC2 instance with CentOS

  1. sudo curl https://d3g5vo6xdbdb9a.cloudfront.net/yum/opendistroforelasticsearch-artifacts.repo -o /etc/yum.repos.d/opendistroforelasticsearch-artifacts.repo
  2. sudo yum install java-11-openjdk-devel
  3. sudo yum install wget unzip
  4. Install ES sudo yum install opendistroforelasticsearch-1.11.0 , then start sudo systemctl start elasticsearch.service
  5. Ingest test data and create detector
  6. Install Kibana , sudo yum install opendistroforelasticsearch-kibana-1.11.0, shen start

Upgrade

  1. Stop ES and Kibana, then upgrade with sudo yum install opendistroforelasticsearch-1.13.2 and sudo yum install opendistroforelasticsearch-kibana-1.13.2
    . Start Kibana and ES , can still see the test detector created before.
11

We had detectors built during 1.8 ver don’t know if that had anything do with this… IS there a way leveraging API i can delete all my Detectors so that i can recreate them ?

12

Yes, you can check this doc Anomaly Detection API - Open Distro Documentation
These steps may help you

  1. Search all detectors
GET _opendistro/_anomaly_detection/detectors/_search
{
  "size": 100, 
    "query": {
        "match_all": {}
    }
}

  1. Stop detector if it’s running
    POST _opendistro/_anomaly_detection/detectors/<detectorId>/_stop

  2. Delete detector
    DELETE _opendistro/_anomaly_detection/detectors/<detectorId>