Derivative Features for Anomaly Detection Plugins

[ridingbikes]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
1.3.2

Describe the issue:
I am trying to create an anomaly detector that generates anomalies from a derivative of aggregated counts for a specific field in my index. I am using an ElasticSearch blog post as reference (Custom Elasticsearch Aggregations for Machine Learning Jobs | Elastic Blog). I am able to create aggregations with the following expression. However, whenever I attempt to move this expression (without the outer-level “aggs” key) to the Feature’s Custom Expression box within the Anomaly Detector interface, I keep getting an error. I’m not sure what to make of this error or what my actual issue is. Any ideas? Thanks in advance for your help.

{
“aggs”: {
“changes_in_count”: {
“date_histogram”: {
“field”: “DateTime”,
“interval”: “month”
},
“aggs”: {
“DateTime”: {
“max”: {
“field”: “DateTime”,
}
},
“doc_version_count”: {
“value_count”: {
“field”: “Id.keyword”
}
},
“deriv”: {
“derivative”: {
“buckets_path”: “_count”
}
}
}
}
}
}

Relevant Logs or Screenshots:

[kaituo]

I’d like to bring to your attention that the configuration format in OpenSearch’s Anomaly Detection differs from that of Elastic’s Machine Learning. In OpenSearch, we employ the concept of a “detector” instead of “job” and “data feed” used in Elastic’s ML.

For instance, consider the following Elastic’s ML aggregation:

"aggregations": {
    "buckets": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "5m",
        "time_zone": "UTC"
      },
      "aggregations": {
        "@timestamp": {
          "max": {
            "field": "@timestamp"
          }
        },
        "orders": {
          "sum": {
            "field": "events_per_min"
          }
        },
        "orders_deriv": {
          "derivative": {
            "buckets_path": "orders"
          }
        }
      }
    }
  }

The above can be translated to the following detector settings in OpenSearch:

POST _plugins/_anomaly_detection/detectors
{
	"name": "orders_deriv",
	"description": "Derivative of Order Volume",
	"time_field": "@timestamp",
	"indices": [
		"it_ops_kpi-2017"
	],
	"feature_attributes": [{
		"feature_name": "orders sum",
		"feature_enabled": true,
		"aggregation_query": {
			"orders": {
				"sum": {
					"field": "events_per_min"
				}
			}
		}
	}],
	"detection_interval": {
		"period": {
			"interval": 5,
			"unit": "MINUTES"
		}
	}
}

We encourage you to reference the OpenSearch documentation at OpenSearch Anomaly Detection and Amazon OpenSearch Service to better understand and leverage the power of OpenSearch’s anomaly detection capabilities.