Anamoly Detection for the exact host.ip or agent.name

Plugins Machine Learning
1

I can filter only value fields / boolean fields from indices. i am unable to choose the string filter from index.

This is very much important for the case of anomaly detection for an host.
Is it possible to do the anomaly detection for the specific host.ip ?
Opendistro seems good to use and have more features related to siem , kindly assist to get the above use case in filter.

I have pasted the code used for my network infrastructure,
{
“bool”: {
“filter”: [
{
“range”: {
"system.network.out.bytes ": {
“from”: 500,
“to”: null,
“include_lower”: false,
“include_upper”: true,
“boost”: 1
}
}
},
{
“range”: {
“system.process.cpu.total.value”: {
“from”: 600,
“to”: null,
“include_lower”: false,
“include_upper”: true,
“boost”: 1
}
}
},
{
“range”: {
“system.process.cgroup.memory.mem.usage.bytes”: {
“from”: 500,
“to”: null,
“include_lower”: false,
“include_upper”: true,
“boost”: 1
}
}
},
{
“range”: {
“system.process.cpu.total.pct”: {
“from”: 0.08,
“to”: null,
“include_lower”: false,
“include_upper”: true,
“boost”: 1
}
}
},
{
“term”: {
“system.cpu.system.pct”: {
“value”: 9.81,
“boost”: 1
}
}
}
],
“adjust_pure_negative”: true,
“boost”: 1
}
}

For this usecase , I like to add ( string value 0specific hostname / agent.name.
By the way I will filter out the machine which is really affected, otherwise it is simply a measurement.

2

By the way I will filter out the machine which is really affected

We are working on high cardinality feature now, which will group by some field like hostname in your case, then do anomaly detection for each category(hostname in your case). So you can find out which host has anomaly. Does this feature meet your requirement?

Github issue: Enable support high cardinality in anomaly detection · Issue #147 · opendistro-for-elasticsearch/anomaly-detection · GitHub,

BTW, can you clarify more about your requirements? Like how many hostnames do you want to support, what features do you want: like filtering only anomalous hostnames, sending our alerts if any host has anomaly etc.

3

My Use Case ,

In filter I have chosen the cpu usage as filed from metricbeat index, it shows the anomaly detection of the CPU used more than the threshold value but it is not showing as a result that this (ip / hostname’s ) cpu has utilized the value.

I like to find out which pc has really ustilised more than the threshold value.

4

Hi Ganesh,

This should now be possible with High cardinality support.Have you tried it? We would love to get community’s feedback on the high cardinality feature.

Thanks,
Pavani