Anomaly detection Url based detection

OpenSearch
1

Hi Team,

I use apache filebeat module. I want to enable anomaly detection based on url.original field. Is is possible to that?

If one url sum count increase for some intervals, I want to get alert.

example output;
url.original: /gateway/v1/services/language/translate?platform=web&language=tr

thanks for help

2

Also even if I create any detecter, I see the anomaly detections but no details what is the anomaly.

3

Hello!

What exactly do you mean by your description? So do you want a notification when the amount of calls to a certain URL is above or below the usual amount?

And in regard to your second point: The details of why the anomaly occured should correspond to your detector configuration. So if your detector monitors the amount of logs, if an anomaly occurs the amount is different than expected.

BR, AES

5

Hi,

Thanks for response. your understanding totally right
“I want a notification when the amount of calls to a certain URL is above or below the usual amount?” I use standard apache log and filebeat apache module.

another question is that I left detector interval as default 10 min. So system just check data in 10 min yo understand anomaly or system check anomaly between last 10min data and other 10 min datas.

Does system compare all data in 10 min intervals to understand anomalies?

Thanks,
Emre

6

Hello,

As far as I can tell there is no way for you to automatically create separate buckets for each and every url in Anomaly detection.
In your case I would maybe sum the total amount of access logs to your web page, or to specific paths (eg. /gateway/v1/, /home/ etc) manually.

And in regards to the detector:
Yeah so if you leave it at the default of 10 minutes, it loads the last 10 minutes of available data and basically attempts to assign it an anomaly score.
You can get more detailed information in the docs:
https://opensearch.org/docs/latest/monitoring-plugins/ad/index/

BR, AES

8

Hi,

Thank you.

" In your case I would maybe sum the total amount of access logs to your web page, or to specific paths (eg. /gateway/v1/, /home/ etc) manually."

How can I sum total amount of access logs for specific paths manually? I’m not able to select “url.original” field which show urls in my access log index. I dont see all fields in detector field configuration. it just let me select few of them.

Emre

9

Hello,

let me show you with an example. Here I have my auditbeat documents in the auditbeat-* index-pattern.
Some documents have the file.path field:

file

If I wanted to create an anomaly detector for every file.path which starts with /usr/bin the configuration would look like this in the Anomaly Detection configurator:

detect
detect765×889 46.8 KB

In the Data source you should be able to define a Data filter. Here I set the path to begin with /usr/bin, so my anomaly detector would only check the documents with this filter.

Or are you not able to select url.original at all in this wizard?

BR,
Andreas

11

Hi,

Thank you very much details. I added filter as you showed. But next page, it ask “Field” for “find anomalies based on”. I guess it’s must. what should we select ?

image
image605×629 69.8 KB

And I want to ask you that I created 2-3 detectors according to different detector intervals for testing but when I set it 1 minute,it find multiple anomalies but when I set it as 3,5,10 min, detector didnt find any anomalies. I’m not get correctly how detector interval affected anomalies finding.

Thanks for you all help.

Regards,

Emre