Hi all. I’m testing Elastic Stack and OpenSearch to see if Anomaly Detection will work for my company. The Elastic Stack (paid) version clearly worked out of the box. However, the OS version seems to barely work at all. Could anyone help me?
I suspect the problem is that OS Anomaly Detection really can’t handle it when data is missing (or sparse). If so, I need to find a way around this problem, or abandon OS altogether.
I set up a test case by pulling data from a database with the JDBC connector. I got 500 days of data from a single table, and set the @timestamp to row creation date. It all appears as expected. But note that for this app, there is often no data on the weekends. It looks like this:
I manually deleted all but four items for 10/31 (right arrow), expecting it to show a big anomaly for that date.
I then created a new anomaly detector, to just get the count of records, in one-day buckets. It analyzed all 500 days of history, and did not report a problem doing so. However, it showed no anomaly for 10/31 (or at all). Note that all the buckets with zero or near-zero counts are on the weekend. But 10/31 was a Tuesday.
Could anyone tell me if this is expected behavior? As a developer, I understand the technical difference between zero and null (no data). However, it’s crucial for me that it report anomalies even if the bucket has no data. The Elastic Stack Anomaly Detection had no problem with empty buckets. If it can’t report on empty buckets, it should at least report on near-empty buckets. And it should certainly know the difference between a weekend and a weekday after 500 days.
Please help me! Thanks! 