# Anomaly Detection for Continuous Ingesting Data **URL:** \ **Category:** Machine Learning\ **Tags:** troubleshoot\ **Created:** [April 22, 2021, 9:25pm UTC](https://forum.opensearch.org/t/anomaly-detection-for-continuous-ingesting-data/5721 "2021-04-22T21:25:44Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![rlinux57](https://avatars.discourse-cdn.com/v4/letter/r/3bc359/32.png) [@rlinux57](https://forum.opensearch.org/u/rlinux57)\ **Post date:** [April 22, 2021, 9:25pm UTC](https://forum.opensearch.org/t/anomaly-detection-for-continuous-ingesting-data/5721/1 "2021-04-22T21:25:44Z")
Hi, Currently I have ingesting a 20-25GB of daily data in opendistro. And I have created multiple detectors and set aggregation method avg but its getting failed after a couple of hours and got message not have enough data to initialize detector. But when i set aggregation method SUM. The detectors running successfully. Please let me know where I am having mistake. What would be the settings for Detector Interval, Window delay and window size for it ? ---
**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\ **Post date:** [April 23, 2021, 9:15pm UTC](https://forum.opensearch.org/t/anomaly-detection-for-continuous-ingesting-data/5721/2 "2021-04-23T21:15:15Z")
AD will detect realtime streaming data. Can you check the date range your data? For `SUM`, it will return 0 if no data found, but for `AVG`, it will return `null` if no data found. So if you are using `SUM`, the detector can run successfully as it can get 0 even no data. ---
**Author:** ![rlinux57](https://avatars.discourse-cdn.com/v4/letter/r/3bc359/32.png) [@rlinux57](https://forum.opensearch.org/u/rlinux57)\ **Post date:** [April 24, 2021, 5:26pm UTC](https://forum.opensearch.org/t/anomaly-detection-for-continuous-ingesting-data/5721/3 "2021-04-24T17:26:05Z")
Hi, I’m getting following error message for the detector I have created: " The detector is not initialized because no sufficient data is ingested. Make sure your data is ingested correctly. If your data source has infrequent ingestion, increase the detector time interval and try again." Also I have shared the screenshot of detector settings. **Window Size: 8** ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/4/4f1c190ee51f663ede7224c702f50b4380c3076d.png) ---
**Author:** ![rlinux57](https://avatars.discourse-cdn.com/v4/letter/r/3bc359/32.png) [@rlinux57](https://forum.opensearch.org/u/rlinux57)\ **Post date:** [April 26, 2021, 8:08pm UTC](https://forum.opensearch.org/t/anomaly-detection-for-continuous-ingesting-data/5721/4 "2021-04-26T20:08:50Z")
@ylwu Please check my previous message. Can you help me on it ? ---
**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\ **Post date:** [April 28, 2021, 12:28am UTC](https://forum.opensearch.org/t/anomaly-detection-for-continuous-ingesting-data/5721/5 "2021-04-28T00:28:14Z")
From the screenshot, the detector interval is `40 minutes`. The error message means you have no enough data to train model. The model need about 150 data points to train. Here one data point is the aggregated result of one `40 minutes` interval. If you don’t have historical data, then the detector will wait 150 intervals to get enough training data. That will be `150 * 40 = 6000 minutes`. To make the model pass training faster, you can ingest enough historical data or reduce detector interval, make sure every interval (`40 minutes` in this case) has data. ---
**Author:** ![zack](https://avatars.discourse-cdn.com/v4/letter/z/f6c823/32.png) [@zack](https://forum.opensearch.org/u/zack)\ **Post date:** [March 7, 2022, 7:56pm UTC](https://forum.opensearch.org/t/anomaly-detection-for-continuous-ingesting-data/5721/6 "2022-03-07T19:56:56Z")
> make sure every interval ( `40 minutes` in this case) has data. If an interval, or multiple intervals, has no data can the real time detector ever recover? I have a detector that seems to start normally with historical data but if there is ever a lull in the index receiving data the real time detector appears to stop working. The only way the detector recovers is with a restart of the OpenSearch node. ---
**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\ **Post date:** [March 8, 2022, 3:46am UTC](https://forum.opensearch.org/t/anomaly-detection-for-continuous-ingesting-data/5721/7 "2022-03-08T03:46:11Z")
The realtime detector job assumes there are always some data under your interval and data filtering settings. It’s ok that data missing in some interval, detector job may skip these intervals. If data back to normal (there is data in all or most intervals), then detector job can work correctly again. Can you explain why you have to restart OpenSearch nodes? Can restart detector realtime job work? If don’t restart nodes, what exception do you see? ---
**Author:** ![zack](https://avatars.discourse-cdn.com/v4/letter/z/f6c823/32.png) [@zack](https://forum.opensearch.org/u/zack)\ **Post date:** [March 8, 2022, 3:55pm UTC](https://forum.opensearch.org/t/anomaly-detection-for-continuous-ingesting-data/5721/8 "2022-03-08T15:55:10Z")
> [@ylwu](#): > > Can you explain why you have to restart OpenSearch nodes? Can restart detector realtime job work? If don’t restart nodes, what exception do you see? Restarting the OpenSearch nodes is the only thing that seems to get the detector back up and running correctly. I tried to just restart the detector itself but the detector just gets stuck in an Initializing state even when normal data is flowing in again. I have had detectors stuck in Initializing for hours and hours when trying to stop and restart them. After restarting the node, the detector goes into the Initializing state but will fairly quickly (10-15 minutes) go into a Running state and I can see results coming in for each interval. If it can restart within 20 minutes after a node restart, I would assume it would do the same without having to be restarted. The issue is I’m not seeing any exceptions or particularly useful logging. To clarify there are two issues I see: the first is that the detector appears to stop working properly if it doesn’t receive any data in the index for awhile. There are no errors or exceptions in the API calls/logs that I can see. The second is that once the detector is in that state there does not seem to be any recovery when normal data begins to flow again without a restart of the node itself. ---
**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\ **Post date:** [March 9, 2022, 5:37pm UTC](https://forum.opensearch.org/t/anomaly-detection-for-continuous-ingesting-data/5721/9 "2022-03-09T17:37:20Z")
@zack , that seems some bug. Can you create a Github issue [Issues · opensearch-project/anomaly-detection · GitHub](https://github.com/opensearch-project/anomaly-detection/issues) ? You can add details like which OpenSearch version you are running, how many nodes in your cluster, how many detectors are running, your detector configuration (you can get by [get detector API](https://opensearch.org/docs/latest/monitoring-plugins/ad/api/#get-detector)), what’s your OS version etc.