# Anomaly detection - rolling forecasts + index data structure

**URL:** <https://forum.opensearch.org/t/anomaly-detection-rolling-forecasts-index-data-structure/4481>\
**Category:** Machine Learning\
**Created:** [December 26, 2020, 9:58am UTC](https://forum.opensearch.org/t/anomaly-detection-rolling-forecasts-index-data-structure/4481 "2020-12-26T09:58:28Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![javaguy44](https://avatars.discourse-cdn.com/v4/letter/j/45deac/32.png) [@javaguy44](https://forum.opensearch.org/u/javaguy44)\
**Post date:** [December 26, 2020, 9:58am UTC](https://forum.opensearch.org/t/anomaly-detection-rolling-forecasts-index-data-structure/4481/1 "2020-12-26T09:58:28Z")

</div>

Hi,

I have a rolling forecast timeseries prediction being computed 3x per day and I’m wondering how I should structure my data / index in the ES index so that the anomaly detection can find errors?

Currently the data is structured in a way so that there’s an ‘observation datetime’ which is when the rolling forecast was computed being associated with a timeseries of datetime, value points which represent the forecasted timeseries itself.

ie one rolling forecast would be:  
observation\_datetime = ‘2020-12-26 10:00’,  
forecast\_timeseries = {  
2020-12-26 10:30, 2.34,  
2020-12-26 11:00, 4.34,  
2020-12-26 11:30, 4.44,  
2020-12-26 12:00, 4.44,  
2020-12-26 12:30, 4.44,  
2020-12-26 13:00, 4.44,  
2020-12-26 13:30, 2.34,  
2020-12-26 14:00, 4.34,  
2020-12-26 14:30, 4.44,  
continuing for 5 years  
}

and a second forecast later today will be  
observation\_datetime = ‘2020-12-26 11:00’,  
forecast\_timeseries = {  
2020-12-26 11:30, 4.44,  
2020-12-26 12:00, 4.44,  
2020-12-26 12:30, 4.44,  
2020-12-26 13:00, 4.44,  
2020-12-26 13:30, 2.34,  
2020-12-26 14:00, 4.34,  
2020-12-26 14:30, 4.44,  
continuing for 5 years  
}

etc.

I would like to use the anomaly detection feature to compare the forecast\_timeseries themselves to each other to detect if there any errors ie data gaps, outliers in values etc.

Any suggestions on how I should structure my ES index of my rolling forecast data so that I can take advantage of the OpenDistro anomaly detection feature for this dataset above?

Thx

---

<div class="post-metadata">

**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\
**Post date:** [January 5, 2021, 8:14pm UTC](https://forum.opensearch.org/t/anomaly-detection-rolling-forecasts-index-data-structure/4481/2 "2021-01-05T20:14:30Z")

</div>

hi, @javaguy44, thanks for your interest in ODFE anomaly detection plugin. Just clarify more about your requirements, do you want to detect anomalies in one forecast or anomalies/big gaps in second forecast compared with first forecast?

---

<div class="post-metadata">

**Author:** ![javaguy44](https://avatars.discourse-cdn.com/v4/letter/j/45deac/32.png) [@javaguy44](https://forum.opensearch.org/u/javaguy44)\
**Post date:** [January 9, 2021, 10:37am UTC](https://forum.opensearch.org/t/anomaly-detection-rolling-forecasts-index-data-structure/4481/3 "2021-01-09T10:37:31Z")

</div>

Hi @ylwu thanks for your reply - sorry I missed this and Happy New Year to you!

I am interested in the anomalies / big gaps in the second forecast compared with the first forecast!

Thanks!

---

<div class="post-metadata">

**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\
**Post date:** [January 19, 2021, 6:44pm UTC](https://forum.opensearch.org/t/anomaly-detection-rolling-forecasts-index-data-structure/4481/4 "2021-01-19T18:44:34Z")

</div>

Cool, this is an interesting feature. We don’t support this currently, but we have plan to build some trend analysis which may address this requirements. Keep an eye on our Github release [Releases · opendistro-for-elasticsearch/anomaly-detection · GitHub](https://github.com/opendistro-for-elasticsearch/anomaly-detection/releases)

---

<div class="post-metadata">

**Author:** ![jdbright](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/jdbright/32/4979_2.png) [@jdbright](https://forum.opensearch.org/u/jdbright)\
**Post date:** [July 19, 2022, 3:58am UTC](https://forum.opensearch.org/t/anomaly-detection-rolling-forecasts-index-data-structure/4481/5 "2022-07-19T03:58:23Z")

</div>

For those who are tracking the rolling forecasts thread here, I’ve opened up a [Github meta issue](https://github.com/opensearch-project/anomaly-detection/issues/615) to track the progress. We are eager to hear about use cases the community is interested in solving for.