Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.7 (in AWS)
Describe the issue:
I need to request date histogram from OpenSearch (running in AWS) over multiple (rollup) indices. For some reason it returns data only for the last index only. In the documention they say that it supports aggregation over multiple rollup indices, but nothing about date histogram. If I use fixed_interval=1d it returns data from the last index only, but if I increase interval (2 days for example) then it starts to see the last part of the previous index. We use OpenSearch 2.7 in AWS. I tried to use alias, template, list of indices.
I have two rollup indices:
rollup-2023-8
rollup-2023-9
They have the same mapping and fields: timestamp and several keyword fields.
{
“_meta”: {
“rollups”: {
“rollup-2023-8”: {
“enabled_time”: 1692713967590,
“target_index”: “rollup-2023-8”,
“description”: “Rollup job for source index 2023-8”,
“source_index”: “snapshot-2023-8”,
“enabled”: true,
“rollup_id”: “rollup-2023-8”,
“schema_version”: 17,
“schedule”: {
“interval”: {
“schedule_delay”: 0,
“start_time”: 1692713967590,
“period”: 1,
“unit”: “Hours”
}
},
“delay”: null,
“last_updated_time”: 1692713967590,
“continuous”: true,
“metadata_id”: “jOnUHYoBo8PNreDGT1gP”,
“metrics”: ,
“user”: null,
“page_size”: 1000,
“dimensions”: [
{
“date_histogram”: {
“fixed_interval”: “1d”,
“source_field”: “timestamp”,
“target_field”: “timestamp”,
“timezone”: “UTC”
}
},
{
“terms”: {
“source_field”: “field1”,
“target_field”: “field1”
}
},
{
“terms”: {
“source_field”: “field2”,
“target_field”: “field2”
}
}
]
}
}
},
“dynamic_templates”: [
{
“strings”: {
“match_mapping_type”: “string”,
“mapping”: {
“type”: “keyword”
}
}
},
{
“date_histograms”: {
“path_match”: “*.date_histogram”,
“mapping”: {
“type”: “date”
}
}
}
],
“properties”: {}
}
The following request:
GET rollup-2023-/_search
{
“size”: 0,
“query”: {
“bool”: {
“filter”: [
{
“term”: {
“field2”: {
“value”: “value”,
“boost”: 1
}
}
},
{
“bool”: {
“filter”: [
{
“term”: {
“field1”: {
“value”: “value”,
“boost”: 1
}
}
}
],
“adjust_pure_negative”: true,
“boost”: 1
}
}
],
“adjust_pure_negative”: true,
“boost”: 1
}
},
“_source”: {
“includes”: [
""
],
“excludes”: [
“copy_of_id”
]
},
“track_total_hits”: 2147483647,
“aggregations”: {
“timestamp”: {
“date_histogram”: {
“field”: “timestamp”,
“fixed_interval”: “1d”,
“offset”: 0,
“order”: {
“_key”: “asc”
},
“keyed”: false,
“min_doc_count”: 0
},
“aggregations”: {
“field2”: {
“terms”: {
“field”: “field2”,
“size”: 10,
“min_doc_count”: 1,
“shard_min_doc_count”: 0,
“show_term_doc_count_error”: false,
“order”: [
{
“_count”: “desc”
},
{
“_key”: “asc”
}
]
}
}
}
}
}
}
Then response starts from the 2023-9, and doesn’t include data from index for date 2023-8.
Could you help with it?