Searching multiple rollup indexes not working

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
“version”: {
“distribution”: “opensearch”,
“number”: “2.4.0”,
“build_type”: “tar”,
“build_hash”: “744ca260b892d119be8164f48d92b8810bd7801c”,
“build_date”: “2022-11-15T04:42:29.671309257Z”,
“build_snapshot”: false,
“lucene_version”: “9.4.1”,
“minimum_wire_compatibility_version”: “7.10.0”,
“minimum_index_compatibility_version”: “7.0.0”
}

Describe the issue:
Followed the example provided at Index rollups - OpenSearch Documentation

I do see Rollups for multiple indexes being created (rollup_ndx-metric-000001, rollup_ndx-metric-000002, rollup_ndx-metric-000003) and I am able to query them individually. But when I try to search multiple Rollups via the following way, it does not seem to be using all the rolled up indexes to get the results as I am not seeing the results as expected.

GET /rollup_ndx-metric-*/_search

{
    "size": 0,
    "query": {
        "match_all": {}
    },
    "aggs": {
        "max": {
            "max": {
                "field": "value"
            }
        },
        "min": {
            "min": {
                "field": "value"
            }
        },
        "avg": {
            "avg": {
                "field": "value"
            }
        },
        "sum": {
            "sum": {
                "field": "value"
            }
        }
    }
}

But if I remove the match all query, I see the expected results. Seems like there is an issue with using query.

    "query": {
        "match_all": {}
    },
4

This is a show stopper as rollup feature is unusable when we want to search/filter data via query. Am I missing something?

5

@akkina9 Your version 2.4.0 is quite old. Have you tested the latest one?

According to 2.5.0 release notes, the query string support in index rollups was implemented in that version.

6

@pablo Thats good to know. Can you point me to the release notes that talk about this? Thanks

7

Nevermind. I found it. opensearch-build/release-notes/opensearch-release-notes-2.5.0.md at main · opensearch-project/opensearch-build · GitHub

8

@pablo I upgraded to 2.11 and ran the query and it has the same behavior. Maybe I need to create all the rollup indexes again. I will try and let you know.

9

@pablo I tried it from scratch using 2.11 and I see that same issue, i.e. GET /rollup_ndx-metric-*/_search is not using all rolled up indices.

10

There is a setting: plugins.rollup.search.search_all_jobs which enables multiple rolled up indices to be searched together. Default value of this is false, It is looking better now…

Reference: [BUG] Searching multiple rollup indexes not working as expected · Issue #1265 · opensearch-project/index-management · GitHub

1 Like
11

@akkina9 Thanks for sharing the solution.