Extraction query response limited to 10000 hits

Plugins Alerting
1

I experienced that the query results of the extraction queries are limited to 10000 hits, which make sense if a Monitor is designed to trigger an alarm when result represents an error state:

    "hits": {
        "hits": [],
        "total": {
            "value": 10000,
            "relation": "gte"
        },

"size": 0 doesn’t have any effect
setting the size manually above 10000 doesn’t work either

I might have a rather different use case, I need to monitor daily imports and check if the right amount of records have been loaded, for that the lowest threshold I’d need starts at 1mio hits.
This means my Monitor is designed to alarm based on results above 10000hits.

Is there a way to configure the limits of the hits, or for this particular use case it might be better to access the _count or _stats API in order to get a count of a daily index , rather that count based on querying records.

thanks

2

Hi @nean,

This was a breaking change in Elasticsearch 7.x which you can read about here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#track-total-hits-10000-default

From the documentation:

You can force the count to always be accurate by setting track_total_hits to true explicitly in the search request.

3

Hi Drew,

confirm this works with "track_total_hits" : true, which is being automatically replaced with "track_total_hits": 2147483647,after first run.

Can this be implemented as default and especially into the Define using visual graph monitor define mode?

Define extraction query:

{
    "size": 0,
    "track_total_hits" : true,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "{{period_end}}||-2d",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {}
}

Result:

{
    "_shards": {
        "total": 374,
        "failed": 0,
        "successful": 374,
        "skipped": 372
    },
    "hits": {
        "hits": [],
        "total": {
            "value": 1548965,
            "relation": "eq"
        },
        "max_score": null
    },
    "took": 14,
    "timed_out": false
}
1 Like
4

Hi @nean,

Have made a GitHub issue here to track this:

1 Like