Row Count limit hit - which rows returned?

OpenSearch
1

**Versions OpenSearch Discover 2.7 AWS Chrome

I prepare and run a query against an index. I get back 10000 rows/documents. OK I can live with the limit as I understand the implications.
But which rows are returned? If my filter/query would have returned 1mm records, but I only get back 10K, are they the first ones? Sorted how? It looks like a random sampling amidst the time interval.

Thanks for any help
Dean

Configuration:

Relevant Logs or Screenshots:

2

For clarification, I am specifically asking a query with 1mm documents returned, but then with a CSV report export, I get 10000 rows. I know why the report is limited to 10K. But which rows are returned? First 10K in order of matcvh score? Date? random?
Thanks
Dean

3

@dangelic0 The search result in OpenSearch is sorted by relevance score " _score".
Why would you need to return all the documents? You should narrow your results by building efficient queries.

4

I don’t really need all those rows. I need to perform aggregation across a large set of rows/documents. Like a bucket aggregation, or a cardinality aggregation. But I found that even the aggregations only accept 10000 rows from the query.

So I was going to do the aggregation outside of OpenSearch by exporting the rows. So wondering which rows are returned when limited by the 10K limit.

How can I do an aggregation on 1mm rows? I see the doc count in the aggregation set to 10K…

5

@dangelic0 The query will process all the documents in the index. The limitation regards the output.

If you’d like to increase the 10k cap, then modify the index.max_result_window.

6

Excellent. Thanks very much. This solved it.
Dean

1 Like