Opensearch fetch restriction to 10K records

Hi Team,

I came across below error while running the opensearch sql plugin query which shows there is restriction on maximum records (10K) to be fetched. So I have updated index.max_result_window and fetch more records but wanted to know is there any specific reason to keep this restriction and also increasing this does we need take care something like infrastructure, any settings?

{
“error”: {
“root_cause”: [
{
“type”: “illegal_argument_exception”,
“reason”: “Result window is too large, from + size must be less than or equal to: [10000] but was [11000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.”
}
],
“type”: “search_phase_execution_exception”,
“reason”: “all shards failed”,
“phase”: “dfs”,
“grouped”: true,
“failed_shards”: [
{
“shard”: 0,
“index”: “performance_data_logs”,
“node”: “fdBqWfY5S76pSjnqu_hcnw”,
“reason”: {
“type”: “illegal_argument_exception”,
“reason”: “Result window is too large, from + size must be less than or equal to: [10000] but was [11000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.”
}
}
],
“caused_by”: {
“type”: “illegal_argument_exception”,
“reason”: “Result window is too large, from + size must be less than or equal to: [10000] but was [11000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.”,
“caused_by”: {
“type”: “illegal_argument_exception”,
“reason”: “Result window is too large, from + size must be less than or equal to: [10000] but was [11000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.”
}
}
},
“status”: 400
}

Regards

The 10K default limit is to limit the amount of heap used. The more records you get in one go, the more heap that particular request will use. If you want to get 1M records, it might be problematic.

The usual solution is to use a scroll, but I’m not familiar enough with the SQL plugin to know if it can use that.

If you increase the value, watch your heap usage. You might need to add more heap or more nodes if it looks like you’re running out of heap or if you’re getting an OOM.

Thanks @radu.gheorghe for quick reply.

You’re welcome, @purulalwani !