How to retrieve more than 10000 items

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard v 2.16/Server OS/Browser):

Describe the issue:

I would like to retrieve the documents generated for 1yr. There are more than 5 million items. I know I can export them as a csv file but only for 10K items. How can I retrieve all of the items?

Thanks in advance.

Configuration:

Relevant Logs or Screenshots:

2

Hi @nevermet ,

This limit can be increased by bumping the max result window for your search.

PUT your-index/_settings
{
  "index.max_result_window": 5000000
}

Then I would suggest writing a script to pull all logs in bathes and generate a CSV file with all logs.

Leeroy.

3

Dear @Leeroy ,

Thank you for your answer.

Can I just run the PUT command in the Dev Tools page?

Also could you explain more how to write the script to download all logs, please?

Thank you.

4

This is highly discouraged - this configuration is there for a reason.

Rather, @nevermet you should use Scroll / PIT , see here for more information: OpenSearch & Elasticsearch Scroll and Deep Paging Methods Compared - BigData Boutique Blog . In some cases search_after can be a solid, more efficient option.