Deep pagination with jump to random page using search_after?


I am working on a website, where I need to search through an index with millions of documents, and be able to display them as pages of 500 documents. With pagination we need to support the jump to a random page.

I looked at Elastic Search documentation and it says we should use search_after for deep pagination. Documentation till 7.2 also says

search_after is not a solution to jump freely to a random page but rather to scroll many queries in parallel. It is very similar to the scroll API but unlike it, the search_after parameter is stateless, it is always resolved against the latest version of the searcher. For this reason the sort order may change during a walk depending on the updates and deletes of your index.

Reference: Search After | Elasticsearch Guide [7.2] | Elastic

Does that mean Elastic Search from version 7.3 onwards support jump to random page with search_after ?

Latest version’s documentation also says that scroll is not recommended.

Then what options do I have for deep pagination along with jump to random page?

Thanks and regards,

Deep, random pagination is a pretty classic problem. scroll is compounded by having to keep the state around and to manage that. And search_after isn’t a silver bullet, I don’t think that after 7.2 search_after changed that much, what did change is that 7.10 added Point In Time (PIT) in X-Pack (not Open Distro!) that made search_after more sane (search_after suffers badly in data sets with moving data).

What is your pagination sort context? Can you integrate that into your query? Have you considered aggregations to paginate?

Seems opensearch also support search_after and pit. :slight_smile: wonderful

I have an article on how I handled pagination with support to jump to a random page -

Please do take a look and provide feedback if anything can be improved

Swaroop Gupta