Losing top documents when query reaches the `terminate_after` limit

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch/Dashboard = 2.9.0
OS = Windows 11

Describe the issue:
I have indexed some documents containing nested documents and I perform queries with the .NET OpenSearch.Client to retrieve the results sorted by a document field.

What I notice is that, when the results are many, I don’t get back the most recent results (the ones that according to my sort should be at the top).
From what I can see, this is related to the terminate_after setting.
It appears to happen whenever I hit this limit, either after I set the limit in my query, or if I don’t set it and I hit some default internal limit of 10000 documents.

I have tried various things, like :

  • Trying to change my sort field/order of my index, and make it align with my query sort, which was not possible since my documents contained nested documents.
  • Trying to use the .TrackTotalHits() instead of the .TerminateAfter() method, but it didn’t help.
  • Trying to set a document score dependent on the sort field, using the .FieldValueFactor() method. But that created other complications, that required me to restructure my query.

Does anyone have any idea why this happens and how to fix it?
I am totally clueless.
I don’t know if the last option (with the FieldValueFactor) could work, but I was hoping there is a better option, because what I have seen is that my filtering stopped working when I used it.

BTW, this is my code:


            var response = openSearchClient.Search<Document>(s => s
                .Index(documentsIndexName)
                .Query(q => q
                    //.FunctionScore(fs => fs
                    //    .Functions(fu => fu
                    //        .FieldValueFactor(fvq => fvq
                    //            .Field(f => f.IdDocument)
                    //            .Factor(4)
                    //            .Modifier(FieldValueFactorModifier.None)
                    //            .Filter(ff => ff.Bool(bq => bq
                    //                  //////// <------ When I move the contents of the Bool query below in here, filtering stops working.
                    //            ))
                    //        )
                    //    )
                    //)
                    .Bool(bq => bq
                        .Must(mq =>
                        {
                            var query = GetUserAppliedFilters(mq );
                            return query;
                        })
                        .Filter(mq => {
                            var query = GetGroupAccessFilters(mq ) && GetUserAccessFilters(mq );
                            return query;
                        })
                        )
                    )
                    .Sort(sort => {
                        // Some sort logic here....
                    })
                    .TerminateAfter(maxNumberOfResultsToReturn)
                    //.TrackTotalHits(new TrackTotalHits(maxNumberOfResultsToReturn))
                    .TrackScores(false)
                    .From(request.CurrentPage * request.PageSize)
                    .Size(request.PageSize)
                    //.Explain(true)
                    .Source(false)
                );

Configuration:

Relevant Logs or Screenshots:

2

I thought about index sorting, too - maybe there’s a way you can do without nested documents?

3

Unfortunately, no.
I really need to use nested documents, and I need to query their fields using AND operators.

4

It turns out that using this works:

  //.TerminateAfter(maxNumberOfResultsToReturn) // Remove this!
  .TrackTotalHits(new TrackTotalHits(maxNumberOfResultsToReturn))

I had tried this in the past and didn’t seem to work, but it does now.
Some weird build issue, most likely…

Update:

Actually, no. This was still missing results. This did help, but it is a bit horrible:

  .TrackTotalHits(new TrackTotalHits(1000000000/*a huge value here*/))