High Latency in KNN queries in v3.1

OpenSearch
,
1

Versions - OpenSearch v2.19 and v3.1 (AWS managed cluster)

Describe the issue:

We migrated from OpenSearch v2.19 to v3.1 and changed from single-shard-per-tenant to multi-shard-per-tenant. Earlier each query hit one shard; now it hits multiple shards (scatter-gather).
The query is should with the KNN query and a has_parent join in the filter.

For the same query, shard 2 finishes in under 3s while shard 9 takes 15–18s.

We set index.knn.advanced.filtered_exact_search_threshold to -1 to disable exact search fallback, but the latency difference remains. Does setting this to -1 fully prevent exact search in all cases, or can exact search still run internally?

We saw issue #2249 mentioning NativeEngineKnnVectorQuery performing an exact search in create_weight for nested fields. In 3.1, is this expected even when filtered_exact_search_threshold is disabled and also for joins?

Could the 3.1 native engine changes (compared to 2.19 plugin behavior) explain why only one shard is very slow and other one is also slow comparatively where earlier it used to take under 200 ms?

Configuration:

OpenSearch 3.1 on AWS managed cluster
Faiss engine for k-NN
Nested field with parent-child relationship
Multi-shard per tenant
index.knn.advanced.filtered_exact_search_threshold set to -1

Relevant Logs or Screenshots:

Query profile shows shard 2 completing in under 3 seconds while shard 9 takes 15–18 seconds, with most of the time spent in create_weight / NativeEngineKnnVectorQuery execution.

https:// docs.opensearch. org/latest/vector-search/filter-search-knn/efficient-knn-filtering/
https:// github. com/opensearch-project/k-NN/issues/2936
profile 1 - Context – share whatever you see with others in seconds
profile 2 - Context – share whatever you see with others in seconds

2

The query is of form in both cases

{
 "docvalue_fields": [
  "child_object_source_id"
 ],
 "query": {
  "knn": {
   "knn-field-name": {
    "filter": {
     "bool": {
      "filter": [
       {
        "has_parent": {
         "parent_type": "parent",
         "query": {
          "bool": {
           "filter": [
            {
             "bool": {
              "minimum_should_match": 1,
              "should": [
               {
                "term": {
                 "tenant_id": "tenant-id"
                }
               },
               ... some more term filters
              ]
             }
            },
            {
             "bool": {
              "filter": [
               {
                "bool": {
                 "minimum_should_match": 1,
                 "should": [
                  {
                   "term": {
                    "object_type": {
                     "value": "parent-object-type"
                    }
                   }
                  }
                 ]
                }
               }
              ]
             }
            }
           ]
          }
         }
        }
       },
       {
        "bool": {
         "filter": [
          {
           "bool": {
            "minimum_should_match": 1,
            "should": [
             {
              "term": {
               "object_type": {
                "value": "child-object-type"
               }
              }
             }
            ]
           }
          }
         ]
        }
       },
       {
        "term": {
         "child_source_object_type": {
          "value": "parent-object-type"
         }
        }
       },
       {
        "term": {
         "object_type": {
          "value": "child-object-type"
         }
        }
       }
      ]
     }
    },
    "k": 10,
    "rescore": {
     "oversample_factor": 3
    },
    "vector": [...]
   }
  }
 },
 "sort": []
}