# \[BUG\] Insufficient number of hits for nested knn queries with efficient filter #2347

**URL:** <https://forum.opensearch.org/t/bug-insufficient-number-of-hits-for-nested-knn-queries-with-efficient-filter-2347/22867>\
**Category:** k-NN\
**Created:** [December 20, 2024, 3:24pm UTC](https://forum.opensearch.org/t/bug-insufficient-number-of-hits-for-nested-knn-queries-with-efficient-filter-2347/22867 "2024-12-20T15:24:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![corentin](https://avatars.discourse-cdn.com/v4/letter/c/919ad9/32.png) [@corentin](https://forum.opensearch.org/u/corentin)\
**Post date:** [December 20, 2024, 3:24pm UTC](https://forum.opensearch.org/t/bug-insufficient-number-of-hits-for-nested-knn-queries-with-efficient-filter-2347/22867/1 "2024-12-20T15:24:15Z")

</div>

Hello 👋

**What is the bug?**

We use an index to store text documents for semantic search purpose. The text being long, we chunk it in paragraph to embed it using `all-MiniLM-L6-v2` model. Each chunk being stored in that nested field of the document.  
Each document has also an account\_id attribute that we use when querying (efficient filtering).

Then we do approximative knn queries with lucene hnsw.

From these documentations :

- [k-NN search with nested fields](https://opensearch.org/docs/latest/search-plugins/knn/nested-search-knn/)
- [k-NN search with filters](https://opensearch.org/docs/latest/search-plugins/knn/filter-search-knn/)
- and mosty [Enhanced multi-vector support for OpenSearch k-NN search with nested fields](https://opensearch.org/blog/enhanced-multi-vector-support-in-opensearch-knn/)

I expect when executing a knn query on this nested field with efficient filter to get at least n hits, n being the minimum between k and the number of documents that match the efficient filter.

But for some specific input `vector` or `query_text`, we get less than n hits, and sometimes even 0. For the same filter with a different query, we get the correct n hits.

We have two other indices without nested field (only one vector per document) with the same efficient filter and it works as expected.

Seems similar to this [[BUG] Filter on Parent Doc fields inside Nested knn query fails for many Query types · Issue #2222 · opensearch-project/k-NN · GitHub](https://github.com/opensearch-project/k-NN/issues/2222) or [[BUG] OpenSearch 2.17 K-NN efficient filtering with a Date Range Filter No Results · Issue #2339 · opensearch-project/k-NN · GitHub](https://github.com/opensearch-project/k-NN/issues/2339) except the efficient filtering is as simple as a term filter.

**How can one reproduce the bug?**

Error happens on specific queries so it’s hard to reproduce.

Here is the mapping of the index :

```json
{
  "knowledge-index": {
    "mappings": {
      "properties": {
        "accountId": {
          "type": "keyword"
        },
        "id": {
          "type": "keyword"
        },
        "metadata": {
          "type": "text"
        },
        "metadataEmbedding": {
          "type": "nested",
          "properties": {
            "knn": {
              "type": "knn_vector",
              "dimension": 384,
              "method": {
                "engine": "lucene",
                "space_type": "l2",
                "name": "hnsw",
                "parameters": {}
              }
            }
          }
        },
        "timestamp": {
          "type": "date"
        }
      }
    }
  }
}

```

Here is the query :

```json
GET /knowledge-index/_search?preference=_primary&explain=true&request_cache=false
{
  "from": 0,
  "_source": {
    "excludes": [
      "metadataEmbedding"
    ]
  },
  "query": {
    "nested": {
      "score_mode": "max",
      "path": "metadataEmbedding",
      "query": {
        "neural": {
          "metadataEmbedding.knn": {
            "query_text": "<query_text>",
            "model_id": "9QxR8YsBSCN1wquQEH2b",
            "k": <k>,
            "filter": {
                "term": {
                  "accountId": "<account_id>"
              }
            }
          }
        }
      }
    }
  }
}

```

For k = 38, I get 6 hits

```auto
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 0.50342417,

```

But for k = 1000 I get 32 hits, and k = 10000 (max value) 232 hits.

For another `query_text` value, I have different results where hits is always = k (or the max of documents that match filter which is 232)

I have the same results when converting first the text in vector and use directly the vector without the neural instruction :

```auto
POST /_plugins/_ml/_predict/text_embedding/9QxR8YsBSCN1wquQEH2b
{
  "text_docs":["<query_text>"],
  "return_number": true,
  "target_response": ["sentence_embedding"]
}

GET /knowledge-index/_search?preference=_primary&explain=true&request_cache=false
{
  "size": 5,
  "_source": {
    "excludes": [
      "metadataEmbedding"
    ]
  },
  "query": {
    "nested": {
      "path": "metadataEmbedding",
      "query": {
        "knn": {
          "metadataEmbedding.knn": {
            "vector": [
                 ....
             ],
            "k": 38,
            "filter": {
              "term": {
                "accountId": "<account_id>"
              }
            }
          }
        }
      }
    }
  }
}

```

**What is the expected behavior?**

Getting n hits, n being the minimum between k and the number of documents that match the efficient filter.

**What is your host/environment?**

- opensearch version : 2.17.1

Any idea on what could be the issue here ? Am I right to expect k hits for nested fields with efficient filter ?

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![Navneet](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/navneet/32/3645_2.png) [@Navneet](https://forum.opensearch.org/u/Navneet)\
**Post date:** [December 20, 2024, 4:53pm UTC](https://forum.opensearch.org/t/bug-insufficient-number-of-hits-for-nested-knn-queries-with-efficient-filter-2347/22867/2 "2024-12-20T16:53:21Z")

</div>

@corentin I can see in some of your queries the k and size value is different. Did you do a test where your K and size have exactly same value?

---

<div class="post-metadata">

**Author:** ![corentin](https://avatars.discourse-cdn.com/v4/letter/c/919ad9/32.png) [@corentin](https://forum.opensearch.org/u/corentin)\
**Post date:** [December 20, 2024, 5:17pm UTC](https://forum.opensearch.org/t/bug-insufficient-number-of-hits-for-nested-knn-queries-with-efficient-filter-2347/22867/3 "2024-12-20T17:17:03Z")

</div>

Thanks @Navneet ! I just did and get the same issue. Usually, in my understanding the number of hits should be k (whatever size is) and the number of returned documents should be size. But I have the same issue when k = size.

Thanks for your help 🙏

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [February 18, 2025, 5:17pm UTC](https://forum.opensearch.org/t/bug-insufficient-number-of-hits-for-nested-knn-queries-with-efficient-filter-2347/22867/4 "2025-02-18T17:17:53Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.