# kNN (nmslib) returns a fewer results than expected

**URL:** <https://forum.opensearch.org/t/knn-nmslib-returns-a-fewer-results-than-expected/20005>\
**Category:** k-NN\
**Tags:** troubleshoot\
**Created:** [June 26, 2024, 5:26am UTC](https://forum.opensearch.org/t/knn-nmslib-returns-a-fewer-results-than-expected/20005 "2024-06-26T05:26:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![neko](https://avatars.discourse-cdn.com/v4/letter/n/7ab992/32.png) [@neko](https://forum.opensearch.org/u/neko)\
**Post date:** [June 26, 2024, 5:26am UTC](https://forum.opensearch.org/t/knn-nmslib-returns-a-fewer-results-than-expected/20005/1 "2024-06-26T05:26:15Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):  
OpenSearch 2.13

**Describe the issue** :  
Approximate kNN returns only a few hits than expected. I’m wondering what’s wrong with my configuration or my understanding.

The below query is to get all results without `kNN` query.

```js
{
      query: {
        bool: {
          must: [
          ],
          filter: [
            {
              term: {
                some_field: {
                  value: some_field_value
                },
              },              
            },
            {
              term :{
                another_field: {
                  value: another_field_value,
                },
              },
            },
          ]
        }
      },
      size: 100
    }

```

returns

```auto
{
  took: 37,
  timed_out: false,
  _shards: { total: 5, successful: 5, skipped: 0, failed: 0 },
  hits: {
    total: { value: 66, relation: 'eq' },

```

results, which is as expected. However, the `kNN` query only returns much fewer hits, even though `k` and `size` should be big enough.

```js
{
      query: {
        bool: {
          must: [
            {
              knn: {
                embedding: {
                  vector: query_embedding,
                  k: 100,
                }
              }
            },
          ],
          filter: [
            {
              term: {
                some_field: {
                  value: some_field_value
                },
              },              
            },
            {
              term :{
                another_field: {
                  value: another_field_value,
                },
              },
            },
          ]
        }
      },
      size: 100
    }

```

returns only

```auto
{
  took: 12254,
  timed_out: false,
  _shards: { total: 5, successful: 5, skipped: 0, failed: 0 },
  hits: {
    total: { value: 9, relation: 'eq' },

```

. I was expecting it to return all results. I am trying to understand and fix this situation. Currently, my kNN’s recall seems low and have no idea why. I checked missing documents and confirmed it has `embedding`. Thank you for your great job!

**Configuration** :  
Index name  
my-rag-chunks

Health

Green

Status  
Open

Creation date  
6/23/2024, 7:51:10 PM

Total size  
33.2gb

Size of primaries  
16.5gb

Total documents  
635298

Deleted documents  
57353

Primaries  
5

Replicas  
1

About mapping,

```auto
    "embedding": {
      "dimension": 1536,
      "method": {
        "engine": "nmslib",
        "space_type": "innerproduct",
        "name": "hnsw",
        "parameters": {}
      },
      "type": "knn_vector"
    },

```

**Relevant Logs or Screenshots** :

---

<div class="post-metadata">

**Author:** ![neko](https://avatars.discourse-cdn.com/v4/letter/n/7ab992/32.png) [@neko](https://forum.opensearch.org/u/neko)\
**Post date:** [June 27, 2024, 2:42am UTC](https://forum.opensearch.org/t/knn-nmslib-returns-a-fewer-results-than-expected/20005/2 "2024-06-27T02:42:21Z")

</div>

I believe it is because an approximate kNN’s graph is skewed when filtered. Now I’m using kNN’s Scoring script filter, which is calculate kNN exactly.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [August 26, 2024, 2:43am UTC](https://forum.opensearch.org/t/knn-nmslib-returns-a-fewer-results-than-expected/20005/3 "2024-08-26T02:43:07Z")

</div>

Closed on August 26, 2024, 2:43am UTC

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.