# Getting different results in vector scores?

**URL:** <https://forum.opensearch.org/t/getting-different-results-in-vector-scores/20228>\
**Category:** k-NN\
**Created:** [July 10, 2024, 2:42pm UTC](https://forum.opensearch.org/t/getting-different-results-in-vector-scores/20228 "2024-07-10T14:42:34Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![dziedrius](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/dziedrius/32/4279_2.png) [@dziedrius](https://forum.opensearch.org/u/dziedrius)\
**Post date:** [July 10, 2024, 2:42pm UTC](https://forum.opensearch.org/t/getting-different-results-in-vector-scores/20228/1 "2024-07-10T14:42:34Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):  
using latest docker image: opensearchproject/opensearch:latest

**Describe the issue** :  
We’re testing vector search and so far not happy with results, wondering what have we done wrong.

I have test document with content:

> “Ebola Virus Disease (EVD) and encourage U.S. hospitals to prepare for managing patients with\r\nEbola and other infectious diseases. Every hospital should ensure that it can detect a patient with\r\nEbola, protect healthcare workers so they can safely care for the patient, and respond in a coordinated fashion. Many of the signs and symptoms of Ebola are non-specific and similar to those of many common”

and use OpenAI embedding model “text-embedding-3-large” to generate embedding vectors.

The problem is, that if I search for not related word, I get very similar score as to word that actually exists in the document:

> “dog Rex”: score 0.5197706  
> “virus”: score 0.5711079

When I do manual cosine equality in C# code, I get quite different values:

> “dog Rex”: score 0.07607428956253202  
> “virus”: score 0.24901766327076158

While not perfect, but that’s ~6 times difference, compared to 10% difference in Open Search.

My manual cosine equality function looks like this:

```auto
double CalculateCosineSimilarity(float[] vector1, float[] vector2)
{
    if (vector1.Length != vector2.Length)
    {
        throw new ArgumentException("Vectors must be of equal length.");
    }

    double dotProduct = vector1.Zip(vector2, (a, b) => a * b).Sum();
    double magnitude1 = Math.Sqrt(vector1.Sum(a => a * a));
    double magnitude2 = Math.Sqrt(vector2.Sum(b => b * b));
        
    return dotProduct / (magnitude1 * magnitude2);
}

```

**Configuration** :

I’ve created index as following (initially tried with ef\_construction 128, later increased to 500):

```auto
PUT /my-index
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "content": {
        "type": "text"
      },
      "content_vector": {
        "type": "knn_vector",
        "dimension": 3072,
        "method": {
          "name": "hnsw",
          "space_type": "cosinesimil",
          "engine": "nmslib",
          "parameters": {
            "ef_construction": 500,
            "m": 16
          }
        }
      }
    }
  }
}

```

post document:

```auto
POST /my-index/_doc/1
{
  "content": "Ebola Virus Disease (EVD) and encourage U.S. hospitals to prepare for managing patients with\r\nEbola and other infectious diseases. Every hospital should ensure that it can detect a patient with\r\nEbola, protect healthcare workers so they can safely care for the patient, and respond in a coordinated fashion. Many of the signs and symptoms of Ebola are non-specific and similar to those of many common",
  "content_vector": [...omitted for brevity...]
}

```

search:

```auto
POST /my-index/_search
{
  "size": 10,
  "query": {
    "knn": {
      "content_vector": {
        "vector": [...omitted for brevity...],
        "k": 10
      }
    }
  }
}

```

**Relevant Logs or Screenshots** :

---

<div class="post-metadata">

**Author:** ![dziedrius](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/dziedrius/32/4279_2.png) [@dziedrius](https://forum.opensearch.org/u/dziedrius)\
**Post date:** [July 15, 2024, 5:28am UTC](https://forum.opensearch.org/t/getting-different-results-in-vector-scores/20228/2 "2024-07-15T05:28:35Z")

</div>

Seems that there’s not too much traffic or the question is too complex 🙂

I did simplify the problem to the minimum - I try to index two simplest vectors - [0,1] and [0,-1]. If cosine similarity is a cosine over angle of vectors, angle is 180 degrees, cosine value should be -1, but I’m getting 0.33:

```auto
POST /my-index2/_search
{
  "size": 10,
  "query": {
    "knn": {
      "content_vector": {
        "vector": [0, 1],
        "k": 10
      }
    }
  }
}

```

result:

```auto
 "hits": [
      {
        "_index": "my-index2",
        "_id": "1",
        "_score": 0.9999999,
        "_source": {
          "content": "up",
          "content_vector": [
            0,
            1
          ]
        }
      },
      {
        "_index": "my-index2",
        "_id": "2",
        "_score": 0.33333334,
        "_source": {
          "content": "down",
          "content_vector": [
            0,
            -1
          ]
        }
      }
    ]

```

so either score is not a cosine similarity or I’m missing something very obvious.

---

<div class="post-metadata">

**Author:** ![Navneet](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/navneet/32/3645_2.png) [@Navneet](https://forum.opensearch.org/u/Navneet)\
**Post date:** [July 16, 2024, 6:08am UTC](https://forum.opensearch.org/t/getting-different-results-in-vector-scores/20228/3 "2024-07-16T06:08:36Z")

</div>

@dziedrius let me try to ans this.

`cosine_similarity = 1 - cos` in Opensearch, you can read about this here: [Approximate k-NN search - OpenSearch Documentation](https://opensearch.org/docs/latest/search-plugins/knn/approximate-knn/#spaces)

Once we have the cosine\_similarity we then convert it to Opesnearch score. Use the same documentation to how know how the scores are generated. In your case of score `0.99999` the way it is getting calculated is

```auto
Query Vector: [0,1]
Document Vector: [0, -1]

cos(theta) = -1/(1x1) = -1

cosine_similarity = 1 - (-1) = 2

Opensearch score = 1 /(1 + cosine_similarity) = 1/3 = 0.33333

```

I hope this clarifies your doubt.

---

<div class="post-metadata">

**Author:** ![dziedrius](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/dziedrius/32/4279_2.png) [@dziedrius](https://forum.opensearch.org/u/dziedrius)\
**Post date:** [July 17, 2024, 11:42am UTC](https://forum.opensearch.org/t/getting-different-results-in-vector-scores/20228/5 "2024-07-17T11:42:53Z")

</div>

@Navneet - thanks that explains the resulting scores.

Where I still struggle - the last step of normalization. It seems that lucene’s approach (2-d/2) would have better dynamic range - [0,1], while nmslib - [0.333, 1], hence wondering why they chose their approach?

Why dynamic range could be important - seems that at least some of the embedding models have limited range (discussion about it here: [Some questions about text-embedding-ada-002’s embedding - API - OpenAI Developer Forum](https://community.openai.com/t/some-questions-about-text-embedding-ada-002-s-embedding/35299))

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/3f414a0bd4374bb7c7b8435b046f3ad230516ec1.png)

---

<div class="post-metadata">

**Author:** ![Navneet](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/navneet/32/3645_2.png) [@Navneet](https://forum.opensearch.org/u/Navneet)\
**Post date:** [July 24, 2024, 8:08am UTC](https://forum.opensearch.org/t/getting-different-results-in-vector-scores/20228/6 "2024-07-24T08:08:02Z")

</div>

This a really good question, but I don’t have a good ans for you on this. @jmazane what is your thought on the last question.

---

<div class="post-metadata">

**Author:** ![jmazane](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jmazane](https://forum.opensearch.org/u/jmazane)\
**Post date:** [July 24, 2024, 2:41pm UTC](https://forum.opensearch.org/t/getting-different-results-in-vector-scores/20228/7 "2024-07-24T14:41:29Z")

</div>

Hi @dziedrius and @Navneet,

Thats an interesting point and interesting thread. Long ago I think we added and it was simple enough to follow what l2 was doing (see [cosine PR](https://github.com/opendistro-for-elasticsearch/k-NN/pull/90) and [spot where conversion happens](https://github.com/opendistro-for-elasticsearch/k-NN/blob/opendistro-1.3/src/main/java/com/amazon/opendistroforelasticsearch/knn/index/KNNWeight.java#L137)). But, the normalization of (1+cos) does seem like it would preserve more precision than 1/(1+cos). Not sure why they divide by 2 after that in lucene - seems for ordering sake (1+cos) would work well, it just gives range of [0, 2]. Oddly enough, we do this for script scoring. See [here](https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/plugin/script/KNNScoringSpace.java#L149).

That being said, I am not sure what practical consequence of this would be - i.e. when it would become noticeable or how close 2 values would need to be. The other thing is that for prod systems, we recommend for cosine vectors, to normalize vectors as pre-processing step and then using the innerproduct space, which will have same result of cosine, but will be much faster during querying.

Jack

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [September 22, 2024, 2:41pm UTC](https://forum.opensearch.org/t/getting-different-results-in-vector-scores/20228/8 "2024-09-22T14:41:35Z")

</div>

Closed on September 22, 2024, 2:41pm UTC

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.