# Cosine Similarity Formula

**URL:** <https://forum.opensearch.org/t/cosine-similarity-formula/4390>\
**Category:** k-NN\
**Created:** [December 15, 2020, 8:02am UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390 "2020-12-15T08:02:46Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![doc113](https://avatars.discourse-cdn.com/v4/letter/d/8797f3/32.png) [@doc113](https://forum.opensearch.org/u/doc113)\
**Post date:** [December 15, 2020, 8:02am UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/1 "2020-12-15T08:02:46Z")

</div>

Hi! I need help understanding how the CosineSimilarity is calculated.

For example: I created knn-index with parameters ‘knn’: True, ‘knn.space\_type’: ‘cosinesimil’, ‘type’: ‘knn\_vector’, ‘dimension’: 3 (as in example how to use knn). Then i added two vectors: [1, 2, 3] and [3, 4, 6]

Now I’m trying to compare [1, 2, 3] with vector [3, 4, 5]:  
open-distro result: 0.22400923,  
manually cosine = (1_3+2_4+3\*5)/(sqrt(14)\*sqrt(50)) = 0.9827076298239908

compare [3, 4, 6] with vector [3, 4, 5]:  
open-distro result: 0.5,  
manually cosine = (3_3+4_4+6\*5)/(sqrt(61)\*sqrt(50)) = 0.9958932064677039

how calculation works in open-distro? Maybe you can help me, please.  
P.S. Sorry for my English

---

<div class="post-metadata">

**Author:** ![puzzled](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@puzzled](https://forum.opensearch.org/u/puzzled)\
**Post date:** [December 15, 2020, 7:19pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/2 "2020-12-15T19:19:47Z")

</div>

I’ve just signed up to ask similar question!

---

<div class="post-metadata">

**Author:** ![jmazane](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jmazane](https://forum.opensearch.org/u/jmazane)\
**Post date:** [December 15, 2020, 8:48pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/3 "2020-12-15T20:48:18Z")

</div>

Hi @puzzled and @doc113, the issue is that the result is for the l2 space as opposed to the cosinesimil space for some reason.

Here is a note on scoring: [GitHub - opendistro-for-elasticsearch/k-NN: 🆕 A machine learning plugin which supports an approximate k-NN search algorithm for Open Distro.](https://github.com/opendistro-for-elasticsearch/k-NN#scoring)

_nmslib_ returns `1 - cosinesimilarity` as the result. This is because, in their library, the lower score corresponds to a closer result. Intuitively, this makes sense because the nearest neighbors should have smallest distances between them. For the l2 space, they just return the l2 distance. In Elasticsearch, the higher the score indicates a better result. So, we return this score: `1/(1 + score_from_nmslib)`.

So, it appears that the distance function being used is L2 for your examples for some reason:

```auto
sqrt((1 - 3)^2 + (2-4)^2 + (3-5)^2) = 3.4641
1/(1 + 3.4641) = 0.224

```

I am not sure why l2 distance function is being used instead of cosinesimil. We had this issue that caused a similar problem: [KNN graphs occasionally are indexed with wrong spaceType · Issue #239 · opendistro-for-elasticsearch/k-NN · GitHub](https://github.com/opendistro-for-elasticsearch/k-NN/issues/239). We backported the fixes to 1.8, 1.9, 1.10 branches, but the patched artifacts were never released. 1.11 artifacts should have the fix. Which version of ODFE are you using?

Jack

---

<div class="post-metadata">

**Author:** ![puzzled](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@puzzled](https://forum.opensearch.org/u/puzzled)\
**Post date:** [December 15, 2020, 9:59pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/4 "2020-12-15T21:59:34Z")

</div>

That’s true - it indeed returns the l2 score!  
How can I know ODFE version out of managed elasticsearch service (via API / console) ?

---

<div class="post-metadata">

**Author:** ![puzzled](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@puzzled](https://forum.opensearch.org/u/puzzled)\
**Post date:** [December 15, 2020, 10:12pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/5 "2020-12-15T22:12:09Z")

</div>

These are the exact steps to reproduce, as appeared in the issue:

1. Run docker  
docker run -p 9200:9200 -p 9600:9600 -e “discovery.type=single-node” amazon/opendistro-for-elasticsearch:1.11.0

2. Create l2 space (l2 as default, implicit type)

3. Delete index

4. Create cosine space (explicit)

5. Add data

6. Query for it

==\> Score is 1/(1+l2\_dist) instead of using cosine

---

<div class="post-metadata">

**Author:** ![puzzled](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@puzzled](https://forum.opensearch.org/u/puzzled)\
**Post date:** [December 15, 2020, 10:25pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/6 "2020-12-15T22:25:18Z")

</div>

This is what I get when the problem reproduces:  
Take a look specifically at

> **The setting “spaceType” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=l2**  
> Caching issue? That’s after all one of the 2 hardest problems in CS 🙂

The setting “M” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=16  
The setting “efConstruction” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=512  
creating index, cause [api], templates , shards [1]/[1]  
The setting “M” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=16  
The setting “efConstruction” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=512  
deleting index  
Cache evicted. Key /usr/share/elasticsearch/data/nodes/0/indices/o4xUzMUQRgeKh9KF9OmdXw/0/index/\_0\_206\_my\_vector1.hnswc, Reason: EXPLICIT  
The setting “spaceType” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=l2  
The setting “M” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=16  
The setting “efConstruction” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=512  
creating index, cause [api], templates , shards [1]/[1]  
The setting “spaceType” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=l2  
The setting “M” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=16  
The setting “efConstruction” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=512  
deleting index  
The setting “M” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=16  
The setting “efConstruction” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=512  
creating index, cause [api], templates , shards [1]/[1]  
The setting “M” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=16  
The setting “efConstruction” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=512

---

<div class="post-metadata">

**Author:** ![doc113](https://avatars.discourse-cdn.com/v4/letter/d/8797f3/32.png) [@doc113](https://forum.opensearch.org/u/doc113)\
**Post date:** [December 16, 2020, 12:35pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/7 "2020-12-16T12:35:38Z")

</div>

THANKS!!! I installed 1.10.1 and it works!!

---

<div class="post-metadata">

**Author:** ![puzzled](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@puzzled](https://forum.opensearch.org/u/puzzled)\
**Post date:** [December 16, 2020, 2:17pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/8 "2020-12-16T14:17:57Z")

</div>

Hey, are you looking into the issue?  
It’s sometimes reproducible in my side, sometimes not. I don’t think there’s a need to create l2 space for reproducing the error.  
I’ve cycled thru the steps couple of time to reproduce (incl. restarting the 1.11 docker build and starting fresh)

---

<div class="post-metadata">

**Author:** ![jmazane](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jmazane](https://forum.opensearch.org/u/jmazane)\
**Post date:** [December 16, 2020, 3:36pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/9 "2020-12-16T15:36:02Z")

</div>

Hi @puzzled, thanks for providing all of that information. We are looking into it. We will update this thread once we find out anything.

Jack

---

<div class="post-metadata">

**Author:** ![jmazane](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jmazane](https://forum.opensearch.org/u/jmazane)\
**Post date:** [December 16, 2020, 4:44pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/10 "2020-12-16T16:44:17Z")

</div>

Hi @puzzled,

I am trying to reproduce the issue with the ODFE 1.11.0.0 Docker image. It appears some kind of race condition is causing the failure. I haven’t been able to reproduce yet after several attempts.

I wanted to check if our workflows for reproduction differed at all. I am using the Docker image `amazon/opendistro-for-elasticsearch:1.11.0`.

1. Create cosine index

```auto
curl -X PUT "localhost:9200/cosine_1" -H 'Content-Type: application/json' -d'
{
  "settings" : {
    "number_of_shards" : 1,
    "number_of_replicas" : 0,
    "index": {
      "knn": true,
      "knn.space_type": "cosinesimil"
    }
  },
  "mappings": {
      "properties": {
        "my_vector": {
          "type": "knn_vector",
          "dimension": 3
        }
      }
  }
}
'

```

1. Index a document

```auto
curl -X POST "localhost:9200/cosine_1/_doc" -H 'Content-Type: application/json' -d'
{
"my_vector": [1, 2, 3]
}
'

```

1. Run search

```auto
curl -X POST "localhost:9200/cosine_1/_search" -H 'Content-Type: application/json' -d'
{
  "size" : 1,
  "query": {
    "knn": {
      "my_vector": {
        "vector": [3, 4, 5],
        "k": 1
      }
    }
  }
}
'

```

Each time I run, I get 0.9830015:

```auto
{"took":45,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":0.9830015,"hits":[{"_index":"cosine_1","_type":"_doc","_id":"3V9tbHYBLuKlid1SXins","_score":0.9830015,"_source":
{
"my_vector": [1, 2, 3]
}
}]}}

```

Is this how you are reproducing the issue?

---

<div class="post-metadata">

**Author:** ![puzzled](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@puzzled](https://forum.opensearch.org/u/puzzled)\
**Post date:** [December 17, 2020, 7:31am UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/11 "2020-12-17T07:31:23Z")

</div>

Exactly!  
At first I thought there’s a need for creating euclidean space first, but then was also able to reproduce without this step.

The loop that I’m doing for trying to reproduce includes restarting docker - and I think it’s an important step.

---

<div class="post-metadata">

**Author:** ![jmazane](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jmazane](https://forum.opensearch.org/u/jmazane)\
**Post date:** [December 17, 2020, 3:52pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/12 "2020-12-17T15:52:32Z")

</div>

@puzzled Oh okay, thanks. So, your process is something like the following?

1. Start ODFE 1.11 Docker container
2. Create cosine index
3. Index vector into index
4. Search index
5. Stop container
6. Start container
7. Delete cosine index
8. Create cosine index
9. Index vector into index
10. Search index  
…  
If not, could you list the steps you are using to reproduce the issue? I am still unable to reproduce it.

Additionally, did you mention that you were also able to reproduce this issue with Amazon Elasticsearch Service? If so, which version were you using?

---

<div class="post-metadata">

**Author:** ![puzzled](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@puzzled](https://forum.opensearch.org/u/puzzled)\
**Post date:** [December 17, 2020, 4:47pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/13 "2020-12-17T16:47:39Z")

</div>

Yes, and I have a specific cluster in production where the problem replicates constantly.  
I’m actually in a screen share with AWS support - want to join us?

ES version is 7.9.1

---

<div class="post-metadata">

**Author:** ![jmazane](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jmazane](https://forum.opensearch.org/u/jmazane)\
**Post date:** [December 27, 2020, 7:51pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/14 "2020-12-27T19:51:54Z")

</div>

HI @puzzled,

So, I think I found potentially what is causing the error. I am still unable to create a cosine index that creates l2 graphs, but I was able to produce the following log message when cosine is set:

```auto
The setting “spaceType” was not set for the index. Likely caused by recent version upgrade. Setting the setting to the default value=l2

```

I noticed that in Elasticsearch \<= 7.10, in the merge operation in [ParametrizedFieldMapper](https://github.com/elastic/elasticsearch/blob/7.10/server/src/main/java/org/elasticsearch/index/mapper/ParametrizedFieldMapper.java#L104), it passes empty index settings to the builder. When we are building the mapper, we try to [read these settings](https://github.com/opendistro-for-elasticsearch/k-NN/blob/master/src/main/java/com/amazon/opendistroforelasticsearch/knn/index/KNNVectorFieldMapper.java#L122).

Because the settings being passed in during merge are empty, this will lead to falling back to defaults.

The right solution in the long term is to switch these algorithm parameters to mapping parameters instead of index settings. We will fix that in an upcoming release. However, in the short term, I made [a change](https://github.com/jmazanec15/k-NN/commit/78e09bc631e84980147542c7d9ff673af5c439b5) to fix the problem without changing the index configuration.

Because I am still unable to reproduce the problem you are facing, I was wondering if you could test this code change with the Docker image you were able to produce the issue with. To do this, could you do the following steps?

```auto
# Build plugin with fix from source
git clone https://github.com/jmazanec15/k-NN.git
cd k-NN
git fetch
git checkout mapper-bug-fix-1.11
./gradlew build

# Create Docker image with fixed plugin
cat <<EOF >> Dockerfile
FROM amazon/opendistro-for-elasticsearch:1.11.0
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro-knn -p
COPY --chown=elasticsearch:elasticsearch ./build/distributions/opendistro-knn-1.11.0.0.zip /tmp/
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///tmp/opendistro-knn-1.11.0.0.zip --batch
EOF

docker build --tag=odfe-custom-plugin .

# Run container
docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" odfe-custom-plugin

```

Please let me know if there is a problem with that workflow.

Note – before executing those steps, you will need to set JAVA\_HOME, and have cmake, make, g++, git installed.

---

<div class="post-metadata">

**Author:** ![puzzled](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@puzzled](https://forum.opensearch.org/u/puzzled)\
**Post date:** [December 28, 2020, 2:03pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/15 "2020-12-28T14:03:11Z")

</div>

After handling few error in self compiling I stopped at a linking error between different java versions.  
It happens although I run with

> JAVA\_HOME=/usr/lib/jvm/adoptopenjdk-11-hotspot-amd64 ./gradlew build

This is the error I get:

> Error: LinkageError occurred while loading main class org.elasticsearch.gradle.reaper.Reaper  
> java.lang.UnsupportedClassVersionError: org/elasticsearch/gradle/reaper/Reaper has been compiled by a more recent version of the Java Runtime (class file version 58.0), this version of the Java Runtime only recognizes class file versions up to 55.0

* * *

I was trying to reproduce the problem locally, and couldn’t get to the same problem exactly, but was able to replicate a similar problem which seems to originate from same issue:

- I was able to get “spaceType” not set in a cosine index
- It happens on _data insert_ time, and not on index creation
- what happens is that I get no result at all, even though it should return at least one

---

<div class="post-metadata">

**Author:** ![jmazane](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jmazane](https://forum.opensearch.org/u/jmazane)\
**Post date:** [December 28, 2020, 3:09pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/16 "2020-12-28T15:09:30Z")

</div>

@puzzled

Ah okay I think you need to use JDK 14. You can get this from here: [Archived OpenJDK GA Releases](https://jdk.java.net/archive/).

Which version of software were you using when you were trying to reproduce locally?

---

<div class="post-metadata">

**Author:** ![puzzled](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@puzzled](https://forum.opensearch.org/u/puzzled)\
**Post date:** [December 28, 2020, 7:50pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/17 "2020-12-28T19:50:06Z")

</div>

Yes I realized I’m on wrong version but had to be off my computer for few hours.

I was using docker opendistro 1.11.0 …  
One thing that _may_ help reproduce the problem is drastically limiting CPU shares with

> docker run --cpu-shares 256 -p 9200:9200 -p 9600:9600 -e “discovery.type=single-node” amazon/opendistro-for-elasticsearch:1.11.0

Or even lower values

I’m available now, pinged you thru the ticket at AWS support

---

<div class="post-metadata">

**Author:** ![jmazane](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jmazane](https://forum.opensearch.org/u/jmazane)\
**Post date:** [December 29, 2020, 9:09pm UTC](https://forum.opensearch.org/t/cosine-similarity-formula/4390/18 "2020-12-29T21:09:56Z")

</div>

Tracking here: [KNNVectorFieldMapper unable to get algorithm params from settings when built from merge · Issue #288 · opendistro-for-elasticsearch/k-NN · GitHub](https://github.com/opendistro-for-elasticsearch/k-NN/issues/288)