1536 dims vector search issue on v2.14

Plugins k-NN
1

Hello everyone. I’ve encountered an issue while using OpenSearch and I’m posting here for help. Thank you in advance for your reply.

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 2.14

Describe the issue:

  1. I’ve been using a 1024-dim embedding model(bge-m3) with OpenSearch 2.14, and now I want to test OpenAI text-embedding-3-small (1536 dims).
  2. I tried to switch by only changing the vector dimension, but the Lucene engine errors out, saying it only supports dimensions ≤ 1024.
  3. I then tried the FAISS engine. I indexed new vectors produced by the OpenAI small model, but all searches return no results, even when I query with a vector that’s exactly the same as one I ingested.
  4. My hunch is this might be because the search algorithm is using L2 while my use case expects cosine similarity.
  5. I attempted to create a new index with space_type = cosinesimil and name = hnsw, but OpenSearch reported that this combination isn’t supported.
  6. Question: Is this limitation due to being on 2.14? If so, what’s the correct vector_field configuration to test 1536-dim vector search on this version?

Configuration:

  • there was no results when I attempted vector search
    (1. mapping options, 2. python client code)

    •     "vector_field": {
      "dimension": 1536,
      "method": {
        "engine": "faiss",
        "space_type": "l2",
        "name": "hnsw",
        "parameters": {
          "ef_construction": 100,
          "m": 16
        }
      },
      "type": "knn_vector"
    }

    • k = 10
body = {
    "size": k,
    "query": {
        "knn": {
            "vector_field": {
                "vector": embedding,
                "k": k*3,
            }
        }
    }
}
response = client.search(index='myindex', body=body)

  • OpenSearch reported that this combination isn’t supported.

    [mapper_parsing_exception] Failed to parse mapping [_doc]: Validation Failed: 1: “hnsw” configuration does not support space type: “cosinesimil”.;

    •     "vector_field": {
      "dimension": 1536,
      "method": {
        "engine": "faiss",
        "space_type": "cosinesimil",
        "name": "hnsw",
        "parameters": {
          "ef_construction": 100,
          "m": 16
        }
      },
      "type": "knn_vector"
    }

Relevant Logs or Screenshots:

Thanks for your help.

2

@ji99999 According to OpenSearch release notes the cosine similarity for Faiss engine was introduced in version 2.19.0.

Cosine similarity support in the Faiss engine for k-NN and radial search eliminates the need for manual data normalization, offering benefits for use cases such as recommendation systems, fraud detection, and content-based search applications.

3

@pablo So is there any way to do a 1536 dimensional vector search in version 2.14?

4

@ji99999 I think what you want is use cosine similarity with Faiss in 2.14. Currently there is no native way. What you would need to do is either switch to nmslib/lucene engine, or if you want to use Faiss then normaize the vector before ingestion and then use L2 or innerproduct

Other way will be upgrade to 2.19