Creating index for only exact KNN search

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.3

Describe the issue:

I want to create an index that will only be used for exact KNN search. While I plan to index millions of documents, at any given time I’ll only be searching over a subset of 5,000–10,000 documents (pre-filtering). Given this access pattern, I believe exact KNN search is sufficient — and by avoiding the overhead of an HNSW graph, I expected to reduce both index size and cost.

To test this, I created an index with the knn setting explicitly set to false:

PUT /test-exact-knn-index
{
  "settings": {
    "index": {
      "knn": false,
      "number_of_shards": 5,
      "number_of_replicas": 1
    }
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "keyword",
        "doc_values": false
      },
      "text_embedding": {
        "type": "knn_vector",
        "dimension": 768,
        "doc_values": false
      },
      "text": {
        "type": "text"
      }
    }
  }
}

Contrary to my expectation, I noticed that the size of the index is larger than before. All the values below are excluding the replica size.

  • Size of index with knn setting true and in_memory mode: 164GB
  • Size of index with knn setting true and on_disk mode (with 32x compression): 105GB
  • Size of index with knn setting as false: 222GB

I initially noticed that the doc_values setting for the text_embedding field is set to true and thought that was the reason the size of the index is going up. I then tried to explicitly set it to false like above, but it still shows as true after the index is created.

Here are the index settings:

{
  "test-exact-knn-index": {
    "aliases": {},
    "mappings": {
      "properties": {
        "id": {
          "type": "keyword",
          "doc_values": false
        },
        "text": {
          "type": "text"
        },
        "text_embedding": {
          "type": "knn_vector",
          "doc_values": true,
          "dimension": 768
        }
      }
    },
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "number_of_shards": "5",
        "provided_name": "test-exact-knn-index",
        "knn": "false",
        "creation_date": "1771958779417",
        "number_of_replicas": "1",
        "uuid": "ARKqYReGRgsEGZcTNhQfEg",
        "version": {
          "created": "137247498"
        }
      }
    }
  }
}

Questions:

  • Why does the index size increase when knn is set to false? Shouldn’t removing the HNSW graph reduce storage overhead?

  • Is this the correct approach for configuring an index for only exact KNN search?

  • Why is doc_values being forced to true on the knn_vector field even when explicitly set to false?

  • I also observed that enabling on_disk mode (with 32x compression) reduces the index size from 164 GB to 105 GB, with all other settings being the same. My understanding of on_disk mode is that it stores quantized vectors in memory and full-precision vectors on disk. If that’s the case, I would expect the total storage footprint to be slightly larger than in_memory mode, since it now stores both the quantized and full-precision copies. Why does on_disk mode result in a smaller index size compared to in_memory mode? Am I misunderstanding how on_disk mode manages vector storage?