KNN Queries Are Slow And Not Cached

Plugins k-NN
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch Docker image: opensearchproject/opensearch:2.17.0

Describe the issue:
We have built a vector index and knn queries are quite slow (around 20 seconds). We checked the stats and the cache seems empty although “knn_query_with_filter_requests” increments with more queries being executed.

Configuration:
docker compose:

services:
  opensearch:
    restart: unless-stopped
    image: opensearchproject/opensearch:2.17.0
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
    environment:
      - discovery.type=single-node
      - cluster.name=opensearch-cluster
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping  
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=XYZ
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data:/usr/share/opensearch/data

docker compose override:

services:
  opensearch:
    environment:
      OPENSEARCH_JAVA_OPTS: -Xms32g -Xmx32g
    deploy:
      resources:
        limits:
          memory: 34g
          cpus: "20"

Mapping:

settings:
  default:
    index:
        number_of_shards: 2
        number_of_replicas: 2
        max_result_window: 3000000
        knn: true
    mappings:
      default:
        dynamic: false
        properties:
          "embedding":
            type: "knn_vector"
            dimension: 768
            method:
              engine: lucene
              name: hnsw
              space_type: l2
              parameters:
                ef_construction: 128
                m: 16

Opensearch has 32GB of memory. There are around 4 millions docs distributed over several indices.

Relevant Logs or Screenshots:

After warmup (k-NN plugin API - OpenSearch Documentation):

GET /_plugins/_knn/stats?pretty

{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "opensearch-cluster",
  "circuit_breaker_triggered": false,
  "model_index_status": null,
  "nodes": {
    "xyz...": {
      "max_distance_query_with_filter_requests": 0,
      "graph_memory_usage_percentage": 0,
      "graph_query_requests": 0,
      "graph_memory_usage": 0,
      "cache_capacity_reached": false,
      "load_success_count": 0,
      "training_memory_usage": 0,
      "indices_in_cache": {},
      "script_query_errors": 0,
      "hit_count": 0,
      "knn_query_requests": 2174,
      "total_load_time": 0,
      "miss_count": 0,
      "min_score_query_requests": 0,
      "knn_query_with_filter_requests": 819,
      "training_memory_usage_percentage": 0,
      "max_distance_query_requests": 0,
      "lucene_initialized": true,
      "graph_index_requests": 0,
      "faiss_initialized": false,
      "load_exception_count": 0,
      "training_errors": 0,
      "min_score_query_with_filter_requests": 0,
      "eviction_count": 0,
      "nmslib_initialized": false,
      "script_compilations": 0,
      "script_query_requests": 0,
      "graph_stats": {
        "merge": {
          "current": 0,
          "total": 0,
          "total_time_in_millis": 0,
          "current_docs": 0,
          "total_docs": 0,
          "total_size_in_bytes": 0,
          "current_size_in_bytes": 0
        },
        "refresh": {
          "total": 0,
          "total_time_in_millis": 0
        }
      },
      "graph_query_errors": 0,
      "indexing_from_model_degraded": false,
      "graph_index_errors": 0,
      "training_requests": 0,
      "script_compilation_errors": 0
    }
  }
}

If we understand the docs correctly, the cache should not be empty after warmup:

The k-NN plugin builds a native library index of the vectors for each knn-vector field/Lucene segment pair during indexing, which can be used to efficiently find the k-nearest neighbors to a query vector during search. To learn more about Lucene segments, see the Apache Lucene documentation. These native library indexes are loaded into native memory during search and managed by a cache. To learn more about preloading native library indexes into memory, refer to the warmup API. Additionally, you can see which native library indexes are already loaded in memory. To learn more about this, see the stats API section.

I wanted to try the performance analyzer plugin but it is broken: [BUG] Performance Analyzer webserver on port 9600 not responding to any API calls (caused by JDK upgrade?) · Issue #545 · opensearch-project/performance-analyzer-rca · GitHub

Thanks for any hint.

2

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.