Null_pointer_exception with Faiss byte vectors when filtering

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.17.0

Describe the issue:
I have been trying to use the faiss engine with byte vectors. I can successfully load the vectors into an index and searching works unless I provide a filter (Efficient k-NN filtering). Literally any filter, even one that doesn’t filter anything out.

It returns:
null_pointer_exception
Cannot read the array length because the return value of “org.opensearch.knn.index.query.KNNQuery.getByteQueryVector()” is null"

Configuration:

  • Opensearch 2.17
  • Knn engine faiss
  • byte vectors
  • 1024 dimensions
  • hnsw

Relevant Logs or Screenshots:
Error:

{
  "error": {
    "root_cause": [
      {
        "type": "null_pointer_exception",
        "reason": "Cannot read the array length because the return value of \"org.opensearch.knn.index.query.KNNQuery.getByteQueryVector()\" is null"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "development-20241204111110-3de6471c",
        "node": "5ry1X70gTgyIOlMItg6TzA",
        "reason": {
          "type": "null_pointer_exception",
          "reason": "Cannot read the array length because the return value of \"org.opensearch.knn.index.query.KNNQuery.getByteQueryVector()\" is null"
        }
      }
    ],
    "caused_by": {
      "type": "null_pointer_exception",
      "reason": "Cannot read the array length because the return value of \"org.opensearch.knn.index.query.KNNQuery.getByteQueryVector()\" is null",
      "caused_by": {
        "type": "null_pointer_exception",
        "reason": "Cannot read the array length because the return value of \"org.opensearch.knn.index.query.KNNQuery.getByteQueryVector()\" is null"
      }
    }
  },
  "status": 500
}

query:

POST development-20241204111110-3de6471c/_search
{
  "query": {
    "bool": {
      "must": {
        "knn": {
          "subject_embedding": {
            "vector": [
              -7,
              18,
              -13,
              50,
              -34,
              -36,
              -3,
              -33,
              -49,
              21,
              38,
              -29,
              37,
              26,
              41,
              -10,
              22,
              -11,
              8,
              -22,
              -45,
              -1,
              45,
              -35,
              -98,
              15,
              25,
              -16,
              9,
              32,
              -4,
              1,
              60,
              72,
              -17,
              0,
              29,
              -24,
              7,
              27,
              -8,
              13,
              28,
              43,
              -54,
              -24,
              19,
              16,
              18,
              60,
              -14,
              0,
              12,
              16,
              18,
              0,
              31,
              -19,
              86,
              61,
              -16,
              -89,
              -63,
              -37,
              85,
              5,
              25,
              33,
              36,
              32,
              -8,
              8,
              65,
              -14,
              -31,
              26,
              29,
              7,
              29,
              59,
              1,
              31,
              36,
              21,
              56,
              -11,
              21,
              30,
              30,
              -18,
              -10,
              67,
              72,
              2,
              45,
              -22,
              81,
              -49,
              47,
              -73,
              -28,
              -34,
              22,
              12,
              7,
              5,
              -29,
              -19,
              -34,
              19,
              29,
              48,
              -6,
              -16,
              -19,
              -38,
              16,
              -14,
              -13,
              -54,
              -25,
              50,
              -15,
              -30,
              -27,
              -27,
              54,
              18,
              -29,
              -27,
              -63,
              -37,
              -10,
              25,
              59,
              -13,
              -49,
              -66,
              15,
              1,
              -15,
              16,
              -35,
              14,
              -22,
              -25,
              17,
              36,
              -11,
              -54,
              41,
              20,
              31,
              -71,
              -29,
              29,
              52,
              -15,
              -2,
              -29,
              -2,
              -38,
              -13,
              -26,
              -14,
              -18,
              12,
              -5,
              7,
              41,
              49,
              -77,
              -51,
              -18,
              0,
              -41,
              -29,
              57,
              52,
              -50,
              42,
              0,
              30,
              43,
              98,
              44,
              -55,
              -27,
              10,
              127,
              -36,
              -26,
              -53,
              11,
              -25,
              32,
              33,
              -61,
              23,
              97,
              10,
              -2,
              1,
              43,
              3,
              31,
              45,
              0,
              -32,
              70,
              -54,
              43,
              -6,
              79,
              -7,
              -39,
              30,
              -6,
              -11,
              10,
              115,
              -14,
              -4,
              28,
              15,
              -44,
              10,
              -64,
              8,
              -42,
              -15,
              -56,
              26,
              12,
              15,
              21,
              -23,
              -23,
              -71,
              10,
              36,
              50,
              -64,
              -12,
              48,
              -37,
              -8,
              -37,
              16,
              -25,
              0,
              26,
              -30,
              -9,
              25,
              -18,
              -18,
              -1,
              40,
              28,
              -36,
              -40,
              84,
              -30,
              -9,
              -24,
              3,
              29,
              -4,
              16,
              -28,
              -9,
              -36,
              79,
              68,
              94,
              -57,
              59,
              -29,
              32,
              30,
              -3,
              12,
              -21,
              -48,
              20,
              -2,
              8,
              -2,
              -4,
              60,
              -73,
              -6,
              14,
              -4,
              -1,
              -22,
              -13,
              6,
              -18,
              8,
              26,
              29,
              -40,
              58,
              46,
              -14,
              28,
              105,
              -34,
              -6,
              -3,
              36,
              10,
              -10,
              16,
              -50,
              -55,
              -20,
              69,
              -38,
              -28,
              -5,
              37,
              -32,
              -10,
              2,
              -59,
              -29,
              26,
              -43,
              67,
              32,
              16,
              82,
              49,
              87,
              9,
              80,
              -11,
              33,
              -6,
              -15,
              -28,
              65,
              -11,
              30,
              -29,
              -42,
              -28,
              -20,
              19,
              -23,
              18,
              97,
              -40,
              45,
              -19,
              12,
              -72,
              96,
              -51,
              -25,
              12,
              -5,
              -22,
              4,
              14,
              -41,
              -4,
              53,
              -9,
              22,
              49,
              -39,
              1,
              -16,
              14,
              -7,
              15,
              -18,
              13,
              -29,
              71,
              41,
              -5,
              4,
              -13,
              26,
              14,
              105,
              12,
              81,
              -9,
              -2,
              8,
              -8,
              20,
              -34,
              -21,
              96,
              -86,
              16,
              -49,
              20,
              26,
              33,
              -11,
              -23,
              -34,
              -19,
              -41,
              -48,
              17,
              -2,
              9,
              11,
              101,
              -57,
              -7,
              -20,
              -28,
              -4,
              -1,
              -28,
              35,
              26,
              -20,
              101,
              11,
              13,
              -24,
              -1,
              -16,
              -2,
              -47,
              -49,
              -37,
              -26,
              37,
              0,
              1,
              54,
              -26,
              73,
              17,
              -6,
              41,
              -99,
              15,
              16,
              3,
              51,
              -40,
              -4,
              -89,
              -7,
              73,
              -12,
              7,
              -75,
              -5,
              27,
              -94,
              -7,
              -40,
              -29,
              16,
              5,
              -44,
              -32,
              54,
              -34,
              -25,
              36,
              30,
              -33,
              -3,
              -6,
              -34,
              -9,
              24,
              45,
              -30,
              -45,
              29,
              48,
              0,
              116,
              -89,
              -5,
              48,
              -11,
              37,
              -13,
              -89,
              23,
              -74,
              -106,
              -44,
              77,
              34,
              -35,
              -6,
              -12,
              -11,
              -18,
              -28,
              35,
              -5,
              -7,
              0,
              -22,
              41,
              14,
              -66,
              62,
              24,
              53,
              -85,
              -6,
              28,
              -22,
              -12,
              -34,
              6,
              -73,
              31,
              -69,
              38,
              -50,
              -55,
              -23,
              -33,
              -30,
              -103,
              62,
              28,
              40,
              20,
              64,
              -19,
              66,
              0,
              29,
              -30,
              -66,
              -16,
              6,
              99,
              -21,
              21,
              22,
              -26,
              39,
              27,
              -30,
              -20,
              1,
              49,
              -41,
              -47,
              27,
              4,
              -20,
              -5,
              -7,
              28,
              44,
              -17,
              -19,
              21,
              20,
              -7,
              -27,
              -6,
              -30,
              -64,
              -37,
              -6,
              -58,
              32,
              36,
              4,
              -85,
              -27,
              54,
              16,
              71,
              -3,
              66,
              -11,
              20,
              0,
              4,
              52,
              -6,
              18,
              81,
              16,
              -26,
              5,
              -38,
              -12,
              -31,
              59,
              32,
              -57,
              -8,
              10,
              2,
              -11,
              107,
              15,
              17,
              -28,
              -37,
              46,
              43,
              35,
              -25,
              -38,
              -85,
              -28,
              -13,
              22,
              36,
              3,
              -90,
              -24,
              -95,
              -78,
              10,
              -11,
              43,
              -65,
              -66,
              -52,
              -34,
              -20,
              2,
              -55,
              47,
              -10,
              -16,
              -31,
              -13,
              -56,
              -32,
              12,
              -27,
              -8,
              19,
              -68,
              -6,
              78,
              -14,
              20,
              -39,
              112,
              36,
              -25,
              11,
              69,
              -65,
              27,
              -59,
              38,
              17,
              -74,
              25,
              14,
              -3,
              35,
              39,
              -17,
              -37,
              -40,
              -37,
              -72,
              15,
              48,
              -37,
              49,
              -2,
              32,
              -87,
              -37,
              2,
              -23,
              8,
              58,
              -39,
              2,
              19,
              -26,
              -53,
              -18,
              18,
              56,
              38,
              21,
              -32,
              -48,
              -42,
              -46,
              55,
              -58,
              -15,
              17,
              67,
              66,
              -31,
              37,
              -20,
              43,
              19,
              -11,
              -76,
              76,
              -50,
              -19,
              -48,
              -27,
              8,
              -34,
              -15,
              11,
              -55,
              -1,
              -33,
              48,
              -39,
              24,
              -28,
              -5,
              -49,
              -78,
              45,
              -45,
              27,
              92,
              30,
              -64,
              -7,
              4,
              -36,
              18,
              -48,
              -3,
              -12,
              45,
              16,
              11,
              -17,
              -3,
              19,
              -2,
              7,
              52,
              -10,
              -34,
              7,
              -7,
              -47,
              34,
              8,
              -3,
              -10,
              18,
              6,
              -16,
              36,
              -19,
              29,
              -22,
              -4,
              -11,
              28,
              -16,
              -65,
              -43,
              42,
              -43,
              -14,
              14,
              70,
              -5,
              33,
              10,
              -50,
              -9,
              33,
              -20,
              -58,
              -9,
              -23,
              20,
              9,
              -93,
              79,
              -51,
              -57,
              18,
              -1,
              -9,
              -11,
              39,
              -12,
              54,
              -47,
              83,
              -5,
              9,
              -4,
              26,
              49,
              52,
              -39,
              -8,
              11,
              2,
              13,
              -18,
              2,
              40,
              55,
              -43,
              -5,
              19,
              -92,
              -36,
              -14,
              -16,
              -16,
              0,
              42,
              62,
              18,
              -10,
              10,
              -5,
              -7,
              3,
              67,
              -1,
              36,
              102,
              -1,
              50,
              5,
              -12,
              96,
              -25,
              22,
              -31,
              26,
              76,
              -24,
              27,
              -20,
              18,
              -45,
              -15,
              7,
              2,
              -8,
              -2,
              34,
              -37,
              -5,
              -27,
              -41,
              -64,
              25,
              2,
              18,
              -8,
              9,
              9,
              59,
              -34,
              -2,
              -34,
              -19,
              41,
              -38,
              66,
              -34,
              1,
              -13,
              -33,
              9,
              -3,
              -29,
              0,
              37,
              -7,
              -58,
              -21,
              -22,
              28,
              7,
              -41,
              8,
              -4,
              33,
              22,
              49,
              -7,
              -31,
              57,
              -18,
              34,
              -38,
              -15,
              24,
              15,
              -82,
              -22,
              -65,
              0,
              10,
              0,
              -40,
              -40,
              33,
              7,
              4,
              7,
              -10,
              0,
              -14,
              -21,
              -66,
              -74,
              19,
              -19,
              -27,
              -10,
              5,
              -60,
              13,
              -12,
              -31,
              52,
              -39,
              12,
              -27,
              -58,
              72,
              18,
              45,
              -27,
              -10,
              12,
              11,
              -6,
              62,
              -4,
              -26,
              25,
              14,
              -48,
              -25,
              -28,
              91,
              -53,
              -26,
              17,
              -17,
              -18,
              -6,
              10,
              -40,
              62,
              37,
              45,
              1,
              60,
              -50,
              -65,
              -10,
              8,
              19,
              13,
              62,
              16,
              6,
              -69,
              0,
              5,
              -95,
              -73,
              52,
              85,
              -51,
              32,
              78,
              21,
              24,
              -32,
              8,
              -16,
              41,
              1,
              99,
              -10,
              -50,
              27,
              56,
              -8,
              59,
              -53,
              -15,
              -50,
              -17,
              -15,
              -28,
              -8,
              13,
              34,
              -13,
              -29
            ],
            "k": 1,
            "filter": {
              "bool": {
                "must": [
                  {
                    "exists": {
                      "field": "subject"
                    }
                  }
                ]
              }
            }
          }
        }
      }
    }
  },
  "_source": false,
  "size": 5,
  "sort": [
    "_score",
    {
      "id": "desc"
    }
  ],
  "from": 0,
  "track_total_hits": true,
  "timeout": "10s"
}

Everything else is followed from

Have I missed something in the setup?

@matt.haigh Thanks for bringing this bug to our attention. I’m able to replicate this bug which happens only while computing P * Dimension <= Max_distance_computations and works for other scenarios. Created a github issue to track this and will release the fix in OpenSearch 2.19.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.