Hybrid Query Explanation: Only a single min_max_normalization block

OpenSearch
3

Hi @martin.g,

Thanks for looking into this and your answers. Regarding answer 3, I think the example that I provided above shows a match for both subqueries. Otherwise I figured the knn part would say “not in top k docs“.

Here is the further information you requested:

  1. The _explanation output as provided above is from one single hit.

PUT /_search/pipeline/nlp-search-pipeline

    {
  "description": "Post processor for hybrid search with custom weights",
  "phase_results_processors": [
    {
      "normalization-processor": {
        "normalization": {
          "technique": "min_max"
        },
        "combination": {
          "technique": "arithmetic_mean",
          "parameters": {
            "weights": [0.3, 0.7]
          }
        }
      }
    }
  ]
}

which returns

{
  "_explanation": {
    "value": 4.6858487,
    "description": "sum of:",
    "details": [
      {
        "value": 3.8993959426879883,
        "description": "combined score of:",
        "details": [
          {
            "value": 3.899396,
            "description": "weight(name:wind in 10513) [PerFieldSimilarity], result of:",
            "details": [
              {
                "value": 3.899396,
                "description": "score(freq=1.0), computed as boost * idf * tf from:",
                "details": [
                  {
                    "value": 6.6268253,
                    "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                    "details": [
                      {
                        "value": 18,
                        "description": "n, number of documents containing term",
                        "details": []
                      },
                      {
                        "value": 13968,
                        "description": "N, total number of documents with field",
                        "details": []
                      }
                    ]
                  },
                  {
                    "value": 0.58842593,
                    "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                    "details": [
                      {
                        "value": 1,
                        "description": "freq, occurrences of term within document",
                        "details": []
                      },
                      {
                        "value": 1.2,
                        "description": "k1, term saturation parameter",
                        "details": []
                      },
                      {
                        "value": 0.75,
                        "description": "b, length normalization parameter",
                        "details": []
                      },
                      {
                        "value": 3,
                        "description": "dl, length of field",
                        "details": []
                      },
                      {
                        "value": 6.759307,
                        "description": "avgdl, average length of field",
                        "details": []
                      }
                    ]
                  }
                ]
              }
            ]
          },
          {
            "value": 0.78645265,
            "description": "within top 10 docs",
            "details": []
          }
        ]
      },
      {
        "value": 0,
        "description": "match on required clause, product of:",
        "details": [
          {
            "value": 0,
            "description": "# clause",
            "details": []
          },
          {
            "value": 1,
            "description": "FieldExistsQuery [field=_primary_term]",
            "details": []
          }
        ]
      }
    ]
  }
}

There seems to be an additional object JSON in the top-level details object.
Note that this is not the same hit as in the example above (doing this manually makes it hard to keep track, see below).

  1. For ten results, there is just one min max block per hit. I tried this with k = 100, and I count as many min max blocks as hits (using the original search pipeline).

  2. I initially tried this on a single small index with 13’968 docs and then re-ran the same query over all indices (22 indices in total, sizes differ per index as this corresponds to sources or providers in our ETL pipeline) with 4’895’598 docs in total using the wildcard pattern *. I do not see any difference regarding the response’s structure.

This what I can provide for now. I did this using the dev console. I think for a more systematic approach, I’d need to put the steps in a script, so this could be run automatically.

Let me know if this helps or additional details are needed. Thank you very much.

PS: Actually, I observed this already last year but then all of a sudden, I got the structure I expected, see Hybrid Score Explain Output's Structure Diverges from Docs - #2 by tobe I cannot explain this change in behaviour.