More_Like_This Query returns no results via python requests, but shows results in Opensearch Dashboards Dev Tools

OpenSearch
1

Hello everybody,

we are having problems with the More_Like_This query type.

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
AWS Managed Cluster Opensearch Version 2.11 - Software version OpenSearch_2_11_R20240502-P3

Describe the issue:
To query data via a More_Like_This query we have an AWS Lambda function that accesses the cluster and performs the query. This lambda function runs on python 3.11 and sends the query via python requests with authentication information from requests_aws4auth package to an index alias. Unfortunately this query returns no hits (and no error/ timeout). However when i perform the same query via Opensearch Dashboards Dev Tools i get the expected hits.
Does anybody else experienced this behavior or knows a solution, why the lambda does not get the expected results, but Dashboards Dev Tools?

Configuration:

The following query shows the described behavior:

{
        "from": 0,
        "size": 10,
        "query": {
            "more_like_this": {
                "fields": [
                    "content",
                    "ents.all"
                ],
                "like": [
                    {
                        "_id": "ebc7df84-33d1-430a-a9d7-b7f6e9424987"
                    }
                ],
                "max_query_terms": 25,
                "min_term_freq": 1,
                "min_doc_freq": 1
            }
        },
        "fields": [
            "id"
        ],
        "_source": false
    }

Our minimal document schema for this query is:

{
"id": "another id of our document",
"content": "the text content of our document",
"ents": {
    "all": ["ent1", "ent2"]
}

Relevant python script part:

creds = boto3.Session().get_credentials()
auth = requests_aws4auth.AWS4Auth(creds.access_key,
                                  creds.secret_key,
                                  region,
                                  'es',
                                  session_token=creds.token)
session = requests.Session()
session.auth = auth
url = f"{domainUrl}/myIndexAlias/_search"

response = session.get(url=self.url, json=query, timeout=20).json()

Python script response:

{
        "took": 2756,
        "timed_out": false,
        "_shards": {
            "total": 12,
            "successful": 12,
            "skipped": 0,
            "failed": 0
        },
        "hits": {
            "total": {
                "value": 0,
                "relation": "eq"
            },
            "max_score": null,
            "hits": []
        }
    }

Opensearch Dashboards Dev Tools:

GET myIndexAlias/_search

{
        "query": {
            "more_like_this": {
                "fields": [
                    "content",
                    "ents.all"
                ],
                "like": [
                    {
                        "_id": "ebc7df84-33d1-430a-a9d7-b7f6e9424987"
                    }
                ],
                "max_query_terms": 25,
                "min_term_freq": 1,
                "min_doc_freq": 1
            }
        },
        "fields": [
            "id"
        ],
        "_source": false
    }

Response (shortened):

{
  "took": 483,
  "timed_out": false,
  "_shards": {
    "total": 12,
    "successful": 12,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 154,
      "relation": "eq"
    },
    "max_score": 119.72375,
    "hits": [
      {
        "_index": "myIndex1",
        "_id": "a4475c30-ba64-460d-ae22-608ece79dc2a",
        "_score": 119.72375,
        "fields": {
          "id": [
            "a4475c30-ba64-460d-ae22-608ece79dc2a"
          ]
        }
      },
      {
        "_index": "myIndex1",
        "_id": "93113ea3-5a5d-4260-8f90-fc4f8b0ec98d",
        "_score": 28.89265,
        "fields": {
          "id": [
            "93113ea3-5a5d-4260-8f90-fc4f8b0ec98d"
          ]
        }
      },
      {
        "_index": "myIndex2",

Relevant Logs or Screenshots:

  • we experienced the same behavior with AWS managed cluster Opensearch version 2.9
  • we investigated permissions, but did not find anything, that could be the cause of this behavior

Thank you already for any tips & leads!