Point in Time Errors

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.6
OpenSearch Dashboards 2.6

I see the same behavior in my deployment as well as in the playground.

Describe the issue:

I’m trying to paginate search results using PIT following the documentation at Point in Time - OpenSearch documentation, but I’m seeing some errors:

  • OpenSearch doesn’t seem to know about the field format under sort
  • OpenSearch doesn’t seem to know about _shard_doc

Is the documentation incorrect, could my OpenSearch (and the playground) be misconfigured, or am I making a mistake in my queries?

I’d like to use _shard_doc as a tie breaker since using _id results in high memory use and I’d prefer not to generate my own field with a unique id for each document if I don’t have to.

Thanks for the help.

Configuration:

playground.opensearch.org

Relevant Logs or Screenshots:

Here are the series of requests in the OpenSearch playground console:

POST sample-ecommerce/_search/point_in_time?keep_alive=30m to get a PID to use in requests below

GET _search
{
  "size": 1000,
  "query": {
    "match_all": {}
  },
  "pit": {
    "id": "...",
    "keep_alive": "5m"
  },
  "sort": [
    {"timestamp": {"order": "asc", "format": "strict_date_optional_time_nanos"}},
    {"_shard_doc": "desc"}
  ]
}

results in:

{
  "error": {
    "root_cause": [
      {
        "type": "x_content_parse_exception",
        "reason": "[11:36] [field_sort] unknown field [format]"
      }
    ],
    "type": "x_content_parse_exception",
    "reason": "[11:36] [field_sort] unknown field [format]"
  },
  "status": 400
}

Removing format:

GET _search
{
  "size": 1000,
  "query": {
    "match_all": {}
  },
  "pit": {
    "id": "...",
    "keep_alive": "5m"
  },
  "sort": [
    {"timestamp": {"order": "asc"}},
    {"_shard_doc": "desc"}
  ]
}

results in

{
  "error": {
    "root_cause": [
      {
        "type": "query_shard_exception",
        "reason": "No mapping found for [_shard_doc] in order to sort on",
        "index": "sample-ecommerce",
        "index_uuid": "79PvG4CEQom9zBxQxRi3hw"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "sample-ecommerce",
        "node": "8Q__D4LrSQCv9Tn0h7269w",
        "reason": {
          "type": "query_shard_exception",
          "reason": "No mapping found for [_shard_doc] in order to sort on",
          "index": "sample-ecommerce",
          "index_uuid": "79PvG4CEQom9zBxQxRi3hw"
        }
      }
    ]
  },
  "status": 400
}

removing _shard_doc results in:

GET _search
{
  "size": 1000,
  "query": {
    "match_all": {}
  },
  "pit": {
    "id": "...",
    "keep_alive": "5m"
  },
  "sort": [
    {"timestamp": {"order": "asc"}}
  ]
}

which works.

2

The documentation is not correct, OpenSearch does not support _shard_doc parameter and format parameter under the date field when sorting, both of the parameters were introduced in versions after 7.10 in Elasticsearch, the sample in OpenSearch’s documentation needs to be corrected.

3

Can you open an issue for that please :smiley: Sign in to GitHub · GitHub (if you havent already)