Updating documents only if request is newer version

OpenSearch
,
1

Hi all, at the moment my team is attempting to create a search service for a platform and we are contemplating different methods of keeping the documents in the index up to date. One potential solution that was raised was use of the _update_by_query API.

Our documents have 2 relevant fields

  • Urn is our identifying field, it is a string.
  • Id indicates the revision of the file, it is an integer and increases for each version of the file.

Our concern is receiving update events from the platform out of order, we want to make sure we are not overwriting the document in the opensearch index with an outdated version. We would only like to insert the document by Urn if it doesnt already exist, or if it exists and the Id is less than what we are trying to insert.

After some investgation we found it was possible to create a script in opensearch, which copies all params over into the document, and specifying in the _update_by_query query that the id must be GTE the id we are about to attempt insertion of, but this doesnt seem to work for inserting a new document, doesnt seem to be usable with the bulk operation API.

I have come to the community in search of potential ideas and solutions, about how to “upsert” a document, only if it does not exist or if it exists with a field being less than what we are trying to insert. Interested to hear your thoughts!

2

You can use update API to achieve that, like this:

POST test11/_update/1
{
 "script": {
    "source": "if (ctx._source.id<=params.id) { ctx._source.id = params.id } else { ctx.op = 'noop' }",
    "lang": "painless",
    "params": {
      "id": 4
    }
  },
  "upsert": {
    "id":2,
    "urn": "abc"
  }
}

, if the document with id 1 does not exist, then insert a new document using the content of upsert, if the document with id 1 exists, then use the painless script to update the document only if the original id is LTE the new id.

1 Like