OpenSearch TTL for Indexed Documents

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch

Describe the issue:

We propose that the OpenSearch team implement a Time-To-Live (TTL) mechanism for indexed documents, similar to what’s available in DynamoDB.

Background and Rationale:
Many teams, including ours, frequently need to soft delete (or tombstone) documents in OpenSearch. Currently, OpenSearch does not support an automatic TTL mechanism for these tombstoned documents. As a result, teams have to manually write and execute queries to find and prune these documents, which is both time-consuming and error-prone.

Proposed Feature:

  • TTL Mechanism: Allow documents in OpenSearch to have a TTL attribute. Once the TTL expires, the document should be automatically deleted from the index.
  • Use-Case Alignment: This feature would greatly assist in scenarios where documents are marked as tombstoned and need to be automatically deleted after a certain period.

Example Use-Case:

{
  "query": {
    "bool": {
      "must": [
        { "term": { "isTombstoned": true } },
        { "range": { "timeToLive": { "lte": "now" } } }
      ]
    }
  }
}

In the above query, documents marked as tombstoned would ideally be automatically removed post the TTL period, reducing the need for manual cleanup.

Implementing this feature could significantly streamline operations for numerous OpenSearch users, enhancing the platform’s efficiency and usability.

Configuration:

Relevant Logs or Screenshots:

1 Like