Issue with OpenSearch Snapshot Deletion When Multiple Snapshots Exist in AWS S3 Repository

OpenSearch
1

Hi Team,

We are running OpenSearch of version 2.14 in a Kubernetes environment and backing up OpenSearch data to AWS S3. During testing, we have created multiple snapshots, which has led to the following issue.

Snapshot Repository Creation:

curl -X PUT "http://indexsearch:9200/_snapshot/is_backup" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "bucket2",
    "base_path": "/without-security/opensearch-backups"
  }
}'

Index Creation for Testing:

curl -X PUT "http://indexsearch:9200/my_index/_doc/1" -H 'Content-Type: application/json' -d'
{
  "title": "Sample Document1",
  "content": "This is a sample document 1.",
  "date": "2024-08-25"
}'

curl -X PUT "http://indexsearch:9200/my_index/_doc/2" -H 'Content-Type: application/json' -d'
{
  "title": "Sample Document2",
  "content": "This is a sample document 2.",
  "date": "2024-08-25"
}'

Snapshot Creation for Testing:

curl -X PUT "http://indexsearch:9200/_snapshot/is_backup/snapshot1"
{"accepted":true}

curl -X PUT "http://indexsearch:9200/_snapshot/is_backup/snapshot2"
{"accepted":true}

curl -X PUT "http://indexsearch:9200/_snapshot/is_backup/snapshot3"
{"accepted":true}

Available Snapshots:

curl -X GET "http://indexsearch:9200/_snapshot/is_backup/_all?pretty"

This command returns:

{
  "snapshots" : [
    {
      "snapshot" : "snapshot1",
      "uuid" : "titwItAaTrKwFvWX6U3eZg",
      "version_id" : 136357827,
      "version" : "2.14.0",
      ...
    },
    {
      "snapshot" : "snapshot2",
      "uuid" : "e8lNtDQCRT2tHoQrY7bDNg",
      "version_id" : 136357827,
      "version" : "2.14.0",
      ...
    },
    {
      "snapshot" : "snapshot3",
      "uuid" : "bWmjhaCyQPCcNINjNKqnew",
      "version_id" : 136357827,
      "version" : "2.14.0",
      ...
    }
  ]
}

Issue:

curl -XDELETE "http://indexsearch:9200/_snapshot/is_backup/snapshot1"

Error message:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"groupSize must be greater than 0 but was -1"}],"type":"illegal_argument_exception","reason":"groupSize must be greater than 0 but was -1"},"status":400}

Note :- Despite this error message, the snapshot will still be deleted in OpenSearch.

Available Snapshots after deleting snapshot1:

curl -X GET "http://indexsearch:9200/_snapshot/is_backup/_all?pretty"

This command returns:

{
  "snapshots" : [
        {
      "snapshot" : "snapshot2",
      "uuid" : "e8lNtDQCRT2tHoQrY7bDNg",
      "version_id" : 136357827,
      "version" : "2.14.0",
      ...
    },
    {
      "snapshot" : "snapshot3",
      "uuid" : "bWmjhaCyQPCcNINjNKqnew",
      "version_id" : 136357827,
      "version" : "2.14.0",
      ...
    }
  ]
}

However, when only one snapshot is present in the repository, the deletion proceeds without issues:

curl -X GET "http://indexsearch:9200/_snapshot/is_backup/_all?pretty"
{
  "snapshots" : [
    {
      "snapshot" : "snapshot3",
      "uuid" : "bWmjhaCyQPCcNINjNKqnew",
      "version_id" : 136357827,
      "version" : "2.14.0",
      "remote_store_index_shallow_copy" : false,
      "indices" : [
        "my_index"
      ],
      "data_streams" : [ ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2024-09-05T13:02:43.707Z",
      "start_time_in_millis" : 1725541363707,
      "end_time" : "2024-09-05T13:02:43.907Z",
      "end_time_in_millis" : 1725541363907,
      "duration_in_millis" : 200,
      "failures" : [ ],
      "shards" : {
        "total" : 1,
        "failed" : 0,
        "successful" : 1
      }
    }
  ]
}

curl -XDELETE "http://indexsearch:9200/_snapshot/is_backup/snapshot3"
{"acknowledged":true}

It appears this issue occurs when there are multiple snapshots in the repository, while deleting a single snapshot does not cause any errors. We would greatly appreciate any guidance or suggestions to resolve this issue.

Thank you!