Orphaned blobs Cleanup

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.7

Describe the issue: When I try to run the following API

POST /_snapshot/<repo>/_cleanup

I am getting the following error.

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "groupSize must be greater than 0 but was -10"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "groupSize must be greater than 0 but was -10"
  },
  "status": 400
}

Getting the same error when I reduced the chunk_size to 100mb.

2

@muraliv I reproduced this error. Here’s what’s happening and how to work around it.

The error groupSize must be greater than 0 but was -10 is thrown deep in BlobStoreRepository.cleanupStaleIndices where a GroupedActionListener is constructed with:

groupSize = foundIndices.size() - survivingIndexIds.size()

foundIndices is the set of index blob-container directories actually present in your repository storage. survivingIndexIds is the set of index IDs recorded in the repository’s index-N metadata file. When 10 more index IDs appear in the metadata than exist on disk, the subtraction is negative and the exception is thrown before any cleanup work starts.

In other words, your repository metadata references 10 index blob-container directories that no longer exist in storage. This is the inverse of what _cleanup normally handles (extra blobs with no metadata). The _cleanup code does not guard against this case and crashes instead of reporting a meaningful error, I would recommend to raise a bug issue for this here.

This typically happens when objects are removed from underlying storage out-of-band, for example, an S3 lifecycle policy, a bucket cleanup, or accidental manual deletion of objects under indices/, while the index-N metadata file was left intact. The metadata still believes those 10 index directories exist; the storage does not.

The chunk_size setting is unrelated to this error.

Since those 10 index directories are already gone, any snapshots that included them are no longer fully restorable. The fix is to remove those snapshots so that the metadata no longer references the missing directories.

  1. List your snapshots and identify which ones included the affected indices:

    GET /_snapshot/<repo>/_all

  2. Delete the affected snapshots:

    DELETE /_snapshot/<repo>/<snapshot-name>

    During deletion, OpenSearch writes updated metadata that drops the missing index IDs from survivingIndexIds. Because the directories are already gone from foundIndices too, the mismatch disappears and no negative groupSize is computed.

  3. Once all affected snapshots are removed, run _cleanup again, it should complete successfully.

Hope this helps

3

Hi @Anthony ,

I tried to list all of the snapshots in this repo and got an error straight away.

Blob object [snap-g1xDpNJOQiW43sjKqqSLKg.dat] not found: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: bb3862e9-9a81-4088-9e58-140726268083) (SDK Attempt Count: 1)

Option is to use an S3 browser and manually delete the snapshots ?

Thanks

Murali

4

@muraliv
The new error confirms the repository has lost more than just the index blob-container directories, the snapshot metadata file snap-g1xDpNJOQiW43sjKqqSLKg.dat is also missing from S3. OpenSearch reads that file during GET /_snapshot/_all, so the listing itself fails, which means the OpenSearch API can no longer manage this repository at all.

On your S3-browser question, yes, manual S3 deletion is the right path at this point, but do it as a full wipe of the repository prefix rather than selectively, partial deletion risks leaving the repo in an even more inconsistent state.

Steps:

  1. In S3, navigate to the bucket and prefix where this repository lives (the location value from your PUT /_snapshot/<repo> settings).

  2. Delete all objects under that prefix (or just move the prefix if you want to preserve the data as a backup).

  3. In OpenSearch, delete the repository registration:

    DELETE /_snapshot/<repo>

  4. Re-register it pointing to the same (now empty) prefix:

    PUT /_snapshot/<repo>
{ "type": "s3", "settings": { ... same settings ... } }

Before doing this, it’s worth understanding what caused the data loss, a missing .dat file alongside missing index directories suggests something deleted S3 objects outside of OpenSearch (S3 lifecycle rule, accidental delete, bucket replication issue, etc.). If that root cause isn’t fixed, the same thing will happen to your next set of snapshots.

Hope this helps