Repository-s3: Delete snapshot operation times out without issuing DeleteObject request to MinIO

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.3.2

Describe the issue: 1. Data is not getting deleted on s3 storage(minio) when the Delete snapshot command is run.

  1. Have tried checking via _verify command too, minio’s logs shows no delete command being sent to minio…

Configuration:
OpenSearch Version: 3.3.2

  1. repository-s3 Plugin: 3.3.2

  2. Deployment: Kubernetes (Helm)

  3. Storage Backend: MinIO

  4. MinIO Authentication: Root user (minioadmin)

  5. Repository Type: S3

    Repository cofiguration:

    {
    “type”: “s3”,
    “settings”: {
    “bucket”: “opensearch-snapshots”,
    “base_path”: “abc-logs”,
    “client”: “default”,
    “compress”: true,
    “disable_chunked_encoding”: true,
    “connect_timeout”: “60s”,
    “read_timeout”: “60s”,
    “client.max_retries”: “3”
    }
    }

S3 configuration in `opensearch.yml`:

s3.client.default.path_style_access: true
s3.client.default.endpoint: “:”
s3.client.default.protocol: “https”
s3.client.default.region: “us-east-1”
s3.client.default.read_timeout: “60s”
s3.client.default.use_throttle_retries: true
s3.client.default.max_retries: 5

Relevant Logs or Screenshots:
Tried running the cleanup command, but it always shows the following error(given no other task is running, minio’s user is the root user) :

{
"error": {
"root_cause": [
{
"type": "illegal_state_exception",
"reason": "Cannot cleanup [minio-cold-abc] - a repository cleanup is already in-progress in [{"repository_cleanup":[{"repository":"minio-cold-abc"}]}]"
}
],
"type": "illegal_state_exception",
"reason": "Cannot cleanup [minio-cold-abc] - a repository cleanup is already in-progress in [{"repository_cleanup":[{"repository":"minio-cold-abc"}]}]"
},
"status": 500
}

Repository verification fails with:
{
"error": {
"type": "repository_verification_exception",
"reason": "[minio-cold-abc] cannot delete test data at [abc-logs]",
"caused_by": {
"type": "i_o_exception",
"reason": "Delete operation timed out after 30 seconds",
"caused_by": {
"type": "timeout_exception",
"reason": "Timeout waiting for task."
}
}
},
"status": 500
}

2

@yashg Do you use a snapshot policy or just a manual snapshot delete action?

3

@yashg I’ve tested your configuration with OS 3.3.2 and MinIO RELEASE.2025-01-20T14-49-07Z.

image
image960×497 30.6 KB

The only differences between our configurations were the protocol and the deployment type: I used http instead of https, and Docker instead of Kubernetes.

I had no issues creating the repository or taking and deleting the snapshot.

4
  1. manual deletion via dev console, data didn’t get deleted from minio
  2. So, tried via _cleanup command, it gives error of already running process:
    {
    "error": {
    "root_cause": [
    {
    "type": "illegal_state_exception",
    "reason": "Cannot cleanup [minio-cold-abc] - a repository cleanup is already in-progress in [{"repository_cleanup":[{"repository":"minio-cold-abc"}]}]"
    }
    ],
    "type": "illegal_state_exception",
    "reason": "Cannot cleanup [minio-cold-abc] - a repository cleanup is already in-progress in [{"repository_cleanup":[{"repository":"minio-cold-abc"}]}]"
    },
    "status": 500
    }
  3. Tried _verify command too, that one is also giving the error of data could not be deleted from opensearch:
    {
    "error" : {
    "root_cause" : [
    {
    "type" : "repository_verification_exception",
    "reason" : "[minio-cold-abc] cannot delete test data at [abc-logs]"
    }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[minio-cold-abc] cannot delete test data at [abc-logs]",
    "caused_by" : {
    "type" : "i_o_exception",
    "reason" : "Delete operation timed out after 30 seconds",
    "caused_by" : {
    "type" : "timeout_exception",
    "reason" : "Timeout waiting for task."
    }
    }
    },
    "status" : 500
    }
5

Also, our MinIO version is RELEASE.2024-12-18T13-15-44Z, and there aren’t any major differences compared to your MinIO configuration.

I’m not sure what else could be causing this issue. Does the snapshot delete operation typically take significantly longer than other operations, even longer than a restore?

Also, my understanding is that the _verify operation shouldn’t create a large object in MinIO, so deleting it should be relatively quick. Is that correct?

This is my values files for reference: PKIX path building failed when trying to connect Opensearch to minio with https - #6 by yashg

6

@yashg Now I remember. We were working on enabling the HTTPS protocol for S3 snapshots.
I understand that you have two issues: performance and snapshot deletion.

Did you run your Minio on HTTP before, or was the initial deployment HTTPS? Would it be possible to switch to HTTP temporarily for testing?

7

Hi @pablo ,

We have been using MinIO over HTTPS from the beginning. I tested the deployment, but I somehow missed verifying the snapshot deletion step.

Are you suggesting that I temporarily test the same setup over HTTP to determine whether the issue is related to HTTPS? If so, I’ll do that tomorrow (it’s already 9 PM here) and share the results.

Also, aside from the HTTP/HTTPS comparison, could you let me know what other changes or checks you’d like me to perform so we can reach a clear conclusion tomorrow?

Thanks!

8

@yashg I’ve seen a few forum cases related to Minio S3 Snapshots with HTTPS protocol enabled.
I’ve already tested your configuration and couldnt find any issue there.

Do you use that Minio for any other application? If not, have you notice any issues in other apps connected to that Minio instance?