Snapshot repository timeouts after converting to Opensearch

OpenSearch
1

I have 4 clusters that I recently converted from Elasticsearch with the OpenDistro plugins to Opensearch. Unfortunately I just realized that none of our snapshots are accessible. I get timeout errors when I hit the _snapshot/REPO/_all endpoint

{
  "error" : {
    "root_cause" : [
      {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: Failed to connect to service endpoint: "
      }
    ],
    "type" : "repository_exception",
    "reason" : "[b2_security] Unexpected exception when loading repository data",
    "caused_by" : {
      "type" : "sdk_client_exception",
      "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
      "caused_by" : {
        "type" : "i_o_exception",
        "reason" : "connect timed out"
      }
    }
  },
  "status" : 500
}

I get a similar error when I try and create a new repository.

2

I think this is related to the fact that I’m not using AWS.
I’m seeing errors in the logs from the repository-s3 plugin about not being able to locate things.
I’ve just tried creating /home/opensearch/.aws/config and set AWS_EC2_METADATA_DISABLED and I still can’t create an endpoint.

It is possible in Opensearch to write snapshots to S3-compatible endpoints, just like in Elasticsearch, right?

3

I got the same error when I tried to create an s3 snapshot repo.

PUT _snapshot/s3_repository
  {
     "type": "s3",
      "settings": {
         "bucket": "opensearch-repo",
         "endpoint": "seaweedfs:8333",
         "path_style_access": "true",
         "protocol": "http"
      }
  }

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[s3_repository] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[s3_repository] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-31RLM5EsQl2cqxpVpaZWYQ/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "connect timed out"
        }
      }
    }
  },
  "status" : 500
}

OpenSearch version: 1.3.2.
I use the seaweedfs as a s3 storage system
And I have an Elasticsearch 7.13.4 cluster running on the same Kubernetes cluster, and s3 repo works perfectly fine there

4

Hi @deng47,

Do you provide the endpoint to connect to S3 compatible storage? Mind sharing the plugin configuration (without access/secret keys)? Thank you.

5

I built my own local S3 storage system with seaweeds
And I created the repo successfully(have not yet tried taking a snapshot thou)
Please check this issue [BUG][OpenSearch-1.11.1] the opensearchJavaOpts variable doesn't work · Issue #271 · opensearch-project/helm-charts · GitHub