OpenSearch 3.1.0 repository-s3 error with x-amz-trailer

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

3.1.0

Describe the issue:

After upgrading from version 2.17.1 to 3.1.0 and installing the S3 plugin, an error occurs when trying to add a repository or take snapshot.

Configuration:

s3.client.default.endpoint: s3_service
s3.client.default.region: region
s3.client.default.protocol: https

Relevant Logs or Screenshots:

{
“error”:{
“root_cause”:[
{
“type”:“repository_verification_exception”,
“reason”:“[test] path [test-restore] is not accessible on cluster-manager node”
}
],
“type”:“repository_verification_exception”,
“reason”:“[test] path [test-restore] is not accessible on cluster-manager node”,
“caused_by”:{
“type”:“i_o_exception”,
“reason”:“Unable to upload object [test-restore/tests-bGvIIxyISEWGstdP7Qn0rg/master.dat] using a single upload”,
“caused_by”:{
“type”:“s3_exception”,
“reason”:“Invalid trailing header names in x-amz-trailer. (Service: S3, Status Code: 400, Request ID: 42d9e331-e9ac-1faf-817e-1070fdfaaaa4) (SDK Attempt Count: 1)”
}
}
},
“status”:500
}

I tried upgrading to version 3.2.0, but it did not resolve the issue.
Please help me resolve this issue, I would be very grateful for any assistance.

2

@Maksim Is this AWS S3 or other vendor i.e. Minio?

3

@Maksim There is also a GitHub issue related to your problem that was reported a few days ago.

1 Like
4

@pablo

  1. Cloud provider with s3 service, it’s not minio, but it’s not Amazon S3 either. The provider promises compatibility with Amazon, but they probably can’t update the service as quickly.
  2. Yes, I saw this issue, but there are no comments with a possible solution.
5

@Maksim I’ve tested OpenSearch s3-repository plugin with AWS S3 and Minio. All OS 3.x versions worked without issue.

Do you use any proxy between OpenSearch and your S3 provider?

6

Unfortunately, I don’t know if there is a proxy in the provider’s infrastructure
I have never added any proxy settings to the OpenSearch configuration
I found a similar problem in GitHub

7

@Maksim, There was a fix to support S3-compatible storage created recently and is waiting to be merged.

8

@pablo
Yes, I resolve the issue with solutions from this topic.
Thanks for your help!