OpenSearch 2.19&3.8 Remote-backed storage(S3 compatible storage) error with x-amz-trailer

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):OpenSearch 2.19.6 and OpenSearch 3.8.0

Describe the issue: I have a Dell PowerScale storage, which has a huge storage capacity and can provide S3 compatible storage services. So I attempted to deploy it following the Remote-backed storage architecture.

Both deployment versions were successful, but each time when writing data in the test (the data source was from vector 0.59), the total documents counted by Indexes would increase to over 300,000 and then stop growing.

Checking KB of DELL, it does not support the x-amz-trailer feature. It is recommended that the client disable this feature.

However, Opensearch seems unable to disable the aforementioned feature.Then, I attempted to trigger a snapshot to my object storage, and it was successful.

So, does the current or future version support S3 compatible storage without the x-amz-trailer feature and deploy the Remote-backed storage architecture?

Configuration:

# ===== Remote Store in “/etc/opensearch/opensearch.yml”=====

 node.attr.remote_store.segment.repository: powerscale-segment
node.attr.remote_store.translog.repository: powerscale-translog
node.attr.remote_store.state.repository: powerscale-state

cluster.remote_store.state.enabled: true
cluster.remote_store.publication.enabled: true
node.attr.remote_store.repository.powerscale-segment.type: s3
node.attr.remote_store.repository.powerscale-segment.settings.bucket: opensearch-remote
node.attr.remote_store.repository.powerscale-segment.settings.base_path: segment
node.attr.remote_store.repository.powerscale-segment.settings.endpoint: repository.xxx.net:9020
node.attr.remote_store.repository.powerscale-segment.settings.protocol: http
node.attr.remote_store.repository.powerscale-segment.settings.region: AAA
node.attr.remote_store.repository.powerscale-segment.settings.path_style_access: true


node.attr.remote_store.repository.powerscale-translog.type: s3
node.attr.remote_store.repository.powerscale-translog.settings.bucket: opensearch-remote
node.attr.remote_store.repository.powerscale-translog.settings.base_path: translog
node.attr.remote_store.repository.powerscale-translog.settings.endpoint: repository.xxx.net:9020
node.attr.remote_store.repository.powerscale-translog.settings.protocol: http
node.attr.remote_store.repository.powerscale-translog.settings.region: AAA
node.attr.remote_store.repository.powerscale-translog.settings.path_style_access: true


node.attr.remote_store.repository.powerscale-state.type: s3
node.attr.remote_store.repository.powerscale-state.settings.bucket: opensearch-remote
node.attr.remote_store.repository.powerscale-state.settings.base_path: cluster-state
node.attr.remote_store.repository.powerscale-state.settings.endpoint: repository.xxx.net:9020
node.attr.remote_store.repository.powerscale-state.settings.protocol: http
node.attr.remote_store.repository.powerscale-state.settings.region: AAA
node.attr.remote_store.repository.powerscale-state.settings.path_style_access: true

Relevant Logs or Screenshots:

/var/log/opensearch/opensearch.log:Caused by: software.amazon.awssdk.services.s3.model.S3Exception: x-amz-trailer is n
ot supported. (Service: S3, Status Code: 501, Request ID: 572378068)
/var/log/opensearch/opensearch.log:   at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResp
onseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
/var/log/opensearch/opensearch.log:   at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError
(ResponseHandler.java:250)
2

this looks like the backend, not your yml. x-amz-trailer is the trailing checksum the aws sdk adds on the segment uploads for remote backed storage, and a 501 there means powerscale s3 rejected that signing style outright. unless dell supports aws chunked trailer checksums on your version, the practical options are a gateway in front that handles it or a different remote store backend, because no path style or provider setting changes that header.

3

Hi @lnanhao,

I tested Remote-backed Storage with OpenSearch 3.8.0 and MinIO, which worked without issues. However, when I introduced a proxy that rejects requests containing the x-amz-trailer header, I was able to reproduce the HTTP 501 error reported in your case.

Based on my investigation and testing, OpenSearch currently does not provide a configuration option to disable the x-amz-trailer header for Remote-backed Storage.

If you would like to request this functionality, please consider opening a feature request on GitHub and sharing the issue link here for traceability:

Kind regards,

Yanlin.