Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 2.8.0
Describe the issue:
When trying to create a snapshot repository the following error is observed:
{
"error": {
"root_cause": [
{
"type": "repository_verification_exception",
"reason": "[Search_Backup] path [/] is not accessible on cluster-manager node"
}
],
"type": "repository_verification_exception",
"reason": "[Search_Backup] path [/] is not accessible on cluster-manager node",
"caused_by": {
"type": "i_o_exception",
"reason": "Unable to upload object [//tests-bRSiVpRzT92kRFDqSwguaw/master.dat] using a single upload",
"caused_by": {
"type": "sdk_client_exception",
"reason": "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by": {
"type": "i_o_exception",
"reason": "Read timed out"
}
}
}
},
"status": 500
}
The command used to create the snapshot repository is:
curl -H ‘Content-Type: application/json’ -X POST https://search-host.net:9200/_snapshot/search_backup --data ‘{“type”: “s3”,“settings”: {“bucket”: “search-backup”,“endpoint”: “https://ceph-endpoint.net:443” ,“protocol”: “https”}}’
The Keystore contains the following on all nodes:
/usr/share/opensearch/bin/opensearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key
Each node has been restarted after adding the entries to the Keystore.
The cluster master is trying to connect to 169.254.169.254, which to my understanding is an EC2 Metadata instance. This should only be done if the truststore does not contain the required access and secret key.
Configuration:
opensearch.yml:
s3.client.default.endpoint: https://ceph-endpoint.net
s3.client.default.max_retries: 3 # number of retries if a request fails
s3.client.default.path_style_access: false # whether to use the deprecated path-style bucket URLs.
# You probably don't need to change this value, but for more information, see https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#path-style-access.
s3.client.default.protocol: https # http or https
s3.client.default.read_timeout: 50s # the S3 connection timeout
s3.client.default.use_throttle_retries: true # whether the client should wait a progressively longer amount of time (exponential backoff) between each successive retry
s3.client.default.region: us-east-2 # AWS region to use. For non-AWS S3 storage, this value is required but has no effect.
Is anyone able to advise what is going wrong here?