Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.15
Describe the issue:
I set the default access /secret key of S3 and setup repository, the snapshot upload was perfectly working fine until opensearch data nodes are restarted, but getting below exception of _snapshot/abc_bucket/_all response for that index
IndexShardSnapshotFailedException[Failed to write shard level snapshot metadata for [index-snapshot-00.000/x69HqZw1] to [index-9997F3iVT]]; nested: IOException[Unable to upload object [12345/indices/uEsUhj/1/index-9997F3iVT] using a single upload]; nested: SdkClientException[Failed to load credentials from IMDS.]; nested: SdkClientException[The requested metadata is not found at http://zzzzz/meta-data/iam/security-credentials/]
This issue is not resolved until i manually execute “_nodes/reload_secure_settings” , since all this happens in background
ISM policy action is defined as below
“name”: “cold”,
“actions”: [
{
“snapshot”: {
“repository”: “repo name”,
“snapshot”: “index-snapshot”
Please help how to mitigate this problem so as to credentials are intact even after nodes restart
Configuration:
Relevant Logs or Screenshots:
IndexShardSnapshotFailedException[Failed to write shard level snapshot metadata for [index-snapshot-00.000/x69HqZw1] to [index-9997F3iVT]]; nested: IOException[Unable to upload object [12345/indices/uEsUhj/1/index-9997F3iVT] using a single upload]; nested: SdkClientException[Failed to load credentials from IMDS.]; nested: SdkClientException[The requested metadata is not found at http://zzzzz/meta-data/iam/security-credentials/]