OpenSearch Snapshot Repository with custom S3 endpoint: IMDS credentials have been disabled by environment variable or system property

OpenSearch
1

Versions
OpenSearch 2.19 on Kubernetes with the Chart v2.32.0

Describe the issue:
I am following this documentation Take and restore snapshots - OpenSearch Documentation in order to create new repository for snapshots with a custom S3 endpoint but got this exception

[2025-07-08T15:53:57,877][WARN ][r.suppressed             ] [opensearch-snapshots-master-0] path: /_snapshot/sandbox-demo, params: {repository=sandbox-demo}
org.opensearch.repositories.RepositoryVerificationException: [sandbox-demo] path [opensearch-snapshots/sandbox-demo/snapshots] is not accessible on cluster-manager node
        at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:3055) ~[opensearch-2.19.1.jar:2.19.1]
        at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:380) ~[opensearch-2.19.1.jar:2.19.1]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014) [opensearch-2.19.1.jar:2.19.1]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.19.1.jar:2.19.1]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.io.IOException: Unable to upload object [opensearch-snapshots/sandbox-demo/snapshots/tests-XsiNTqLERF6AUThPFPJnxw/master.dat] using a single upload
        at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:558) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlobWithMetadata$1(S3BlobContainer.java:198) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
        at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:62) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlobWithMetadata(S3BlobContainer.java:196) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:180) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:373) ~[?:?]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:3047) ~[opensearch-2.19.1.jar:2.19.1]
        ... 6 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: IMDS credentials have been disabled by environment variable or system property.
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[?:?]
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) ~[?:?]
        at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:144) ~[?:?]
        at software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$3(CachedSupplier.java:284) ~[?:?]
        at software.amazon.awssdk.utils.cache.NonBlocking.fetch(NonBlocking.java:141) ~[?:?]
        at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:199) ~[?:?]
        at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:128) ~[?:?]
        at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.resolveCredentials(InstanceProfileCredentialsProvider.java:139) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
        at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:56) ~[?:?]
        at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.resolveCredentials(S3Service.java:476) ~[?:?]
        at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:50) ~[?:?]
        at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100) ~[?:?]
        at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77) ~[?:?]
        at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:123) ~[?:?]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69) ~[?:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78) ~[?:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[?:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[?:?]
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[?:?]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[?:?]
        at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:9394) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$24(S3BlobContainer.java:555) ~[?:?]
        at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:71) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
        at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:70) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:554) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlobWithMetadata$1(S3BlobContainer.java:198) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
        at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:62) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlobWithMetadata(S3BlobContainer.java:196) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:180) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:373) ~[?:?]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:3047) ~[opensearch-2.19.1.jar:2.19.1]

Configuration:
The keystore has been created with the bucket access key and secret key:

 /usr/share/opensearch/bin/opensearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

and the nodes have been started with the opensearch.yml containing these lines

cat config/opensearch.yml | grep s3
s3.client.default.endpoint: my.custom.endpoint.com
s3.client.default.protocol: http
s3.client.default.region: us-east-2

the json used for the REST API request PUT /_snapshot/

{
  "type": "s3",
  "settings": {
    "bucket": "opensearch-sandbox-snapshots",
    "client": "default",
    "base_path": "opensearch-snapshots/sandbox-demo/snapshots"
  }
}

As this is a custom S3 endpoint, I also set the AWS_EC2_METADATA_DISABLED env var

printenv AWS_EC2_METADATA_DISABLED
true

Additional information:

My issue is similar to this one Issues setting up s3 bucket for snapshot repository using IAM Role but I don’t have the exception Unable to load credentials from any of the providers in the chain ... as mentioned on the last comment.

Any help would be appreciated. Thanks

2

In fact doing the command like it was mentioned in this doc Take and restore snapshots - OpenSearch Documentation does not work.

I set them as values in a secret and passed it into the values.yaml keystore field resolves the issue. The repo was created correctly.

3

@Hung The S3 configuration described in the OpenSearch documentation works for the binary and Docker implementations.

This is my working Minio S3 storage.

ARG OS_VER

FROM opensearchproject/opensearch:${OS_VER}

ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY

# Environment variables (Persist at runtime)
ENV AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
ENV AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
RUN /usr/share/opensearch/bin/opensearch-plugin install --batch repository-s3
RUN /usr/share/opensearch/bin/opensearch-keystore create

# Add credentials dynamically at runtime
RUN echo $AWS_ACCESS_KEY_ID | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.access_key
RUN echo $AWS_SECRET_ACCESS_KEY | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.secret_key

ARG MINIO_ENDPOINT
ARG MINIO_PROTOCOL

RUN echo "s3.client.default.endpoint: $MINIO_ENDPOINT" >> /usr/share/opensearch/config/opensearch.yml
RUN echo "s3.client.default.protocol: $MINIO_PROTOCOL" >> /usr/share/opensearch/config/opensearch.yml
RUN echo "s3.client.default.region: eu-west-1" >> /usr/share/opensearch/config/opensearch.yml
RUN echo "s3.client.default.path_style_access: true" >> /usr/share/opensearch/config/opensearch.yml

Helm charts have their own rules, and you must use the keystore option to add the AWS Access and Secret keys through the secret.

I agree that documentation is lacking this information.

1 Like