Issues setting up s3 bucket for snapshot repository using IAM Role

OpenSearch
1

I’m trying to setup an s3 bucket snapshot repository, using IAM roles to allow access to the bucket. I am unable to get the repository setup to complete, it gives this error:

        "reason" : "sdk_client_exception: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@6669063f: access denied (\"java.io.FilePermission\" \"/usr/share/opensearch/.aws/credentials\" \"read\"), com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@2ab72ee2: Failed to connect to service endpoint: ]"

I have added the Role ARN and Role Session Name in multiple ways, environment variables, in the opensearch keystore, defining the role in the request to create the repository, etc.

No matter how I define the role information, I keep getting the same error, and the repository setup will not complete.

Any ideas what I might be missing to get this to work? The opensearch cluster is running on AWS ECS.

2

@tfmm could you please share a) what OpenSearch version you are running b) what settings you are using to configure IAM? Thank you.

3

Opensearch 2.2.0, built off the official docker image, just with our certificates and such added.

As far as IAM Settings, the role has complete control over the bucket, and I have confirmed that awscli commands to upload / otherwise interact with the bucket work from the container. I’ll include the full policy below.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:PutAnalyticsConfiguration",
                "s3:GetObjectVersionTagging",
                "s3:ReplicateObject",
                "s3:GetObjectAcl",
                "s3:PutLifecycleConfiguration",
                "s3:GetObjectVersionAcl",
                "s3:PutObjectTagging",
                "s3:HeadBucket",
                "s3:DeleteObject",
                "s3:DeleteObjectTagging",
                "s3:GetBucketPolicyStatus",
                "s3:GetBucketWebsite",
                "s3:DeleteObjectVersionTagging",
                "s3:GetBucketNotification",
                "s3:GetReplicationConfiguration",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetAnalyticsConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetLifecycleConfiguration",
                "s3:ListBucketByTags",
                "s3:GetInventoryConfiguration",
                "s3:GetBucketTagging",
                "s3:PutAccelerateConfiguration",
                "s3:DeleteObjectVersion",
                "s3:GetBucketLogging",
                "s3:ListBucketVersions",
                "s3:ReplicateTags",
                "s3:RestoreObject",
                "s3:ListBucket",
                "s3:GetAccelerateConfiguration",
                "s3:GetBucketPolicy",
                "s3:GetEncryptionConfiguration",
                "s3:GetObjectVersionTorrent",
                "s3:AbortMultipartUpload",
                "s3:GetBucketRequestPayment",
                "s3:GetObjectTagging",
                "s3:GetMetricsConfiguration",
                "s3:GetBucketPublicAccessBlock",
                "s3:ListBucketMultipartUploads",
                "s3:PutObjectVersionTagging",
                "s3:GetBucketVersioning",
                "s3:GetBucketAcl",
                "s3:GetObjectTorrent",
                "s3:GetAccountPublicAccessBlock",
                "s3:ListAllMyBuckets",
                "s3:GetBucketCORS",
                "s3:GetBucketLocation",
                "s3:ReplicateDelete",
                "s3:GetObjectVersion"
            ],
            "Effect": "Allow",
            "Resource": [
                "ARN_OF_BUCKET/*",
                "ARN_OF_BUCKET"
            ]
        },
        {
            "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets"
      ],

            "Resource": [
		"*"
	    ]
        }
    ]
}
4

I just realized you may be asking for what opensearch settings I’m using to configure IAM. I have tried multiple:

Keystore entries:

s3.client.default.role_arn
s3.client.default.role_session_name

Environment variables:

AWS_ROLE_SESSION_NAME
AWS_ROLE_ARN

And several combinations of these. I attempted to follow the instructions here: Take and restore snapshots - OpenSearch documentation

But am still having no luck.

5

Thanks @tfmm, the steps / settings are looking right, I suspect that the keystore you are configuring is not being picked by OpenSearch, could you please confirm that is / is not the case?

6

@reta could you give me a hand knowing how to confirm if the keystore is being picked up?

7

It should be in the folder pointed by OPENSEARCH_PATH_CONF, the exact location of this folder is available in logs, the server prints it on startup. Thank you.

8

Thanks. So the keystore is in the config path specified on startup:

[2022-09-26T15:54:46,830][INFO ][o.o.s.OpenSearchSecurityPlugin] [] OpenSearch Config path is /usr/share/opensearch/config


[opensearch@ config]$ pwd
/usr/share/opensearch/config
[opensearch@ config]$ ls -lah *.keystore
-rw-rw---- 1 opensearch opensearch 335 Sep 26 15:54 opensearch.keystore
[opensearch@ config]$ opensearch-keystore list
keystore.seed
s3.client.default.role_arn
s3.client.default.role_session_name
9

Thanks @tfmm, kystore is looking right, do you mind please to share the stack trace from OpenSearch logs relevant to S3 credentials issue, it would help to identify path being taken for credentials resolution, thank you.

10

I’m also getting this error. opensearch-keystore list command gives the exact output as the above reply. The error stacktrace in opensearch master nodes is the following

[2025-01-21T06:32:21,521][WARN ][r.suppressed             ] [opensearch-master-0] path: /_snapshot/test1, params: {repository=test1}
org.opensearch.repositories.RepositoryVerificationException: [test1] path [my/snapshot/directory] is not accessible on cluster-manager node
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2005) ~[opensearch-2.15.0.jar:2.15.0]
	at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:373) ~[opensearch-2.15.0.jar:2.15.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.15.0.jar:2.15.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.15.0.jar:2.15.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.io.IOException: Unable to upload object [my/snapshot/directory/tests-bMlhzvGQQ8qR4neIR3GCAA/master.dat] using a single upload
	at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:654) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlobWithMetadata$1(S3BlobContainer.java:200) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:62) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlobWithMetadata(S3BlobContainer.java:198) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:182) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:375) ~[?:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1997) ~[opensearch-2.15.0.jar:2.15.0]
	... 6 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])): Profile file contained no credentials for profile 'default': ProfileFile(profilesAndSectionsMap=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[?:?]
	at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:117) ~[?:?]
	at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45) ~[?:?]
	at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:128) ~[?:?]
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:50) ~[?:?]
	at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100) ~[?:?]
	at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77) ~[?:?]
	at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:123) ~[?:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[?:?]
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[?:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[?:?]
	at software.amazon.awssdk.services.sts.DefaultStsClient.assumeRole(DefaultStsClient.java:268) ~[?:?]
	at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.getUpdatedCredentials(StsAssumeRoleCredentialsProvider.java:73) ~[?:?]
	at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.updateSessionCredentials(StsCredentialsProvider.java:88) ~[?:?]
	at software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$3(CachedSupplier.java:284) ~[?:?]
	at software.amazon.awssdk.utils.cache.CachedSupplier$PrefetchStrategy.fetch(CachedSupplier.java:420) ~[?:?]
	at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:199) ~[?:?]
	at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:128) ~[?:?]
	at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.resolveCredentials(StsCredentialsProvider.java:101) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:56) ~[?:?]
	at org.opensearch.repositories.s3.S3Service$PrivilegedSTSAssumeRoleSessionCredentialsProvider.resolveCredentials(S3Service.java:502) ~[?:?]
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:50) ~[?:?]
	at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100) ~[?:?]
	at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77) ~[?:?]
	at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:123) ~[?:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[?:?]
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[?:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[?:?]
	at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:9394) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$32(S3BlobContainer.java:651) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:71) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:70) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:650) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlobWithMetadata$1(S3BlobContainer.java:200) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:62) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlobWithMetadata(S3BlobContainer.java:198) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:182) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:375) ~[?:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1997) ~[opensearch-2.15.0.jar:2.15.0]
	... 6 more