Opensearch 2.16 snapshot to s3 repository using repository-s3 plugin

Plugins Index Management
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.16

Describe the issue: I am trying to do a backup of snapshots to an s3 repository using the credentials from the EKS cluster, I am getting the following error:
{"error":{"root_cause":[{"type":"repository_exception","reason":"[s3-bucket-stg]
Could not determine repository generation from root
blobs"}],"type":"repository_exception","reason":"[s3-bucket-stg] Could
not determine repository generation from root
blobs","caused_by":{"type":"i_o_exception","reason":"Exception when
listing blobs by prefix
[/snapshots/index-]","caused_by":{"type":"sdk_client_exception","reason":"Failed
to load credentials from
IMDS.","caused_by":{"type":"sdk_service_exception","reason":"Unauthorized"}}}},"status":500}

Configuration: Configured IRSA in the kubernetes pod and the region in opensearch.yml for the plugin repository-s3

Also tried to had this configuration to the pod init container:
command:
48 - sh
49 - -c
E 50 - chown -R 1000:1000 /var/lib/wazuh-indexer && mkdir -p /usr/share/wazuh-indexer/repositiry-s3/ && ln -s $AWS_WEB_IDENTITY_TOKEN_FILE /usr/share/wazuh-indexer/repositiry-s3/aws-web-identity-token-file

Tried using environment variables like AWS_ACCESS_KEY_ID and disabling the IMDS using metadata but that also did not work. Tried adding password and login to the opensearch keystore and always get an error.

Relevant Logs or Screenshots:

{"error":{"root_cause":[{"type":"repository_exception","reason":"[s3-bucket-stg]
 Could not determine repository generation from root 
blobs"}],"type":"repository_exception","reason":"[s3-bucket-stg] Could 
not determine repository generation from root 
blobs","caused_by":{"type":"i_o_exception","reason":"Exception when 
listing blobs by prefix 
[/snapshots/index-]","caused_by":{"type":"sdk_client_exception","reason":"Failed
 to load credentials from 
IMDS.","caused_by":{"type":"sdk_service_exception","reason":"Unauthorized"}}}},"status":500}

org.opensearch.transport.RemoteTransportException: [wazuh-indexer-0][100.66.189.252:9300][cluster:admin/snapshot/get]
Caused by: org.opensearch.repositories.RepositoryException: [s3-bucket-stg] Could not determine repository generation from root blobs
	at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:2115) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.16.0.jar:2.16.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.io.IOException: Exception when listing blobs by prefix [/snapshots/index-]
	at org.opensearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:534) ~[?:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2755) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2737) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:2112) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.16.0.jar:2.16.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to load credentials from IMDS.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[?:?]
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47) ~[?:?]
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:157) ~[?:?]
	at software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$3(CachedSupplier.java:284) ~[?:?]
	at software.amazon.awssdk.utils.cache.NonBlocking.fetch(NonBlocking.java:141) ~[?:?]
	at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:199) ~[?:?]
	at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:128) ~[?:?]
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.resolveCredentials(InstanceProfileCredentialsProvider.java:139) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:56) ~[?:?]
	at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.resolveCredentials(S3Service.java:473) ~[?:?]
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:50) ~[?:?]
	at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100) ~[?:?]
	at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77) ~[?:?]
	at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:123) ~[?:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[?:?]
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[?:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[?:?]
	at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:6538) ~[?:?]
	at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:153) ~[?:?]
	at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:144) ~[?:?]
	at software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeListing$29(S3BlobContainer.java:580) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:56) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:576) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:568) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:529) ~[?:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2755) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2737) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:2112) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.16.0.jar:2.16.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: sdk_service_exception: Unauthorized
	at software.amazon.awssdk.core.exception.SdkServiceException$BuilderImpl.build(SdkServiceException.java:276) ~[?:?]
	at software.amazon.awssdk.regions.util.HttpResourcesUtils.handleErrorResponse(HttpResourcesUtils.java:171) ~[?:?]
	at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:132) ~[?:?]
	at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91) ~[?:?]
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.lambda$getSecurityCredentials$3(InstanceProfileCredentialsProvider.java:254) ~[?:?]
	at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108) ~[?:?]
	at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136) ~[?:?]
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getSecurityCredentials(InstanceProfileCredentialsProvider.java:254) ~[?:?]
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.createEndpointProvider(InstanceProfileCredentialsProvider.java:202) ~[?:?]
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:148) ~[?:?]
	at software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$3(CachedSupplier.java:284) ~[?:?]
	at software.amazon.awssdk.utils.cache.NonBlocking.fetch(NonBlocking.java:141) ~[?:?]
	at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:199) ~[?:?]
	at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:128) ~[?:?]
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.resolveCredentials(InstanceProfileCredentialsProvider.java:139) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:56) ~[?:?]
	at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.resolveCredentials(S3Service.java:473) ~[?:?]
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:50) ~[?:?]
	at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100) ~[?:?]
	at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77) ~[?:?]
	at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:123) ~[?:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[?:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[?:?]
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[?:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[?:?]
	at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:6538) ~[?:?]
	at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:153) ~[?:?]
	at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:144) ~[?:?]
	at software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeListing$29(S3BlobContainer.java:580) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:56) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:576) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:568) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:529) ~[?:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2755) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2737) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:2112) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) ~[opensearch-2.16.0.jar:2.16.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.16.0.jar:2.16.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
2

@erdna Did you deploy your OpenSearch cluster with official OpenSearch Helm charts?

Take a look at my previous comments on the same error.

3

I understand but I am using this as a sub-component of wazuh-indexer

according to the code here I see that this uses the keystore but I cannot make it work.
Do I have to set it up before I start opensearch, or can I add the items (Credentials) there after opensearch has started and the code will read it in the keystore?

Best Regards,

4

@erdna OpenSearch is reading keystore during the startup process. You would need to restart the OS service to read it again.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.