# What storage service can I use for snapshots? **URL:** \ **Category:** OpenSearch\ **Tags:** configure\ **Created:** [May 26, 2022, 4:09pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716 "2022-05-26T16:09:32Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://forum.opensearch.org/u/reshippie)\ **Post date:** [May 26, 2022, 4:09pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/1 "2022-05-26T16:09:32Z")
I’ve been trying to add a snapshot repository for several weeks now and I’ve come up empty every time. The `repository-s3` plugin for Elasticsearch worked fine for our chosen S3-compatible provider, but does not work with Opensearch. It seems to fail while trying to connect to an AWS service. We host Opensearch ourselves, so that effectively rules out using that plugin. My next attempt was to use `rclone` to mount our Backblaze B2 bucket as a local filesystem on each node in our cluster. That also failed. ```auto [2022-05-25T19:40:15,501][WARN][o.o.r.VerifyNodeRepositoryAction] [esmaster-ams5-0000] [local_security] failed to verify repository org.opensearch.repositories.RepositoryVerificationException: [local_security] a file written by master to the store [/mnt/snapshots/ cannot be accessed on the node [{esmaster-0000}{fxxx}{zyyy}{10.8.5.105}{10.8.5.105:9300}{mr}{shard_indexing_pressure_enabled=true}]. This might indicate that the store [/mnt/snapshots/] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node ``` The bucket is available to all nodes and I can see the temp file that is being created on other nodes, so I can’t figure out what would be causing this error. Has anyone been able to get snapshots to a remote service to work with a self-hosted Opensearch cluster? ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [June 2, 2022, 6:17pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/2 "2022-06-02T18:17:32Z")
Hey @reshippie , Do you provide the endpoint to connect to S3 compatible storage? Mind sharing the plugin configuration (without access/secret keys)? Thank you. ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://forum.opensearch.org/u/reshippie)\ **Post date:** [June 2, 2022, 11:07pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/3 "2022-06-02T23:07:44Z")
{“type”: “s3”, “settings”: { “bucket”: “SOME\_BUCKET\_NAME”, “base\_path”: “SOME\_DIRECTORY”, “endpoint”: “[s3.us-west-002.backblazeb2.com](http://s3.us-west-002.backblazeb2.com)”, “region”: “us-west-002”}} I’ve got the `s3.client.default.access_key` and `s3.client.default.secret_key` saved in the keystore on each node. ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [June 3, 2022, 12:44pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/4 "2022-06-03T12:44:07Z")
Checked the connectivity, seems to be no issues, could you please share the stack trace of the exception that appears in the logs/opensearch.log? Thank you. ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://forum.opensearch.org/u/reshippie)\ **Post date:** [June 3, 2022, 4:13pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/5 "2022-06-03T16:13:55Z")
```auto [2022-05-26T23:49:05,642][WARN][r.suppressed] [esmaster-ams5-0000] path: /_snapshot/ams5_security, params: {repository=ams5_security} org.opensearch.transport.RemoteTransportException: [graylog-ams5-0000][10.8.5.45:9300][cluster:admin/repository/put] Caused by: org.opensearch.repositories.RepositoryVerificationException: [ams5_security] path [PROD-AC-AMS5] is not accessible on master node at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1549) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: java.io.IOException: Unable to upload object [PROD-AC-AMS5/tests-CluPofPPTlG7tTGrUDgM2Q/master.dat] using a single upload at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:379) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?] at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] Caused by: org.opensearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint: at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[?:?] at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?] at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?] at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?] at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?] at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?] at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?] at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?] at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?] at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:55) ~[?:?] at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:294) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$18(S3BlobContainer.java:377) ~[?:?] at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:70) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:69) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:377) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?] at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] Caused by: java.io.IOException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?] at java.net.Socket.connect(Socket.java:609) ~[?:?] at sun.net.NetworkClient.doConnect(NetworkClient.java:177) ~[?:?] at sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[?:?] at sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[?:?] at sun.net.www.http.HttpClient.(HttpClient.java:242) ~[?:?] at sun.net.www.http.HttpClient.New(HttpClient.java:341) ~[?:?] at sun.net.www.http.HttpClient.New(HttpClient.java:362) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1232) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015) ~[?:?] at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[?:?] at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[?:?] at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?] at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?] at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?] at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?] at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?] at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?] at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?] at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?] at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:55) ~[?:?] at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:294) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$18(S3BlobContainer.java:377) ~[?:?] at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:70) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:69) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:377) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?] at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?] at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] ``` ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [June 3, 2022, 7:37pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/6 "2022-06-03T19:37:20Z")
This one is interesting, seems like the plugin does not pick the access\_key and secret\_key but falls back to InstanceProfileCredentialsProvider: > [@reshippie](#): > > ```auto > at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015) ~[?:?] > at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[?:?] > at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[?:?] > at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?] > > ``` Consequently, it uses EC2 auth endpoint, not the one you have configured. The only guess I have is that not all nodes have `s3.client.default.access_key` and `s3.client.default.secret_key` configured in keystores. As an experiment, you could try to send `access_key` and `secret_key` inside `settings` as well (if there are no risks to leak them). ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://forum.opensearch.org/u/reshippie)\ **Post date:** [June 6, 2022, 9:31pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/7 "2022-06-06T21:31:33Z")
I have tried deleting and recreating the keystore with no luck. Updating the config file will require a rolling restart, so that will take a little bit of time to test. ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://forum.opensearch.org/u/reshippie)\ **Post date:** [June 6, 2022, 10:56pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/8 "2022-06-06T22:56:09Z")
It seems that putting the credentials inside of `opensearch.yml` is not allowed. ```auto [2022-06-06T22:53:27,900][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [esmaster-ams5-0000] uncaught exception in thread [main] org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.repositories.s3.S3RepositoryPlugin] ... Caused by: java.lang.IllegalArgumentException: Setting [s3.client.default.access_key] is a secure setting and must be stored inside the OpenSearch keystore, but was found inside opensearch.yml ``` ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [June 7, 2022, 12:13pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/9 "2022-06-07T12:13:54Z")
Correct, only keystore, but you should be able to: > [@reshippie](#): > > {“type”: “s3”, > “settings”: { > “access\_key”: “…”, > “secret\_key”: “…”, > “bucket”: “SOME\_BUCKET\_NAME”, > “base\_path”: “SOME\_DIRECTORY”, > “endpoint”: “[s3.us-west-002.backblazeb2.com](http://s3.us-west-002.backblazeb2.com)”, > “region”: “us-west-002”}} This is just for the sake experiment to find out where the problem might be. ---
**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://forum.opensearch.org/u/reshippie)\ **Post date:** [June 9, 2022, 9:09pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/10 "2022-06-09T21:09:14Z")
We finally made it work. I don’t remember how I initially created `/etc/opensearch/opensearch.keystore` but when I run `/opt/opensearch/bin/opensearch-keystore create` it puts the new keystore file in `/opt/opensearch/config` So I removed all of the credentials from the keystore, made sure it was created in the `/opt/opensearch` subdirectory on every node, then copied that into `/etc/opensearch`, reloaded settings via the API, then I was actually able to create a repository and take a snapshot. ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [June 10, 2022, 12:04pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/11 "2022-06-10T12:04:56Z")
Thanks for the @reshippie ---
**Author:** ![punitmundra](https://avatars.discourse-cdn.com/v4/letter/p/53a042/32.png) [@punitmundra](https://forum.opensearch.org/u/punitmundra)\ **Post date:** [August 22, 2022, 4:03pm UTC](https://forum.opensearch.org/t/what-storage-service-can-i-use-for-snapshots/9716/12 "2022-08-22T16:03:04Z")
Aug 16 18:05:47 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): [2022-08-16T18:05:47,425][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [ip-10-1-1-222] this request accesses system indices: [.opendistro\_security], but in a future major version, direct access to system indices will be prevented by default Aug 16 18:05:47 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): [2022-08-16T18:05:47,425][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [ip-10-1-1-222] this request accesses system indices: [.opendistro\_security], but in a future major version, direct access to system indices will be prevented by default Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): [2022-08-16T18:06:05,631][WARN][o.o.r.s.S3BlobStore] [ip-10-1-1-222] Expected request count to be tracked for request [PUT [https://a2ha-bloomberg.s3.amazonaws.com](https://a2ha-bloomberg.s3.amazonaws.com) /Aug16/opensearch/automate-elasticsearch-data/chef-automate-es6-event-feed-service/tests-B30bi3D4QNm7tNZm58zJhw/master.dat Headers: (Content-Length: 22, Content-Type: application/octet-stream, x-amz-acl: private, x-amz-storage-class: STANDARD, ) ] but found not count. Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): [2022-08-16T18:06:05,631][WARN][r.suppressed] [ip-10-1-1-222] path: /\_snapshot/chef-automate-es6-event-feed-service, params: {verify=true, repository=chef-automate-es6-event-feed-service} Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): org.opensearch.repositories.RepositoryVerificationException: [chef-automate-es6-event-feed-service] path [Aug16/opensearch/automate-elasticsearch-data/chef-automate-es6-event-feed-service] is not accessible on master node Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1549) ~[opensearch-1.2.4.jar:1.2.4] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.2.4.jar:1.2.4] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) [opensearch-1.2.4.jar:1.2.4] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) [opensearch-1.2.4.jar:1.2.4] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at java.lang.Thread.run(Thread.java:829) [?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Caused by: java.io.IOException: Unable to upload object [Aug16/opensearch/automate-elasticsearch-data/chef-automate-es6-event-feed-service/tests-B30bi3D4QNm7tNZm58zJhw/master.dat] using a single upload Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:379) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at java.security.AccessController.doPrivileged(Native Method) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.2.4.jar:1.2.4] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): … 6 more Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Caused by: com.amazonaws.SdkClientException: The requested metadata is not found at [http://169.254.169.254/latest/meta-data/iam/security-credentials/](http://169.254.169.254/latest/meta-data/iam/security-credentials/) Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:89) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at java.security.AccessController.doPrivileged(Native Method) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:55) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:246) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5842) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:5815) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5038) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$18(S3BlobContainer.java:377) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:70) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at java.security.AccessController.doPrivileged(Native Method) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:69) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:377) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at java.security.AccessController.doPrivileged(Native Method) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.2.4.jar:1.2.4] Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): … 6 more Aug 16 18:06:17 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): [2022-08-16T18:06:17,486][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [ip-10-1-1-222] this request accesses system indices: [.opendistro\_security], but in a future major version, direct access to system indices will be prevented by default Aug 16 18:06:17 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): [2022-08-16T18:06:17,486][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [ip-10-1-1-222] this request accesses system indices: [.opendistro\_security], but in a future major version, direct access to system indices will be prevented by default Aug 16 18:06:21 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): [2022-08-16T18:06:21,402][INFO][o.o.j.s.JobSweeper] [ip-10-1-1-222] Running full sweep