What storage service can I use for snapshots?

reshippie · May 26, 2022, 4:09pm

I’ve been trying to add a snapshot repository for several weeks now and I’ve come up empty every time.

The repository-s3 plugin for Elasticsearch worked fine for our chosen S3-compatible provider, but does not work with Opensearch. It seems to fail while trying to connect to an AWS service. We host Opensearch ourselves, so that effectively rules out using that plugin.

My next attempt was to use rclone to mount our Backblaze B2 bucket as a local filesystem on each node in our cluster. That also failed.

[2022-05-25T19:40:15,501][WARN ][o.o.r.VerifyNodeRepositoryAction] [esmaster-ams5-0000] [local_security] failed to verify repository
org.opensearch.repositories.RepositoryVerificationException: [local_security] a file written by master to the store [/mnt/snapshots/ cannot be accessed on the node [{esmaster-0000}{fxxx}{zyyy}{10.8.5.105}{10.8.5.105:9300}{mr}{shard_indexing_pressure_enabled=true}]. This might indicate that the store [/mnt/snapshots/] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node

The bucket is available to all nodes and I can see the temp file that is being created on other nodes, so I can’t figure out what would be causing this error.

Has anyone been able to get snapshots to a remote service to work with a self-hosted Opensearch cluster?

reta · June 2, 2022, 6:17pm

Hey @reshippie ,

Do you provide the endpoint to connect to S3 compatible storage? Mind sharing the plugin configuration (without access/secret keys)? Thank you.

reshippie · June 2, 2022, 11:07pm

{“type”: “s3”,
“settings”: {
“bucket”: “SOME_BUCKET_NAME”,
“base_path”: “SOME_DIRECTORY”,
“endpoint”: “s3.us-west-002.backblazeb2.com”,
“region”: “us-west-002”}}

I’ve got the s3.client.default.access_key and s3.client.default.secret_key saved in the keystore on each node.

reta · June 3, 2022, 12:44pm

Checked the connectivity, seems to be no issues, could you please share the stack trace of the exception that appears in the logs/opensearch.log? Thank you.

reshippie · June 3, 2022, 4:13pm

[2022-05-26T23:49:05,642][WARN ][r.suppressed             ] [esmaster-ams5-0000] path: /_snapshot/ams5_security, params: {repository=ams5_security}
org.opensearch.transport.RemoteTransportException: [graylog-ams5-0000][10.8.5.45:9300][cluster:admin/repository/put]
Caused by: org.opensearch.repositories.RepositoryVerificationException: [ams5_security] path [PROD-AC-AMS5] is not accessible on master node
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1549) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.IOException: Unable to upload object [PROD-AC-AMS5/tests-CluPofPPTlG7tTGrUDgM2Q/master.dat] using a single upload
	at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:379) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.opensearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[?:?]
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?]
	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?]
	at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?]
	at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
	at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:55) ~[?:?]
	at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:294) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$18(S3BlobContainer.java:377) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:70) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:69) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:377) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.io.IOException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]
	at java.net.Socket.connect(Socket.java:609) ~[?:?]
	at sun.net.NetworkClient.doConnect(NetworkClient.java:177) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:341) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:362) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1232) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015) ~[?:?]
	at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[?:?]
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[?:?]
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?]
	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?]
	at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?]
	at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
	at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:55) ~[?:?]
	at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:294) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$18(S3BlobContainer.java:377) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:70) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:69) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:377) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
	at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]

reta · June 3, 2022, 7:37pm

This one is interesting, seems like the plugin does not pick the access_key and secret_key but falls back to InstanceProfileCredentialsProvider:

reshippie:

at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015) ~[?:?]
	at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[?:?]
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[?:?]
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]

Consequently, it uses EC2 auth endpoint, not the one you have configured. The only guess I have is that not all nodes have s3.client.default.access_key and s3.client.default.secret_key configured in keystores. As an experiment, you could try to send access_key and secret_key inside settings as well (if there are no risks to leak them).

reshippie · June 6, 2022, 9:31pm

I have tried deleting and recreating the keystore with no luck. Updating the config file will require a rolling restart, so that will take a little bit of time to test.

reshippie · June 6, 2022, 10:56pm

It seems that putting the credentials inside of opensearch.yml is not allowed.

[2022-06-06T22:53:27,900][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [esmaster-ams5-0000] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.repositories.s3.S3RepositoryPlugin]
...
Caused by: java.lang.IllegalArgumentException: Setting [s3.client.default.access_key] is a secure setting and must be stored inside the OpenSearch keystore, but was found inside opensearch.yml

reta · June 7, 2022, 12:13pm

Correct, only keystore, but you should be able to:

This is just for the sake experiment to find out where the problem might be.

reshippie · June 9, 2022, 9:09pm

We finally made it work.
I don’t remember how I initially created /etc/opensearch/opensearch.keystore but when I run /opt/opensearch/bin/opensearch-keystore create it puts the new keystore file in /opt/opensearch/config

So I removed all of the credentials from the keystore, made sure it was created in the /opt/opensearch subdirectory on every node, then copied that into /etc/opensearch, reloaded settings via the API, then I was actually able to create a repository and take a snapshot.

reta · June 10, 2022, 12:04pm

Thanks for the @reshippie

punitmundra · August 22, 2022, 4:03pm

Aug 16 18:05:47 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:05:47 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:05 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:17 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:17 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): Aug 16 18:06:21 ip-10-1-1-222 bash[12835]: automate-ha-opensearch.default(O): [2022-08-16T18:05:47,425][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [ip-10-1-1-222] this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default
[2022-08-16T18:05:47,425][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [ip-10-1-1-222] this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default
[2022-08-16T18:06:05,631][WARN ][o.o.r.s.S3BlobStore ] [ip-10-1-1-222] Expected request count to be tracked for request [PUT https://a2ha-bloomberg.s3.amazonaws.com /Aug16/opensearch/automate-elasticsearch-data/chef-automate-es6-event-feed-service/tests-B30bi3D4QNm7tNZm58zJhw/master.dat Headers: (Content-Length: 22, Content-Type: application/octet-stream, x-amz-acl: private, x-amz-storage-class: STANDARD, ) ] but found not count.
[2022-08-16T18:06:05,631][WARN ][r.suppressed ] [ip-10-1-1-222] path: /_snapshot/chef-automate-es6-event-feed-service, params: {verify=true, repository=chef-automate-es6-event-feed-service}
org.opensearch.repositories.RepositoryVerificationException: [chef-automate-es6-event-feed-service] path [Aug16/opensearch/automate-elasticsearch-data/chef-automate-es6-event-feed-service] is not accessible on master node
at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1549) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) [opensearch-1.2.4.jar:1.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.IOException: Unable to upload object [Aug16/opensearch/automate-elasticsearch-data/chef-automate-es6-event-feed-service/tests-B30bi3D4QNm7tNZm58zJhw/master.dat] using a single upload
at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:379) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.2.4.jar:1.2.4]
… 6 more
Caused by: com.amazonaws.SdkClientException: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:89) ~[?:?]
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?]
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?]
at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?]
at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?]
at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:55) ~[?:?]
at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:246) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5842) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:5815) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5038) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$18(S3BlobContainer.java:377) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:70) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:69) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:377) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.2.4.jar:1.2.4]
… 6 more
[2022-08-16T18:06:17,486][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [ip-10-1-1-222] this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default
[2022-08-16T18:06:17,486][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [ip-10-1-1-222] this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default
[2022-08-16T18:06:21,402][INFO ][o.o.j.s.JobSweeper ] [ip-10-1-1-222] Running full sweep

Topic		Replies	Views
Snapshot repository timeouts after converting to Opensearch OpenSearch	4	741	June 2, 2022
Plugin repository-gcs not working for snapshots Index Management configure	3	781	March 8, 2023
Failing to read from opensearch-keystore OpenSearch	0	144	November 22, 2023
Use MinIO as snapshot repository failed if use https Open Source Elasticsearch and Kibana	2	3175	November 7, 2020
Snapshot repository (CEPH) connecting to EC2 Metadata instance OpenSearch	0	182	October 31, 2023

What storage service can I use for snapshots?

Related Topics