Searchable snapshot stop working after upgrade from v2.7 to v2.9

jockjiang · September 21, 2023, 3:26am

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

v2.9

Describe the issue:
We upgrade opensearch from 2.7 to 2.9 recently, we can restore the searchable snapshot , it works well before upgrade. no configuration changes.

Configuration:

Relevant Logs or Screenshots:
[2023-09-21T02:38:24,321][WARN ][o.o.i.c.IndicesClusterStateService] [opensearch-search-0] [security-auditlog-2023.09.20_restored][0] marking and sending shard failed due to [failed to create shard]
java.lang.IllegalStateException: java.util.concurrent.ExecutionException: BlobStoreException[Failed to check if blob [snap-TxKLAgBeRCCP2WWFlRIgHQ.dat] exists]; nested: SdkClientException[Unable to execute HTTP request: Timeout waiting for connection from pool]; nested: ConnectionPoolTimeoutException[Timeout waiting for connection from pool];
at org.opensearch.index.store.remote.directory.RemoteSnapshotDirectoryFactory.newDirectory(RemoteSnapshotDirectoryFactory.java:68) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.index.IndexService.createShard(IndexService.java:479) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.indices.IndicesService.createShard(IndicesService.java:951) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.indices.IndicesService.createShard(IndicesService.java:210) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:675) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:652) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:297) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:606) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:593) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:561) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.9.0.jar:2.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.util.concurrent.ExecutionException: BlobStoreException[Failed to check if blob [snap-TxKLAgBeRCCP2WWFlRIgHQ.dat] exists]; nested: SdkClientException[Unable to execute HTTP request: Timeout waiting for connection from pool]; nested: ConnectionPoolTimeoutException[Timeout waiting for connection from pool];
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at org.opensearch.index.store.remote.directory.RemoteSnapshotDirectoryFactory.newDirectory(RemoteSnapshotDirectoryFactory.java:66) ~[opensearch-2.9.0.jar:2.9.0]
… 17 more
Caused by: org.opensearch.common.blobstore.BlobStoreException: Failed to check if blob [snap-TxKLAgBeRCCP2WWFlRIgHQ.dat] exists
at org.opensearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:123) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3333) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.index.store.remote.directory.RemoteSnapshotDirectoryFactory.lambda$createRemoteSnapshotDirectoryFromSnapshot$0(RemoteSnapshotDirectoryFactory.java:93) ~[opensearch-2.9.0.jar:2.9.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.9.0.jar:2.9.0]
… 3 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[?:?]
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:83) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[?:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[?:?]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[?:?]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[?:?]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[?:?]
at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:5436) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$blobExists$0(S3BlobContainer.java:117) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:55) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:115) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3333) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.index.store.remote.directory.RemoteSnapshotDirectoryFactory.lambda$createRemoteSnapshotDirectoryFromSnapshot$0(RemoteSnapshotDirectoryFactory.java:93) ~[opensearch-2.9.0.jar:2.9.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.9.0.jar:2.9.0]
… 3 more
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: Timeout waiting for connection from pool
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: Timeout waiting for connection from pool
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: Timeout waiting for connection from pool
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316) ~[?:?]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282) ~[?:?]
at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionRequestFactory$DelegatingConnectionRequest.get(ClientConnectionRequestFactory.java:92) ~[?:?]
at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionRequestFactory$InstrumentedConnectionRequest.get(ClientConnectionRequestFactory.java:69) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) ~[?:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[?:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[?:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[?:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[?:?]
at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) ~[?:?]
at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254) ~[?:?]
at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104) ~[?:?]
at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231) ~[?:?]
at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228) ~[?:?]
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:63) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:77) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:56) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:39) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[?:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[?:?]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[?:?]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[?:?]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[?:?]
at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:5436) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$blobExists$0(S3BlobContainer.java:117) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:55) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:115) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3333) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.index.store.remote.directory.RemoteSnapshotDirectoryFactory.lambda$createRemoteSnapshotDirectoryFromSnapshot$0(RemoteSnapshotDirectoryFactory.java:93) ~[opensearch-2.9.0.jar:2.9.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.9.0.jar:2.9.0]
… 3 more

jockjiang · September 21, 2023, 4:35am

this is the logs from master node.

[2023-09-21T04:33:40,838][WARN ][r.suppressed ] [opensearch-master-0] path: /_snapshot/s3_snapshot_repository/_verify, params: {repository=s3_snapshot_repository}
org.opensearch.repositories.RepositoryVerificationException: [s3_snapshot_repository] [[AgjCIdwRQ-ehKyYom1W9iw, ‘RemoteTransportException[[opensearch-search-0][10.87.14.54:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[s3_snapshot_repository] store location [wap-opensearch-dev-uswest2] is not accessible on the node [{opensearch-search-0}{AgjCIdwRQ-ehKyYom1W9iw}{akjouoIwSduIzSFQi1Ahcg}{10.87.14.54}{10.87.14.54:9300}{rs}{shard_indexing_pressure_enabled=true}]]; nested: IOException[Unable to upload object [tests-A5XvWWMfRRGjrUZ1Sr-blg/data-AgjCIdwRQ-ehKyYom1W9iw.dat] using a single upload]; nested: NotSerializableExceptionWrapper[sdk_client_exception: Unable to execute HTTP request: Timeout waiting for connection from pool]; nested: IOException[Timeout waiting for connection from pool];’]]
at org.opensearch.repositories.VerifyNodeRepositoryAction.finishVerification(VerifyNodeRepositoryAction.java:151) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.repositories.VerifyNodeRepositoryAction$1.handleException(VerifyNodeRepositoryAction.java:135) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:379) [opensearch-security-2.9.0.0.jar:2.9.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1504) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:420) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.InboundHandler.handleException(InboundHandler.java:418) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:410) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:158) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) [transport-netty4-client-2.9.0.jar:2.9.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]

jockjiang · September 21, 2023, 4:36am

errors from search node.

[2023-09-21T04:35:05,209][WARN ][o.o.r.VerifyNodeRepositoryAction] [opensearch-search-0] [s3_snapshot_repository] failed to verify repository
org.opensearch.repositories.RepositoryVerificationException: [s3_snapshot_repository] store location [wap-opensearch-dev-uswest2] is not accessible on the node [{opensearch-search-0}{AgjCIdwRQ-ehKyYom1W9iw}{akjouoIwSduIzSFQi1Ahcg}{10.87.14.54}{10.87.14.54:9300}{rs}{shard_indexing_pressure_enabled=true}]
at org.opensearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3171) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:159) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:196) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:191) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:215) [opensearch-security-2.9.0.0.jar:2.9.0.0]
at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:315) [opensearch-security-2.9.0.0.jar:2.9.0.0]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:163) [opensearch-security-2.9.0.0.jar:2.9.0.0]
at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:756) [opensearch-security-2.9.0.0.jar:2.9.0.0]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:113) [opensearch-index-management-2.9.0.0.jar:2.9.0.0]
at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) [opensearch-performance-analyzer-2.9.0.0.jar:2.9.0.0]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:453) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.9.0.jar:2.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.IOException: Unable to upload object [tests-V16ItAQ6Q1e2swQ1-VIyJg/data-AgjCIdwRQ-ehKyYom1W9iw.dat] using a single upload
at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:443) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:161) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:159) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3166) ~[opensearch-2.9.0.jar:2.9.0]
… 16 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[?:?]
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:83) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[?:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[?:?]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[?:?]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[?:?]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[?:?]
at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:9324) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$22(S3BlobContainer.java:440) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:70) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:69) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:439) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:161) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:159) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3166) ~[opensearch-2.9.0.jar:2.9.0]
… 16 more
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: Timeout waiting for connection from pool
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: Timeout waiting for connection from pool
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: Timeout waiting for connection from pool
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316) ~[?:?]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282) ~[?:?]
at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionRequestFactory$DelegatingConnectionRequest.get(ClientConnectionRequestFactory.java:92) ~[?:?]
at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionRequestFactory$InstrumentedConnectionRequest.get(ClientConnectionRequestFactory.java:69) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) ~[?:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[?:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[?:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[?:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[?:?]
at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) ~[?:?]
at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254) ~[?:?]
at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104) ~[?:?]
at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231) ~[?:?]
at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228) ~[?:?]
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:63) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:77) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:56) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:39) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[?:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[?:?]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[?:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[?:?]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[?:?]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[?:?]
at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:9324) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$22(S3BlobContainer.java:440) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:70) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:69) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:439) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:161) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:159) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3166) ~[opensearch-2.9.0.jar:2.9.0]
… 16 more

Gsmitt · September 22, 2023, 4:27am

Hey @jockjiang

I looked over these tons of logs and picked some questionable findings

Notes:
[security-auditlog-2023.09.20_restored][0] marking and sending shard failed due to [failed to create shard]

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool

failure: Unable to execute HTTP request: Timeout waiting for connection from pool

[opensearch-search-0] [s3_snapshot_repository] failed to verify repository

[[s3_snapshot_repository] store location [wap-opensearch-dev-uswest2] is not accessible on the node [{opensearch-search-0

Seams that you have a connection issue with wap-opensearch-dev-uswest2 && opensearch-search-0

What I would do first would be…

Check network connection/configurations and permission sometime and upgrade/update might change settings and ensure those two nodes/instance can see each other.

jockjiang · September 23, 2023, 5:15pm

Thanks @Gsmitt for your checking, wap-opensearch-dev-uswest2 is the AWS S3 bucket name, opensearch-search-0 is the search node. our config is simple, it define the S3 access with identity token as below, the opensearch cluster use the same token( k8s service account token). As this time, we still can backup the index to S3, but failed to restore.

s3.client.default.identity_token_file: /usr/share/opensearch/config/repository-s3/aws-web-identity-token-file

jockjiang · September 23, 2023, 5:17pm

@Gsmitt , to be more clearly, snapshot works and we could continue backup the index to S3, snapshot restore without searchable works, snapshot restore with searchable settings doesn’t work.

We use the API /_snapshot/s3_snapshot_repository/indexname/_restore to restore the index, searchable one has “storage_type”: “remote_snapshot”.

jockjiang · September 23, 2023, 5:53pm

Gsmitt · September 25, 2023, 9:06pm

Hey @jockjiang

This kind of wierd, in the logs it shows

Maybe to resolve this, ensure that the repository settings are correct, including the repository URL and access credentials. Also, check the network connectivity between Opensearchand the repository. If the repository is on a remote server, ensure that OpenSearch has the necessary permissions to access it.

jockjiang · September 25, 2023, 11:35pm

@Gsmitt , I believe the settings are correct, and the network should be fine between the EKS pod and S3 (everything is on AWS). I login the AWS console, and check the S3 bucket, the data keep updating, there is more and more testing folders, is it possible a bug of searchable snapshot?

jockjiang · September 25, 2023, 11:36pm

screenshot for bucket

Gsmitt · September 25, 2023, 11:52pm

hey @jockjiang

I was just going off of the logs you posted that could posibly be the cause.

Not sure but it could be. Perhaps post this issue is GitHub?

jockjiang · September 26, 2023, 1:36am

Thank you, @Gsmitt , let me post this issue in Github, [BUG] search node failed to verify repository · Issue #10228 · opensearch-project/OpenSearch · GitHub.

jockjiang · October 4, 2023, 6:27pm

@Gsmitt , there is no such issue with v2.10, the same configure of opensearch.
I don’t know the root cause, but I think we could close this topic as it works in v2.10.

Topic		Replies	Views
Opensearch rollback(downgrade) to lower version OpenSearch upgrade	13	2085	June 24, 2023
Partial snapshots created with INTERNAL_SERVER_ERROR OpenSearch	0	158	October 24, 2022
Failed upgrade from 1.3.6 to 2.4.0 but cannot go back OpenSearch troubleshoot , upgrade	4	745	December 13, 2022
Query_shard_exception and HTTP response codes OpenSearch discuss , troubleshoot	0	104	March 8, 2024
Issue with async bulkrequest OpenSearch	1	118	March 27, 2024

Searchable snapshot stop working after upgrade from v2.7 to v2.9

Related Topics