Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.19
Describe the issue:
On our cluster, both nodes were loaded with index/merge activity. We observed that our EBS IOPS/throughput saturated but CPU was only around 40%. When this happened, the cluster entered a state where the follower node (EM1) took too long to respond to follower checks from the cluster manager (EM0). As a result, it repeatedly disconnected then connected to the cluster.
After indexing activity had dropped, we observed that the follower node continued to be kicked and readded from the cluster. This prevented the cluster manager from publishing state in a timely manner [1]. Specifically, we observed that cluster state took a long time to apply with logs showing state application taking O(minutes) up to 34 minutes! [2]. The index on the follower node tried to recover from EXISTING_STORE (local disk) but repeatedly failed once it was kicked for having a lagging state. This caused OS to report index size oscillations [3]
2 questions:
- If CPU is low and disk throughput/IOPS are saturated, what would cause follower checks to slow down? From a readthrough of the code, this codepath shouldn’t hit disk
- If CPU is low and disk throughput is low, what would cause state application to take 34 minutes?
Configuration:
2 x16-core machines, 32 shards. S3 remote backed store.
Relevant Logs or Screenshots
(logs say elasticsearch but we’re running opensearch)
after [10s] publication of cluster state version [2732] is still waiting for
{elasticsearch-master-1}{aZO8SRRPR6CqIDmcUQMJrg}{LK3XYRwaTIS0JJ7Xd_esVw}
{10.1.37.87}{10.1.37.87:9300}{dim}{shard_indexing_pressure_enabled=true}
[SENT_PUBLISH_REQUEST]
cluster state applier task [ApplyCommitRequest{term=4, version=2733, sourceNode=
{elasticsearch-master-0}{_8zea0MvRNiTf0DdneV8dA}{i3GlEZEGR7OY0HzkdYXVOA}
{10.1.44.229}{10.1.44.229:9300}{dim}}] took [34m] which is above the warn
threshold of [30s]: [running task [ApplyCommitRequest{term=4, version=2733,
sourceNode={elasticsearch-master-0}{_8zea0MvRNiTf0DdneV8dA}
{i3GlEZEGR7OY0HzkdYXVOA}{10.1.44.229}{10.1.44.229:9300}{dim}}]] took [0ms],
[connecting to new nodes] took [0ms], [applying settings] took [0ms], [running
applier [org.opensearch.repositories.RepositoriesService@14cf3bfe]] took [0ms],
[running applier
[org.opensearch.indices.cluster.IndicesClusterStateService@27c67e8e]] took
[2043170ms], [running applier [org.opensearch.script.ScriptService@59ba5da4]] took
[0ms], [running applier [org.opensearch.snapshots.RestoreService@14e65ff7]] took
[0ms], [running applier [org.opensearch.ingest.IngestService@40c513d0]] took
[0ms], [running applier
[org.opensearch.search.pipeline.SearchPipelineService@5d64cef4]] took [0ms],
[running applier [org.opensearch.action.ingest.IngestActionForwarder@13e8fc60]]
took [0ms], [running applier
[org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupReposito
ryAction$$Lambda/0x000000b8019572b8@2369f59]] took [0ms], [running applier
[org.opensearch.tasks.TaskManager@79dee285]] took [0ms], [running applier
[org.opensearch.snapshots.SnapshotsService@5bfd979d]] took [0ms], [notifying
listener
[org.opensearch.cluster.applicationtemplates.SystemTemplatesService@39d8f184]]
took [0ms], [notifying listener
[org.opensearch.cluster.InternalClusterInfoService@371cf6f0]] took [0ms],
[notifying listener
[org.opensearch.snapshots.InternalSnapshotsInfoService@6acef1]] took [0ms],
[notifying listener [org.opensearch.wlm.QueryGroupService@1c8f3c09]] took [0ms],
[notifying listener [org.opensearch.node.ResourceUsageCollectorService@325e7b0f]] took [0ms], [notifying listener
[org.opensearch.cluster.metadata.SystemIndexMetadataUpgradeService@641536f3]] took [0ms], [notifying listener
[org.opensearch.cluster.metadata.TemplateUpgradeService@19f310c]] took [0ms],
[notifying listener [org.opensearch.node.ResponseCollectorService@4ea38030]] took
[0ms], [notifying listener
[org.opensearch.snapshots.SnapshotShardsService@f640ce8]] took [0ms], [notifying
listener [org.opensearch.persistent.PersistentTasksClusterService@428b0159]] took
[0ms], [notifying listener
[org.opensearch.cluster.routing.DelayedAllocationService@3a644f55]] took [0ms],
[notifying listener [org.opensearch.indices.store.IndicesStore@6894fa29]] took
[0ms], [notifying listener
[org.opensearch.persistent.PersistentTasksNodeService@5d23f8f8]] took [0ms],
[notifying listener [org.opensearch.gateway.GatewayService@45a51aee]] took [0ms],
[notifying listener
[org.opensearch.indices.recovery.PeerRecoverySourceService@aafe5c1]] took [0ms], [notifying listener
[org.opensearch.indices.replication.SegmentReplicationTargetService@851485b]] took
[0ms], [notifying listener
[org.opensearch.indices.replication.SegmentReplicationSourceService@4a779591]]
took [0ms]
