Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch version 2.16.0 and 2.18.0
Server: Red Hat Enterprise Linux release 9.5 (Plow)
Describe the issue:
Unable to get CCR working.
Configuration:
multi-node cluster
Leader-Cluster:
coordinator: node.roles: [ ]
data: node.roles: [data,ingest]
master: node.roles: [master]
Follower Cluster:
coordinator: node.roles: [remote_cluster_client]
data: node.roles: [data,ingest]
master: node.roles: [master,remote_cluster_client]
Relevant Logs or Screenshots:
[2024-12-28T03:54:00,462][INFO ][o.o.r.a.i.TransportReplicateIndexAction] [os-fra-coordinator2] Setting-up replication for my-connection-alias:leader-01 → follower-01
[2024-12-28T03:54:00,463][DEBUG][o.o.r.a.s.TransportSetupChecksAction] [os-fra-coordinator2] Validation role in threadcontect is null for aero-search-stage1-fra-follower
[2024-12-28T03:54:00,463][INFO ][o.o.r.a.s.TransportValidatePermissionsAction] [os-fra-coordinator2] Replication setup - Permissions validation successful for Index - follower-01 and role all_access
[2024-12-28T03:54:00,463][INFO ][o.o.r.a.s.TransportSetupChecksAction] [os-fra-coordinator2] Permissions validation successful for User [local:aero-search-stage1-fra-follower, resource:follower-01]
[2024-12-28T03:54:00,463][DEBUG][o.o.r.a.s.TransportSetupChecksAction] [os-fra-coordinator2] Validation role in threadcontect is null for my-connection-alias
[2024-12-28T03:54:00,465][INFO ][o.o.r.a.s.TransportSetupChecksAction] [os-fra-coordinator2] Permissions validation successful for User [connection:my-connection-alias, resource:leader-01]
[2024-12-28T03:54:00,465][DEBUG][o.o.r.a.s.TransportSetupChecksAction] [os-fra-coordinator2] Validation role in threadcontect is null for aero-search-stage1-fra-follower
[2024-12-28T03:54:00,465][INFO ][o.o.r.a.s.TransportValidatePermissionsAction] [os-fra-coordinator2] Replication setup - Permissions validation successful for Index - follower-01 and role all_access
[2024-12-28T03:54:00,465][INFO ][o.o.r.a.s.TransportSetupChecksAction] [os-fra-coordinator2] Permissions validation successful for User [connection:my-connection-alias, resource:leader-01]
[2024-12-28T03:54:00,465][DEBUG][o.o.r.a.s.TransportSetupChecksAction] [os-fra-coordinator2] Validation role in threadcontect is null for my-connection-alias
[2024-12-28T03:54:00,466][INFO ][o.o.r.a.s.TransportSetupChecksAction] [os-fra-coordinator2] Permissions validation successful for role [connection:my-connection-alias, resource:leader-01]
[2024-12-28T03:54:00,486][WARN ][r.suppressed ] [os-fra-coordinator2] path: /_plugins/_replication/follower-01/_start, params: {pretty=true, index=follower-01}
java.lang.IllegalStateException: Timed out when waiting for persistent task after 30s
at org.opensearch.persistent.PersistentTasksService$WaitForPersistentTaskListener.onTimeout(PersistentTasksService.java:255) ~[opensearch-2.16.0.jar:2.16.0]
at org.opensearch.persistent.PersistentTasksService$1.onTimeout(PersistentTasksService.java:203) ~[opensearch-2.16.0.jar:2.16.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.16.0.jar:2.16.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.16.0.jar:2.16.0]
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:737) ~[opensearch-2.16.0.jar:2.16.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:882) ~[opensearch-2.16.0.jar:2.16.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-12-28T03:54:00,491][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [os-fra-coordinator2] Cancelling the migration process.
[2024-12-28T03:54:00,507][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [os-fra-coordinator2] Cancelling the migration process.