Cross Cluster Search connection imbalance

OpenSearch
1

Versions: Opensearch 1,2,3.x

Describe the issue: Connections from CCS nodes to remote nodes become imbalanced over time, leading to performance problems.

Configuration: We have a 20 node CCS cluster connected to multiple remote clusters. Remote clusters generally have 3 coordinating nodes and 20+ data nodes. Remotes are setup with sniff mode targeting the nodes with the gateway attribute (coordinating nodes). Initially the 20 CCS nodes each connect to the 3 coordinating nodes and everything looks great, cpu and network resource are well-balanced between the Coordinating nodes. However, if one of the Coordinating nodes is restarted, none of the CCS nodes reconnect to it without some help. The CCS nodes can be restarted to fix the problem, or the remote itself can be adjusted in some small way (increase node_connections, for instance) to trigger the CCS nodes into reconnecting to all of the coordinating nodes again.

I made an issue here quite a while ago, originally thinking this was just a problem with the proxy mode (and not sniff): [Feature Request] Cross Cluster Search Proxy/Sniff Mode Connection Rebalancing · Issue #16529 · opensearch-project/OpenSearch · GitHub

This is a pretty serious problem for us. Any time we have host maintenance we may move and restart some coordinating pods (we’re using Kubernetes). This action takes the node out of the CCS configuration basically forever, so we have only 2 of 3 coordinating nodes active.

It can get worse than just one missing member. Some of our clusters have more Coordinating nodes to handle more traffic, so can have up to 8 Coordinating nodes, and we increase node_connections to 8 as well. If we have some incident where multiple Coordinating nodes are restarted at the same time, the CCS nodes again do not ever try to reconnect to them on their own.

I was poking around the source code and at first glance this shouldOpenMoreConnections() function should help invoke connecting to more nodes, but it only seems to be called at the moment a node is disconnected? That might be the crux of the problem here…

2

@briendi Thank you for the question, looking at the code I can see that the sequence of evens is as follows:

  1. Coordinating node goes down → disconnect detected → shouldOpenMoreConnections() is true → reconnects to another available node → connection count is restored to maxNumRemoteConnections
  2. Restarted coordinating node comes back → nothing triggers a reconnect check → shouldOpenMoreConnections() is now false (count is already at max) → the restarted node is permanently ignored until something forces a strategy rebuild

While the issue is being worked on, you can try the following workaround, which is least disruptive then restarting the coordinating node:

The setting cluster.remote.<alias>.node_connections is dynamic. Any change to it causes strategyMustBeRebuilt() to return true, which tears down and rebuilds the connection strategy, running a fresh sniff cycle that will discover the restarted node:

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.<alias>.node_connections": <current_value + 1>
  }
}

Then immediately set it back to the original value. Two API calls, no restarts needed. You can script this as a post-restart step in your K8s lifecycle hooks on the coordinating node side.

Hope this helps

3

Thanks Anthony, I agree with your suggestion but let me clarify a few points

Connection count would be restored only if there were standby Coordinating nodes ready, but that isn’t very practical; particularly if you have N nodes restart simultaneously (you’d need N standby nodes)

Again, num_nodes_connected would still be less than max_connections_per_cluster unless you had those standby nodes, so shouldOpenMoreConnections() would be true here (it’s just not invoked at a useful time, which would be some time after the node is back online)

I think you mean less disruptive than restarting the CCS node(s), since restarting the Coordinating node wouldn’t help or hurt at all since it isn’t participating in search even after it comes back online.

I find this does indeed work, but I also notice that tearing down the connection (twice here) is definitely noticeable and some search requests will fail during that; so this is not quite an ideal band aid, but probably the best bet for now.

Thanks again!