Is there a way to leverage cross-cluster replication to setup a resilient “store & forward” relationship where remote clusters with limited storage and poor-quality network links forward data to a centralized cluster for long term storage, query, and management.
The idea is that the remote clusters would keep data locally long enough to confirm transfer to a centralized cluster, then remove the local data to free up storage. The primary goal is to account for “less than 100% reliable” WAN links between the remote clusters and the centralized cluster.
Is there a better method for doing this than trying to use cross-cluster replication?
One of the scope requirements that I have to work with is that the logs be locally availble to techs who cycle though the remote facilities during connectivity loss periods, which can last up to several days.
otherwise, buffered logstash would probably work just fine.
How about this: from Logstash/others, send the data to two destinations: a small local OpenSearch and remote OpenSearch. The local one could have low retention.
Actually, Logstash is a bad example, because it will block the queue if one of the destinations isn’t available (i.e. the central OpenSearch), so the local OpenSearch wouldn’t get the data, either.
But it doesn’t have to be Logstash, see the link above. rsyslog comes to mind, because it can have separate queues per action. You’ll see a diagram in this old post. The point is, as long as the action queue is large enough, you’ll be good. And you can decide what rsyslog does when the queue overflows (block, drop only events of a certain severity when the queue is almost full, etc.).