Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch - 2.19.2
Describe the issue: I want to inquire whether the behaviour I have encountered is supported and intended
later in the text, an OpenSearch cluster with the role of remote cluster client (OS Coordinator) will be called OS Proxy as it is the name it was given in documentation I have been working with.
We would like to clarify whether the following Cross-Cluster Search behavior in OpenSearch 2.19.2 is intentional, supported, and safe to rely on in production:
<top-level-remote-cluster-alias>:<regional-remote-cluster-alias>:<index>
In our tests, a top-level OpenSearch proxy cluster can send a CCS request to a regional OpenSearch proxy cluster, and the regional proxy cluster then interprets the remaining : expression as its own CCS request to downstream data clusters
We want to understand whether this is a supported feature/contract or an accidental consequence of how index expressions are parsed today, and whether we can rely on this behavior remaining stable in future OpenSearch versions?
Environment
Client
-> Top Level OpenSearch Proxy cluster
-> Regional OpenSearch Proxy cluster
-> OpenSearch Data clusters
Both the top-level proxy and regional proxies are OpenSearch clusters with the remote_cluster_client role
The top-level proxy is configured with regional proxy clusters as remote clusters
Each regional proxy is configured with local data clusters as remote clusters
Observed behavior
The request succeeds
For example:
- querying only region 1 through the regional proxy returns the expected result;
- querying only region 2 through the regional proxy returns the expected result;
- querying region 1 through the top-level proxy returns the same result as querying region 1 directly;
- querying both regions through the top-level proxy returns merged aggregation results across both regions
The following aggregation types behaved as expected in our tests:
- terms
- avg
- max
- sum
For example, bucket doc_count values were added across regions, sum values were added, max values were globally reduced, and avg appeared to be reduced correctly based on document counts rather than as a simple average of regional averages
This scheme works correctly in our tests
Related component
Search:Remote Search
To Reproduce
- Create 5 OS clusters (or less, it’s enough 1 data, 1 regional proxy and 1 top level)
- data-cluster-1
- data-cluster-2
- regional-proxy-1
- regional-proxy-2
- top-level-proxy
The important part is that:
top-level-proxy
-> knows regional-proxy-* as remote clusters
regional-proxy-*
-> know data-cluster-* as remote clusters
- Put simple test index to data clusters
PUT test-nested-ccs
{
"mappings": {
"properties": {
"service": {
"type": "keyword"
},
"duration_ms": {
"type": "double"
},
"bytes_sent": {
"type": "long"
}
}
}
}
and insert a few test documents into each data cluster. Use different document counts/values on each data cluster so it is easy to verify that aggregation results are merged
POST test-nested-ccs/_bulk
{ "index": {} }
{ "service": "service-1", "duration_ms": 50.0, "bytes_sent": 1000 }
{ "index": {} }
{ "service": "service-1", "duration_ms": 60.0, "bytes_sent": 2000 }
{ "index": {} }
{ "service": "service-2", "duration_ms": 250.0, "bytes_sent": 3000 }
{ "index": {} }
{ "service": "service-3", "duration_ms": 130.0, "bytes_sent": 4000 }
- Configure regional and top-level OS Proxy clusters
- Run nested CCS query through the top-level proxy
POST regional-proxy-1:data-cluster-1:test-nested-ccs/_search
{
"size": 25,
"query": {
"match_all": {}
}
}
Expected behavior
Is this an intended and supported CCS behavior? Maybe it shouldn’t work like this?
Additional Details
We also tested wildcard expressions such as:
POST *:*:test-index/_search
where:
- the first * expands to regional proxy remote cluster aliases on the top-level proxy;
- the second * expands to data cluster remote aliases on each regional proxy
From Source Code
Based on OpenSearch source code analysis, this functionality appears to work because remote index expressions are split on the first : only
For example, RemoteClusterAware.groupClusterIndices(...) appears to take the part before the first : as the remote cluster alias and keep the remainder as the index expression passed to the remote cluster
Then, when the regional proxy receives the search request, its own TransportSearchAction and RemoteClusterService perform the same grouping again using the regional proxy’s own remote cluster configuration
This seems to make chained CCS possible, but we could not find documentation stating whether this is an intended and supported behavior