Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch v2.11.1 running in Podman (aka docker) on RHEL8
Describe the issue:
We have multiple nodes with 128GB of RAM so I have three containers running on them with different data partitions etc etc (Its all running fine) but I also am using host-based networking. This presents a problem when multiple services try to bind to the same IP/Port combinations.
The obvious fix I have in place is that:
- container A binds to host-ip and port 9600 (and RPC 9650)
- container B binds to host-ip and port 9601 (and RPC 9651)
- container C binds to host-ip and port 9602 (and RPC 9652)
This is configured in both files: /usr/share/opensearch/config/opensearch-performance-analyzer and /usr/share/opensearch/performance-analyzer-rca/config/performance-analyzer.properties
My question is: given these non-standard ports the the data nodes etc are running on, do I need to change “something” elsewhere for the PA plugin and RCA plugin to be more ‘aware’?
Querying one type of nodes with:
curl "localhost:9600/_plugins/_performanceanalyzer/metrics?metrics=Latency,CPU_Utilization&agg=avg,max&dim=ShardID&nodes=all" | jq keys
Gives me 5 keys (for the 5 hot indexers)
Querying the other type of cold nodes with:
curl "localhost:9601/_plugins/_performanceanalyzer/metrics?metrics=Latency,CPU_Utilization&agg=avg,max&dim=ShardID&nodes=all" | jq keys
Gives me 9 keys (for the 3x3 cold nodes/containers)
So neither node ‘type’ have the full cluster awareness. This is more of a concern regarding the RCA plugin.
Whats the recommended fix here? Thanks.