Setting up Hot-Warm architecture for an opensearch cluster on k8s with the helm charts

OpenSearch
, ,
1

Currently trying to set up Hot-Warm architecture for an opensearch cluster on k8s with the helm charts.

I’m basically installing the charts for my data nodes twice (hot and warm). The charts are pretty similar with the only changes being the “node.attr.temp: <hot/warm>”, storageClass and nodeGroup.

Everything was working fine before I tried to differentiate the data nodes.

Now, I’m getting the following error msg on dashboards logs " {“type”:“log”,“@timestamp”:“”,“tags”:[“error”,“opensearch”,“data”],“pid”:1,“message”: [search_phase_execution_exception]: all shards failed"} 15756"

I tried running the following command on my masters and the nodes seem to be forming as a cluster.

image
image1708×342 222 KB

However, I did try running the same command on my data nodes and I got a “OpenSearch Security not initialized” msg, which did not happen before. Otherwise no error messages on my masters or data nodes logs.

As I mentioned above, I did change the nodeGroup parameter from “data” to “data-hot” and “data-warm” and I’m not certain if that has something to do with it.

When I list all the indices in the cluster and their current status:

image
image1389×163 18.5 KB

Any tips on how to solve or troubleshoot this would be very helpful. Ty

2

Also, is “node.attr.temp: <hot/warm>” the right way to specify the attribute for the node? Cause I’ve tested with “node.roles: data_hot” and “node.roles: data_warm” and the charts were installed too. I’m getting the same error msg tho and the only difference is that now when I run “curl -XGET https://localhost:9200/_cat/nodeattrs?v -u ‘admin:admin’ --insecure” I can’t see hot or warm on the values

image
image1087×268 53.5 KB

when I tried with “node.attr.temp: <hot/warm>” I got

image
image1693×601 130 KB

What is the right way to define the data node type?

Side Note: I’ve tested installing the data node charts without setting the hot/warm attribute so I’m confident this has nothing to do with the error message I got. I’m sure that has something to do with changing the nodeGroup parameter, but I can’t see any other way of differentiating my data nodes

3

UPDATE:

Fixed. It had nothing to do with the nodeGroup name or anything. I uninstalled every chart (masters, data-hot and data-warm) and deleted every pvc. Installed the charts again and the error message went away. I’m still quite confused and unsure of what may have caused it. All sorted now