Adding Shard Allocation Awareness by Zone in the Helm chart

Build
1

I am testing the OD Helm chart that can be found here: opendistro-build/helm at main · opendistro-for-elasticsearch/opendistro-build · GitHub

The default supported functionalities like rolling out Master and Data nodes work fine within a single zone, and configuring affinity by creating each node type on dedicated k8s nodes (VMs) also work.

Now I want to go to the next step of configuring the ES nodes into multiple zones. I have set-up a node pool in 2 zones. The way to achieve this is by adding node.attr.zone to the elasticsearch.yaml on each ES node. The value of node.attr.zone should be the name of the zone of the k8s node where the pod is being created. Then we need to add to each master node or apply as cluster setting the cluster.routing.allocation.awareness.attributes: zone. And then we are done! How to achieve this with this helm chart?

Another repo discussed this issue and there is a workaround solution offered by using initContainers, see Simplify zone awareness configuration · Issue #3933 · elastic/cloud-on-k8s · GitHub

The commands that need to be run in the initContainer are these:

ZONE=$(kubectl get node $NODE_NAME -o json | jq -r '.metadata.labels[\"failure-    domain.beta.kubernetes.io/zone\"]') \
CONFIG_FILE=/usr/share/elasticsearch/config/elasticsearch.yml \
CONFIG_FILE_BACKUP=/usr/share/elasticsearch/config/elasticsearch.yml.backup; \
mv $CONFIG_FILE $CONFIG_FILE_BACKUP && \
cat $CONFIG_FILE_BACKUP <(echo node.attr.zone: $ZONE) > $CONFIG_FILE

I tried this solution, but I am getting RBAC issues. And when starting this initContainer, the elasticsearch.yaml file cannot be found either. Has anyone else tried? I would really like to use Helm for this, instead of explictly creating multiple deployments for each zone.

2

What exact issue are seeing when you say RBAC issue? How are you attaching the elasticsearch.yml? configmap? baked into the image or setting up on a persistent volume?

3

I fixed these issues and finally got the shard allocation awareness working with this Helm chart.

I had to add an extra ClusterRoleBinding for the service.
For the node attributes I have used environment variables instead to bypass the issue of not being able to write to the elasticsearch.yml file.

I will polish things up and create a PR on GH. I am looking forward to get some feedback about the solution.

1 Like