Why can't OpenSearch accept K8S service for initial_cluster_manager_nodes configuration?

jeongmin · August 29, 2023, 8:28am

Versions: OpenSearch 2.8.0 and under.

Hi. I’m trying to setup OpenSearch cluster on Kubernetes.
I use k8s headless service pointing the OpenSearch node pods for cluster.initial_cluster_manager_nodes but cluster bootstrapping failed with these logs:

[2023-08-22T17:44:13,959][INFO ][o.o.c.c.ClusterBootstrapService] [os-test-cluster-manager-1] skipping cluster bootstrapping as local node does not match bootstrap requirements: [os-cluster-manager-discovery]

So, I looked over ClusterBootstrapService codes, and I found that there’s no DNS resolving for initial_cluster_manager_nodes.
I got confused bc some configuration options, such as discovery.seed_hosts, has a resolver to get proper IP addresses from DNS, so k8s service names are allowed.

Why I can’t use service names for initial_cluster_manager_nodes configuration? Is there a reason not to use resolver?

pablo · August 29, 2023, 12:49pm

@jeongmin Could you share your values.yml file?
Have you tried os-cluster-manager-discovery.default.svc instead?

jeongmin · September 11, 2023, 6:13am

@pablo Sorry for the late reply.
I’ve tested your suggestion but cluster manager discovery failed again.
[2023-09-11T14:59:13,845][INFO ][o.o.c.c.ClusterBootstrapService] [os-test-project-cluster-manager-0] skipping cluster bootstrapping as local node does not match bootstrap requirements: [os-test-project-cluster-manager-discovery.opensearchoperatore2e-createopensearchcluster.svc]

I’m distributing OS clusters via self-made operator, so there’s no values.yaml or helm chart stuff.

I made statefulset with templates and all the OpenSearch configuration options are passed by container env variables.