Why can't OpenSearch accept K8S service for initial_cluster_manager_nodes configuration?

Versions: OpenSearch 2.8.0 and under.

Hi. I’m trying to setup OpenSearch cluster on Kubernetes.
I use k8s headless service pointing the OpenSearch node pods for cluster.initial_cluster_manager_nodes but cluster bootstrapping failed with these logs:

[2023-08-22T17:44:13,959][INFO ][o.o.c.c.ClusterBootstrapService] [os-test-cluster-manager-1] skipping cluster bootstrapping as local node does not match bootstrap requirements: [os-cluster-manager-discovery]

So, I looked over ClusterBootstrapService codes, and I found that there’s no DNS resolving for initial_cluster_manager_nodes.
I got confused bc some configuration options, such as discovery.seed_hosts, has a resolver to get proper IP addresses from DNS, so k8s service names are allowed.

Why I can’t use service names for initial_cluster_manager_nodes configuration? Is there a reason not to use resolver?

@jeongmin Could you share your values.yml file?
Have you tried os-cluster-manager-discovery.default.svc instead?

@pablo Sorry for the late reply.
I’ve tested your suggestion but cluster manager discovery failed again.
[2023-09-11T14:59:13,845][INFO ][o.o.c.c.ClusterBootstrapService] [os-test-project-cluster-manager-0] skipping cluster bootstrapping as local node does not match bootstrap requirements: [os-test-project-cluster-manager-discovery.opensearchoperatore2e-createopensearchcluster.svc]

I’m distributing OS clusters via self-made operator, so there’s no values.yaml or helm chart stuff.

I made statefulset with templates and all the OpenSearch configuration options are passed by container env variables.

@jeongmin I had a look at the OpenSearch cluster deployed with OpenSearch helm charts.
The deployed OpenSearch nodes have cluster.initial_cluster_manager_nodes set to pod names in the env variables.

Charts use a script in the helper to extract the pod name for this option.
Only the discovery.seed_hosts option is set to a service called <chart_name>-cluster-master-headless.

I think you need to do the same in your custom deployment.