Opensearch 2.16.0 Helm Install in GKE 1.29, masters not finding each other

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.16.0

Describe the issue:

I am using the helm chart to try to setup open search 2.16.0. However when it first comes up the masters won’t find each other, and it leaves the cluster in a red state. I feel like I am missing a step.

Here is my stateful set:



**Configuration**:


        values: {
            // nodeGroup: "master",
            // clusterName: "opensearch",            
            roles: [
                "cluster_manager",
                "ingest",
                "data",
                "remote_cluster_client"
            ],
            replicas: 3,
            imageTag: oscVersion,
            image: {
                tag: oscVersion,
            },
            opensearchJavaOpts: "-Xmx1024M -Xms1024M",
            // service: {
            //     type: "NodePort",
            // },
            persistence: {
                enabled: false,
            },
            resources: {
                requests: {
                    memory: "2Gi",
                    cpu: "1000m"
                },
                limits: {
                    memory: "2Gi",
                    cpu: "1000m",
                },
            },
            sysctlInit: {
                enabled: true
            },
            extraEnvs: [
                {
                    name: "OPENSEARCH_INITIAL_ADMIN_PASSWORD",
                    value: "PASSWORD"
                }
            ],


**Relevant Logs or Screenshots**:
2

Hi @rshade,

Not sure if this is helpful, but I’ve noticed that in the docs they are setting the “master” role rather than “cluster_manager” (helm-charts/charts/opensearch/values.yaml at main · opensearch-project/helm-charts · GitHub).

image

best,
mj

3

Do you see any errors/warnings in your logs?

4

master is out of date and throws a panic, you have to use cluster_manager on 2.16.0. Not sure when it became a panic though.

logs from kubectl logs

[2024-09-12T22:05:28,966][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-cluster-master-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_cluster_manager_nodes] is empty on this node: have discovered [{opensearch-cluster-master-0}{oXsGWAt0RHiJwkmnSPIC3w}{0oPp14iITbGAi0lZftPXmw}{240.16.17.7}{240.16.17.7:9300}{dimr}{shard_indexing_pressure_enabled=true}, {opensearch-cluster-master-2}{xhMnFNqPQhudvphpzuTUVA}{H_LcQAPLR2yTmZ2QSwW7tQ}{240.16.7.46}{240.16.7.46:9300}{dimr}{shard_indexing_pressure_enabled=true}, {opensearch-cluster-master-1}{C3InRIBxSOqPuNsdAmiPDA}{pNX3-tX-QYmNw8yr1wqZAw}{240.16.4.23}{240.16.4.23:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [240.16.4.23:9300, 240.16.7.46:9300] from hosts providers and [{opensearch-cluster-master-0}{oXsGWAt0RHiJwkmnSPIC3w}{0oPp14iITbGAi0lZftPXmw}{240.16.17.7}{240.16.17.7:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-09-12T22:05:29,078][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Wait for cluster to be available ...
[2024-09-12T22:05:30,078][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Wait for cluster to be available ...
[2024-09-12T22:05:31,078][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Wait for cluster to be available ...
[2024-09-12T22:05:32,079][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Wait for cluster to be available ..

When I try to run security admin manually:

[opensearch@opensearch-cluster-master-0 ~]$ sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
bash: sudo: command not found
[opensearch@opensearch-cluster-master-0 ~]$ sudo
bash: sudo: command not found
[opensearch@opensearch-cluster-master-0 ~]$ sh  "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=kirk,OU=client,O=client,L=test,C=de"
OpenSearch Version: 2.16.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cn) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
5

This is the problem: Updated charts to follow the OpenSearch Project's inclusive language standards by Phenix66 · Pull Request #560 · opensearch-project/helm-charts · GitHub I am betting. Thanks for the lead.

1 Like