Cluster in 2 host not working

yuvaraj.p · December 16, 2021, 3:45pm

I am working on creating a cluster using OpenSearch with basic config in different machines within same subnet. But the nodes in different machines were working as independent clusters and nodes. Followed the following steps to set up the cluster

Downloaded the .tar.gz file for Linux and extracted it a directory
Change the following attributes in the config.yml

config/opensearch.yml for node-1:

cluster.name: z-cluster

network.host: 0.0.0.0

http.port: 9200

discovery.seed_hosts: ["x.x.x.127", "x.x.x.182"]

cluster.initial_master_nodes: ["x.x.x.127"]

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

config/opensearch.yml for node-2:

cluster.name: z-cluster

node.name: node-1

network.host: 0.0.0.0

http.port: 9200

discovery.seed_hosts: ["x.x.x.127", "x.x.x.182"]

cluster.initial_master_nodes: ["x.x.x.127"]

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

Exectued $ ./bin/opensearch in node-1 followed by node-2

Logs from node-1

[2021-12-16T15:37:14,075][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint

[2021-12-16T15:37:14,076][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing of external configuration is disabled.
[2021-12-16T15:37:14,077][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing of internal configuration is enabled.
[2021-12-16T15:37:14,078][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing only metadata information for read request is enabled.
[2021-12-16T15:37:14,080][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing will watch {} for read requests.
[2021-12-16T15:37:14,087][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing read operation requests from kibanaserver users is disabled.
[2021-12-16T15:37:14,088][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing only metadata information for write request is enabled.
[2021-12-16T15:37:14,091][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing diffs for write requests is disabled.
[2021-12-16T15:37:14,091][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing write operation requests from kibanaserver users is disabled.
[2021-12-16T15:37:14,092][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing will watch <NONE> for write requests.
[2021-12-16T15:37:14,092][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] .opendistro_security is used as internal security index.
[2021-12-16T15:37:14,092][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Internal index used for posting audit logs is null
[2021-12-16T15:37:14,093][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Hot-reloading of audit configuration is enabled
[2021-12-16T15:37:14,094][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node 'node-1' initialized
[2021-12-16T15:38:13,078][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Performing move cluster state metadata.

Logs from node-2

[2021-12-16T15:37:42,313][WARN ][o.o.s.a.r.AuditMessageRouter] [node-2] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2021-12-16T15:37:42,314][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Auditing of external configuration is disabled.
[2021-12-16T15:37:42,314][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Auditing of internal configuration is enabled.
[2021-12-16T15:37:42,315][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Auditing only metadata information for read request is enabled.
[2021-12-16T15:37:42,317][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Auditing will watch {} for read requests.
[2021-12-16T15:37:42,321][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Auditing read operation requests from kibanaserver users is disabled.
[2021-12-16T15:37:42,323][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Auditing only metadata information for write request is enabled.
[2021-12-16T15:37:42,324][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Auditing diffs for write requests is disabled.
[2021-12-16T15:37:42,325][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Auditing write operation requests from kibanaserver users is disabled.
[2021-12-16T15:37:42,325][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Auditing will watch <NONE> for write requests.
[2021-12-16T15:37:42,326][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] .opendistro_security is used as internal security index.
[2021-12-16T15:37:42,326][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Internal index used for posting audit logs is null
[2021-12-16T15:37:42,327][INFO ][o.o.s.c.ConfigurationRepository] [node-2] Hot-reloading of audit configuration is enabled
[2021-12-16T15:37:42,328][INFO ][o.o.s.c.ConfigurationRepository] [node-2] Node 'node-2' initialized
[2021-12-16T15:38:40,816][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-2] Performing move cluster state metadata.

Curl used : curl -XGET https://localhost:9200 -u ‘admin:admin’ --insecure

Curl for node-1

{
  "name" : "node-1",
  "cluster_name" : "z-cluster",
  "cluster_uuid" : "4wFWhEJWTr2H1jLiI8Gihw",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.2.0",
    "build_type" : "tar",
    "build_hash" : "c459282fd67ddb17dcc545ec9bcdc805880bcbec",
    "build_date" : "2021-11-22T16:57:18.360386Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Curl for node-2

{
  "name" : "node-2",
  "cluster_name" : "z-cluster",
  "cluster_uuid" : "SFXa_PL6S565qyEmj5M8kQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.2.0",
    "build_type" : "tar",
    "build_hash" : "c459282fd67ddb17dcc545ec9bcdc805880bcbec",
    "build_date" : "2021-11-22T16:57:18.360386Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

With individual cluster(cluster_uuid) for both nodes is created. And also I can ssh to between machine.

Note: I even tried with ElasticSearch Basic and Opendistro RPM package with the above changes, I was able to create a single cluster for both nodes

Question: Is there anything wrong am configuring with OpenSearch?

rhtbansal · June 21, 2023, 11:47am

were you able to resolve the issue I am getting same issue ??

Topic		Replies	Views
Opensearch cluster: Node don't see the cluster OpenSearch troubleshoot , configure	10	551	April 17, 2024
Opensearch Cluster Highavailability testing question OpenSearch	2	122	April 3, 2024
Cannot make nodes - simple 2-node cluster - talk to each other OpenSearch troubleshoot , configure	0	680	July 28, 2023
3 nodes opensearch cluster, cannot start with only 2 nodes OpenSearch	20	1049	March 28, 2024
OpenSearch single node not initializing : Stuck at waiting for yellow cluster state Security troubleshoot	4	1875	May 24, 2023

Cluster in 2 host not working

Related topics