OpenSearch: cluster nodes bound to localhost?

OpenSearch
1

Hello all!

I used the official Ansible role to deploy a 3x nodes cluster after I fixed this issue Official Ansible role fails to start OpenSearch - #5 by m_mlk

The current issue now is:

Both

curl -XGET https://<private-ip>:9200/_cat/nodes?v -u 'admin:admin' --insecure

or

curl -XGET https://$(hostname):9200/_cat/nodes?v -u 'admin:admin' --insecure

will show:

curl: (7) Failed connect to os-node-1:9200; Connection refused

Only

curl -XGET https://localhost:9200/_cat/nodes?v -u 'admin:admin' --insecure

works.

Port 9200 is listening but seems to be bound to localhost:

$ lsof -Pni:9200
COMMAND   PID       USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    11894 opensearch  496u  IPv4  61147      0t0  TCP 127.0.0.1:9200 (LISTEN)

The problem with this is that only 1 node is shown:

$ curl -XGET https://localhost:9200/_cat/nodes?v -u 'admin:admin' --insecure
ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role cluster_manager name
127.0.0.1           16          24   0    0.00    0.01     0.05 dimr      *               os-node-1

and each single node in the multi-node cluster things its isolated and the master:

$curl https://localhost:9200/_cluster/health?pretty -u admin:admin -k
{
  "cluster_name" : "opensearch", <-- this should be "opensearch-cluster" (see below)
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,         <-- this should be 3
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 1,
  "active_shards" : 1,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

This is the config for the nodes - only the “node.name” and “network.host” values change, depending on the node:

$ cat /usr/share/opensearch/config/opensearch.yml
## BEGIN opensearch main configuration ##
cluster.name: "opensearch-cluster"

node.name: "os-node-1"

network.host: "1.2.3.4"

http.port: 9200

bootstrap.memory_lock: true

discovery.seed_hosts: ["os-node-1","os-node-2","os-node-3"]


node.roles: [data,master]
## END opensearch main configuration ##

I even tried

network.host: 0.0.0.0

but it doesn’t change anything…

Is all this because there is a config file under /etc/opensearch and another one under /usr/share/opensearch/config? I would assume that they add up together at runtime… or?

Am I overseeing the obvious?

TIA!

2

Once again, I reply to myself.

Having 2 config files is not a good idea :slight_smile:
The (dirty) trick was to:

1. cat /usr/share/opensearch/config/opensearch.yml >> /etc/opensearch/opensearch.yml
2. mv /usr/share/opensearch/config/opensearch.yml /usr/share/opensearch/config/opensearch.yml_ORG
3. edit /etc/opensearch/opensearch.yml and change "network.host" to "0.0.0.0"
4. systemctl restart opensearch

After that, everything worked as expected, the cluster was created fine and all 3 nodes are visible from all nodes.

HTH

Best