OpenSearch Cluster Nodes Don't Connect to Cluster Manager

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch V 2.10.0

Describe the issue:
I am trying to connect 2 data/ingest nodes to the cluster manager node. All nodes report an error “OpenSearch Security not Initialized”. I have run the securityadmin.sh tool and it doesn’t give any valuable results. If I set each node to also being a cluster manager node, they no longer have the OpenSearch Security not Initialized error but they do not connect to the main cluster manager. I’m not sure how to get the nodes to connect to the main cluster manager and also not sure how to get the nodes working without having them all set to be cluster managers. I’ve added the configs for the cluster manager and a data/ingest node below (some info has been changed for privacy reasons).

Configuration:
config for the cluster manager node

======================== OpenSearch Configuration =========================

---------------------------------- Cluster -----------------------------------

cluster.name: demo-opens-cluster

------------------------------------ Node ------------------------------------

node.name: opensearch-clustermanager
node.roles: [ cluster_manager, data ]

----------------------------------- Paths ------------------------------------

path.data: /var/lib/opensearch
path.logs: /var/log/opensearch

---------------------------------- Network -----------------------------------

network.host: 127.0.0.1,10.29.200.2
network.bind_host: 127.0.0.1,10.29.200.2

--------------------------------- Discovery ----------------------------------

discovery.seed_hosts: [“10.29.200.3”, “10.29.200.4”]
cluster.initial_cluster_manager_nodes: [“10.29.200.2”]
################ OpenSearch Security Configuration ########################################
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • ‘CN=A,OU=UNIT,O=OUR_ORG,L=OUR_CITY,ST=OUR_STATE,C=US’
    plugins.security.nodes_dn:
  • ‘CN=msp-lsbd183.dns.a-record,OU=UNIT,O=OUR_ORG,L=OUR_CITY,ST=OUR_STATE,C=US’
    plugins.security.audit.type: internal_opensearch
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.check_snapshot_restore_write_privileges: true
    plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
    Config for the data/ingest nodes

======================== OpenSearch Configuration =========================

---------------------------------- Cluster -----------------------------------

cluster.name: demo-opens-cluster

------------------------------------ Node ------------------------------------

node.name: opensearch-d2
node.roles: [ data, ingest ]

----------------------------------- Paths ------------------------------------

path.data: /var/lib/opensearch
path.logs: /var/log/opensearch

---------------------------------- Network -----------------------------------

network.host: 10.29.200.4
network.bind_host: 10.29.200.4

--------------------------------- Discovery ----------------------------------

discovery.seed_hosts: [“10.29.200.2”, “10.29.200.3”]
cluster.initial_cluster_manager_nodes: [“10.29.200.2”]
################ OpenSearch Security Configuration ########################################
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • ‘CN=A,OU=UNIT,O=OUR_ORG,L=OUR_CITY,ST=OUR_STATE,C=US’
    plugins.security.nodes_dn:
  • ‘CN=msp-lsbd183.dns.a-record,OU=UNIT,O=OUR_ORG,L=OUR_CITY,ST=OUR_STATE,C=US’
    plugins.security.audit.type: internal_opensearch
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.check_snapshot_restore_write_privileges: true
    plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]

Relevant Logs or Screenshots:

2

Hi @amorgan,

try moving your certificates to ../opensearch/config/ as per " …certificate’s key file (PKCS #8), which must be under the config directory…":

Best,
Mantas

3

Hi @amorgan,

Could you please clarify a few things:

How did you deploy your cluster?
How many nodes do you currently have in your cluster and what node roles are assigned?
Could you share opensearch.yml from one of the running manager cluster nodes? (note: please blank all/any sensitive information)

Thanks,
Mantas