Adding new node to existing one node cluster

opensearchforme · July 11, 2025, 5:20pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

3.1

Describe the issue:
Adding new node to existing one node cluster using same certificate

Configuration:

I have one node cluster running fine. trying to add new one to it using same certifcates

this is my node1 configuration. and node2 is exactly same.

# Disable https
plugins.security.ssl.http.enabled: false
plugins.security.allow_default_init_securityindex: true

# disable demo certi, and use custom cert
plugins.security.allow_unsafe_democertificates: false


# enable REST API access
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

# internode security
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/ca_config/os_shared_node_key.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/ca_config/os_shared_node.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/ca_config/root_ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false

now copy whole /etc/opensearch/ca_config dir to new node2.
setup same thing on node2 and started opensearch

Relevant Logs or Screenshots:
This is error message I get

Node1:
[2025-07-11T11:48:17,793][ERROR][o.o.s.t.SecurityRequestHandler] [osdev01] OpenSearchException[Node presenting certificate with SSL Principal {O=XYZ,L=HOUSTON,ST=TX,C=US} could not securely connect to the cluster. Please ensure the principal is correct and present in the nodes_dn list.]

Node2:
[2025-07-11T11:48:22,887][INFO ][o.o.s.c.ConfigurationRepository] [osdev02] Wait for cluster to be available ...
[2025-07-11T11:48:23,789][WARN ][o.o.d.HandshakingTransportAddressConnector] [osdev02] handshake failed for [connectToRemoteMasterNode[10.21.226.97:9300]]
org.opensearch.transport.RemoteTransportException: [osdev01][10.21.226.97:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Node presenting certificate with SSL Principal {O=XYZ,L=HOUSTON,ST=TX,C=US} could not securely connect to the cluster. Please ensure the principal is correct and present in the nodes_dn list.

Anthony · July 14, 2025, 9:49am

@opensearchforme The nodes need to know which other nodes they need to connect to form a cluster using `discovery.seed_hosts, further details are available here

Also the nodes need to know which certificates should be treated as “node-to-node” communication, this is achieved using the plugins.security.nodes_dn configuration example available here

Hope this helps

opensearchforme · July 14, 2025, 4:19pm

I already have that information in node2

node2 ( new node)

cluster.name: dev_os
node.name: houosdev02
node.roles: [ "master", "data", "ingest", "ml", "remote_cluster_client", "transform" ]
path.data: /s1/opensearch
path.logs: /s1/log/opensearch
network.host: 10.21.226.98
http.port: 9200
discovery.seed_hosts: ["osdev01", "osdev02"]
cluster.initial_cluster_manager_nodes: ["osdev01"]
plugins.security.ssl.http.enabled: false
plugins.security.allow_default_init_securityindex: true
plugins.security.allow_unsafe_democertificates: false
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/ca_config/os_shared_node_key.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/ca_config/os_shared_node.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/ca_config/ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false

Anthony · July 15, 2025, 1:10pm

@opensearchforme this configuration is needed on both nodes and I dont see the nodes_dn configuration on the above, perhaps it was omitted?

opensearchforme · July 15, 2025, 1:55pm

added this on node2 still same error. yes node1 has same configuration.


plugins.security.authcz.admin_dn:
  - "O=XYX,L=HOUSTON,ST=TX,C=US"

Anthony · July 15, 2025, 1:58pm

@opensearchforme the admin_dn is not the correct configuration, this configuration tells opensearch cluster which certificate should be treated as admin certificate.

The following is the correct configuration:

plugins.security.nodes_dn

Can you provide the opensearch.yml file for both of the nodes please?

opensearchforme · July 15, 2025, 4:13pm

node1

cluster.name: dev_os
node.name: osdev01
node.roles: [ "master", "data", "ingest", "ml", "remote_cluster_client", "transform" ]
path.data: /s1/opensearch
path.logs: /s1/log/opensearch
network.host: 10.21.226.97
http.port: 9200
discovery.seed_hosts: ["osdev01", "osdev02"]
plugins.security.ssl.http.enabled: false
plugins.security.allow_default_init_securityindex: true
plugins.security.allow_unsafe_democertificates: false
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/ca_config/os_shared_node_key.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/ca_config/os_shared_node.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/ca_config/ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false

Node2

cluster.name: dev_os
node.name: osdev02
node.roles: [ "master", "data", "ingest", "ml", "remote_cluster_client", "transform" ]
path.data: /s1/opensearch
path.logs: /s1/log/opensearch
network.host: 10.21.226.98
http.port: 9200
discovery.seed_hosts: ["osdev01", "osdev02"]
cluster.initial_cluster_manager_nodes: ["osdev01"]
plugins.security.ssl.http.enabled: false
plugins.security.allow_default_init_securityindex: true
plugins.security.allow_unsafe_democertificates: false
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/ca_config/os_shared_node_key.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/ca_config/os_shared_node.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/ca_config/ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.authcz.admin_dn:
  - "O=XYZ,L=HOUSTON,ST=TX,C=US"

All three files are same os_shared_node_key.pem, os_shared_node.pem, ca.pem. I created that on Node1.
On Node1 build the cluster I copy them over to Node2

Anthony · July 16, 2025, 7:46am

@opensearchforme There are 2 configurations that needs update:

The cluster.initial_cluster_manager_nodes setting is only needed when you first bootstrap your (one node) cluster, after this setting is ignored (and should be removed). Therefore should not be present in node2 opensearch.yml configuration at all.
As mentioned in the last replies, the nodes_dn needs to be set on both opensearch.yml files (node1 and node2). In the demo configuration this setting looks like this:

plugins.security.nodes_dn:
  - CN=node-*.example.com,OU=node,O=node,L=test,C=de

Make sure that the admin_dn configuration that you have added to node2, doesn’t conflict with node certificates. Meaning that the subject name of the node certificate that will be used for inter-node communication is not included in the below (if you are using wildcards)

plugins.security.authcz.admin_dn:
  - "O=XYZ,L=HOUSTON,ST=TX,C=US"

Perhaps in your case removing this setting altogether until you get a running cluster might be a good idea.

For reference, please see my node1 configuration:

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: ['CN=kirk,OU=client,O=client,L=test,C=de']
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
  .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
  .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
  .plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
  .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
  .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
  .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
  .opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
  .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
  .plugins-flow-framework-state]
network.host: 0.0.0.0
node.name: smoketestnode1
  #cluster.initial_cluster_manager_nodes: smoketestnode1
plugins.security.nodes_dn:
  - CN=node-*.example.com,OU=node,O=node,L=test,C=de
node.max_local_storage_nodes: 3
discovery.seed_hosts: ["<node1-IP>", "<node2-IP>"]
######## End OpenSearch Security Demo Configuration ########

and “node2”

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: ['CN=kirk,OU=client,O=client,L=test,C=de']
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
  .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
  .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
  .plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
  .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
  .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
  .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
  .opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
  .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
  .plugins-flow-framework-state]
network.host: 0.0.0.0
node.name: smoketestnode2
plugins.security.nodes_dn:
  - CN=node-*.example.com,OU=node,O=node,L=test,C=de
node.max_local_storage_nodes: 3
discovery.seed_hosts: ["<node1-IP>", "<node2-IP>"]
######## End OpenSearch Security Demo Configuration ########

opensearchforme · July 16, 2025, 1:22pm

Thanks @Anthony for detail reply. I do have correct nodes_dn as I got that inform from .pem. But like you said let me remove everything and do it fresh start.
I do fully understand about initial cluster and stuff. just got tangle up on this security thing. I have good experience with Elasticsearch and this is identical.

opensearchforme · July 16, 2025, 6:40pm

It is working now.

at first node I have to enable ssl.https and once it is setup. I can disable them.
both nodes are up in cluster now.
even added third node. now fun part settting up template,ilm, role,user,dashbaord etc..

Topic		Replies	Views
Why can't I create a cluster? OpenSearch discuss , troubleshoot , configure , install	10	116	July 1, 2025
Unable to configure newly generated SSL certificates on a 3 node opensearch cluster Security configure	3	681	April 20, 2022
How to add nodes to an existing cluster Security	8	3823	July 6, 2021
OpenSearch Security not initialized in Cluster nodes Security configure	5	330	April 3, 2025
Nodes unable to communicate while creating cluster with custom CA files Security configure , install	2	76	September 20, 2024

Adding new node to existing one node cluster

Related topics