Versions (opensearch 3.4.0/Server OS: Windows 2019/Browser: No browser):
Describe the issue
Please help me understand why my nodes aren’t forming a cluster.
OpenSearch 3.4.0 is installed on two Windows nodes.
Ports 9300 are open in both directions. Both nodes are on the corporate network.
The certificates are different but were issued by the same CA (an internal corporate CA).
Each node works fine individually, but they won’t join together into a cluster.
Here is what I see on the first node:
curl -X GET <os_service_node1_url:9200> -u “admin:OpSrch2026@pa$$wd” --insecure
10.34.200.26 1 61 5 dimr cluster_manager,data,ingest,remote_cluster_client * osbr01
And here is what I see on the second node:
curl -X GET <os_service_node1_url:9200> -u “admin:OpSrch2026@pa$$wd” --insecure
11.60.14.23 3 58 16 dimr cluster_manager,data,ingest,remote_cluster_client * osbr02
Interestingly, if the `plugins.security.nodes_dn` parameter is removed from `opensearch.yml`:
An error appears in the second node’s log.
[2026-06-26T23:13:26,343][WARN ][o.o.d.HandshakingTransportAddressConnector] [osbr02] handshake failed for [connectToRemoteMasterNode[10.34.200.26:9300]]
org.opensearch.transport.RemoteTransportException: [osbr01][10.34.200.26:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Node presenting certificate with SSL Principal {EMAILADDRESS=br_support@mts.ru,CN=osbr02,OU=OSPJSC,O=OS Public Joint Stock Company,L=Town,ST=State,C=COM} could not securely connect to the cluster. Please ensure the principal is correct and present in the nodes_dn list.
I added the `plugins.security.nodes_dn` parameter;
the error disappeared, but the nodes still did not join the cluster.
My notes:
-
It might not be working because an internal corporate CA is being used; perhaps it needs to be registered as trusted in a Java keystore.
However, I couldn’t find anything relevant in the documentation. -
The issue might be that the nodes are located in different network subdomains—specifically, the third-level subdomains differ. Compare `wcaexp.nnv.stt.com` and `osbr02.twn.stt.com`.
-
After starting both nodes, I observe network traffic on port 9300.
netstat -an | findstr “:9200 :9300” | find /v “[::]”
TCP 0.0.0.0:9200 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9300 0.0.0.0:0 LISTENING
TCP 10.34.200.26:60857 11.60.14.23:9300 ESTABLISHED
and
netstat -an | findstr “:9200 :9300” | find /v “[::]”
TCP 0.0.0.0:9200 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9300 0.0.0.0:0 LISTENING
TCP 11.60.14.23:9300 10.34.200.26:60857 ESTABLISHED
It seems like they are trying to unite, but something is standing in the way.
Any ideas?
Configuration:
Warn!
The email address in the `nodes_dn` line is correct. I inserted a space between the @ symbol and the domain because the system wouldn’t allow the request to be submitted otherwise.
first node: opensearch.yml. IP address:10.34.200.26
cluster.name: osbr
node.name: osbr01
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: [“10.34.200.26:9300”, “11.60.14.23:9300”]
cluster.initial_cluster_manager_nodes: [“osbr01”, “osbr02”]
plugins.security.ssl.transport.pemcert_filepath: certs/osbr-bundle.pem
plugins.security.ssl.transport.pemkey_filepath: certs/osbr.key
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/ca-bundle.pem
transport.ssl.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certs/osbr-bundle.pem
plugins.security.ssl.http.pemkey_filepath: certs/osbr.key
plugins.security.ssl.http.pemtrustedcas_filepath: certs/ca-bundle.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.nodes_dn:
- “EMAILADDRESS=br_support @mts.rumts.ru,CN=osbr01,OU=OSPJSC,O=OS Public Joint Stock Company,L=Town,ST=State,C=COM”
- “EMAILADDRESS=br_su@mts.ruport @mts.ru,CN=osbr02,OU=OSPJSC,O=OS Public Joint Stock Company,L=Town,ST=State,C=COM”
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
node.max_local_storage_nodes: 3
Second node: opensearch.yml. IP address 11.60.14.23
cluster.name: osbr
node.name: osbr02
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: [“10.34.200.26:9300”, “11.60.14.23:9300”]
cluster.initial_cluster_manager_nodes: [“osbr01”, “osbr02”]
plugins.security.ssl.transport.pemcert_filepath: certs/osbr-bundle.pem
plugins.security.ssl.transport.pemkey_filepath: certs/osbr.key
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/ca-bundle.pem
transport.ssl.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certs/osbr-bundle.pem
plugins.security.ssl.http.pemkey_filepath: certs/osbr.key
plugins.security.ssl.http.pemtrustedcas_filepath: certs/ca-bundle.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.nodes_dn:
- “EMAILADDRESS@mts.rubr_support @mts.ru,CN=osbr01,OU=OSPJSC,O=OS Public Joint Stock Company,L=Town,ST=State,C=COM”
- “EMAILADDRESS=br_support @mts.ru,CN=osbr02,OU=OSPJSC,O=OS Public Joint Stock Company,L=Town,ST=State,C=COM”
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
node.max_local_storage_nodes: 3
Relevant Logs or Screenshots
--------- first node log without [INFO] string ----------
[2026-06-28T13:41:26,125][WARN ][stderr ] [osbr01] июн. 28, 2026 1:41:26 PM org.opensearch.javaagent.bootstrap.AgentPolicy setPolicy
[2026-06-28T13:41:26,131][WARN ][stderr ] [osbr01] INFO: Policy attached successfully: org.opensearch.bootstrap.OpenSearchPolicy@84487f4
[2026-06-28T13:41:26,493][WARN ][o.o.p.PluginsService ] [osbr01] Missing plugin [workload-management], dependency of [opensearch-security]
[2026-06-28T13:41:26,494][WARN ][o.o.p.PluginsService ] [osbr01] Some features of this plugin may not function without the dependencies being installed.
[2026-06-28T13:41:26,741][WARN ][stderr ] [osbr01] июн. 28, 2026 1:41:26 PM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider
[2026-06-28T13:41:26,742][WARN ][stderr ] [osbr01] INFO: Java vector incubator API enabled; uses preferredBitSize=128; floating-point vectors only
[2026-06-28T13:41:37,784][WARN ][stderr ] [osbr01] SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”.
[2026-06-28T13:41:37,785][WARN ][stderr ] [osbr01] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2026-06-28T13:41:37,791][WARN ][stderr ] [osbr01] SLF4J: See SLF4] Error Codes for further details.
[2026-06-28T13:41:43,406][WARN ][stderr ] [osbr01] SLF4J(W): No SLF4J providers were found.
[2026-06-28T13:41:43,407][WARN ][stderr ] [osbr01] SLF4J(W): Defaulting to no-operation (NOP) logger implementation
[2026-06-28T13:41:43,412][WARN ][stderr ] [osbr01] SLF4J(W): See SLF4] Error Codes for further details.
[2026-06-28T13:41:43,673][WARN ][o.o.s.c.Salt ] [osbr01] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2026-06-28T13:41:43,673][WARN ][o.o.s.c.Salt ] [osbr01] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2026-06-28T13:41:45,098][WARN ][o.o.s.p.SQLPlugin ] [osbr01] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here:
[2026-06-28T13:41:48,852][WARN ][o.o.g.DanglingIndicesState] [osbr01] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2026-06-28T13:41:48,923][WARN ][o.o.m.a.p.TransportPredictionStreamTaskAction] [osbr01] StreamTransportService is not available.
[2026-06-28T13:41:49,696][WARN ][o.o.m.a.e.TransportExecuteStreamTaskAction] [osbr01] StreamTransportService is not available.
[2026-06-28T13:41:51,883][WARN ][o.o.s.SecurityAnalyticsPlugin] [osbr01] Failed to initialize LogType config index and builtin log types
[2026-06-28T13:41:54,012][WARN ][o.o.s.a.r.AuditMessageRouter] [osbr01] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2026-06-28T13:42:21,575][WARN ][o.o.c.InternalClusterInfoService] [osbr01] No resource usage stats available for node: osbr01
--------- second node log without [INFO] string ----------
[2026-06-28T13:42:09,930][WARN ][stderr ] [osbr02] Jun 28, 2026 1:42:09 PM org.opensearch.javaagent.bootstrap.AgentPolicy setPolicy
[2026-06-28T13:42:09,935][WARN ][stderr ] [osbr02] INFO: Policy attached successfully: org.opensearch.bootstrap.OpenSearchPolicy@72456279
[2026-06-28T13:42:10,241][WARN ][o.o.p.PluginsService ] [osbr02] Missing plugin [workload-management], dependency of [opensearch-security]
[2026-06-28T13:42:10,243][WARN ][o.o.p.PluginsService ] [osbr02] Some features of this plugin may not function without the dependencies being installed.
[2026-06-28T13:42:10,578][WARN ][stderr ] [osbr02] Jun 28, 2026 1:42:10 PM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider
[2026-06-28T13:42:10,580][WARN ][stderr ] [osbr02] INFO: Java vector incubator API enabled; uses preferredBitSize=512; FMA enabled
[2026-06-28T13:43:06,415][WARN ][stderr ] [osbr02] SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”.
[2026-06-28T13:43:06,417][WARN ][stderr ] [osbr02] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2026-06-28T13:43:06,420][WARN ][stderr ] [osbr02] SLF4J: See SLF4] Error Codes for further details.
[2026-06-28T13:43:11,298][WARN ][stderr ] [osbr02] SLF4J(W): No SLF4J providers were found.
[2026-06-28T13:43:11,300][WARN ][stderr ] [osbr02] SLF4J(W): Defaulting to no-operation (NOP) logger implementation
[2026-06-28T13:43:11,303][WARN ][stderr ] [osbr02] SLF4J(W): See SLF4] Error Codes for further details.
[2026-06-28T13:43:11,600][WARN ][o.o.s.c.Salt ] [osbr02] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2026-06-28T13:43:11,602][WARN ][o.o.s.c.Salt ] [osbr02] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2026-06-28T13:43:12,797][WARN ][o.o.s.p.SQLPlugin ] [osbr02] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here:
[2026-06-28T13:43:16,186][WARN ][o.o.g.DanglingIndicesState] [osbr02] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2026-06-28T13:43:16,241][WARN ][o.o.m.a.p.TransportPredictionStreamTaskAction] [osbr02] StreamTransportService is not available.
[2026-06-28T13:43:16,888][WARN ][o.o.m.a.e.TransportExecuteStreamTaskAction] [osbr02] StreamTransportService is not available.
[2026-06-28T13:43:18,957][WARN ][o.o.s.SecurityAnalyticsPlugin] [osbr02] Failed to initialize LogType config index and builtin log types
[2026-06-28T13:43:21,285][WARN ][o.o.s.a.r.AuditMessageRouter] [osbr02] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2026-06-28T13:43:48,685][WARN ][o.o.c.InternalClusterInfoService] [osbr02] No resource usage stats available for node: osbr02