Transport SSL Certificate Error

SPrinz · March 11, 2022, 11:11am

Hello Everyone,

i encountered the following Error:

This is the error

[2022-03-11T11:59:14,160][WARN ][o.o.d.HandshakingTransportAddressConnector] [xxxdb12] handshake failed for [connectToRemoteMasterNode[[xxx:7:b::11]:9300]]
org.opensearch.transport.RemoteTransportException: [xxxdb11][[xxx:7:b::11]:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/

This is the excerpt from the opensearch.yml
plugins.security.nodes_dn: 'C = DE,ST = Hamburg,L = Hamburg,O =xxx AG,CN = \*.xxx.de'

This is the Subject of our SSL Wildcard Certificate
Subject: C = DE, ST = Hamburg, L = Hamburg, O = xxx AG, CN = *.xxx.de

Where is our error?

Thx in advance!

pablo · March 11, 2022, 12:02pm

@SPrinz Could you check if your node certificate has TLS Web Server Authentication, in Extended Key Usage?

Did you try to use FQDN instead of the wildcard in the CN of plugins.security.nodes_dn:?

SPrinz · March 11, 2022, 2:58pm

Hey Pablo ,

ty for the Reply.

the Node certificate indeed has TLS Web Server Authentication in Extended Key Usage.

Tho when we use the FQDN the error still occurs. Same one nothing changed.

Have you got any more ideas ?
Kind regards

pablo · March 11, 2022, 3:02pm

@SPrinz That error appears when you try to connect to a remote cluster?
I can see IPv6 in WARN content. Is it enabled in your environment?

SPrinz · March 14, 2022, 12:00pm

@pablo Yes we only use IPv6 in our environment.

pablo · March 23, 2022, 3:51pm

@SPrinz I didn’t get the answer to my first question. Does it happen when you connect to a remote cluster?
Do you have any issues in your environment because of that error?

SPrinz · March 24, 2022, 11:22am

Yes it happens when i try to form a Cluster. As of now i dont have a Cluster i just have Seperate Servers which im trying to merge into a Cluster.

pablo · March 24, 2022, 1:07pm

@SPrinz Could you share opensearch.yml from two different nodes?
Is it docker or physical deployment?

SPrinz · March 25, 2022, 2:26pm

@pablo Its a Manual Install on a virtual Maschine

Node 1

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: Prod
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: Node1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
node.attr.dc: 01
node.attr.temp: warm
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
network.host: "[xxx:xxx:x:a::11]"
http.port: 9200
transport.host: "[xxx:xxx:x:b::11]"
transport.tcp.compress: true
#

# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
discovery.seed_hosts: ["[xxx:xxx:x:b::11]", "[xxx:xxx:x:b::10]", "[xxx:xxx:x:b::9]", "[xxx:xxx:x:b::8]", "[xxx:xxx:x:b::12]"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
cluster.initial_master_nodes: ["[xxx:xxx:x:b::12]", "[xxx:xxx:x:b::11]", "[xxx:xxx:x:b::10]", "[xxx:xxx:x:b::9]", "[xxx:xxx:x:b::8]"]
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: certs/wildcard.xxx.de_GeoTrust.crt
plugins.security.ssl.transport.pemkey_filepath: certs/wildcard.xxx.de_GeoTrust.key
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/wildcard.xxx.de_GeoTrust.ca.crt
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certs/wildcard.xxx.de_GeoTrust.crt
plugins.security.ssl.http.pemkey_filepath: certs/wildcard.xxx.de_GeoTrust.key
plugins.security.ssl.http.pemtrustedcas_filepath: certs/wildcard.xxx.de_GeoTrust.ca.crt
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.nodes_dn: 'C = DE,ST = Hamburg,L = Hamburg,O = xxx AG,CN = \*.xxx.de'

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

Node 2

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: Prod
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: Node2
#
# Add custom attributes to the node:
#
node.attr.dc: 02
node.attr.temp: warm
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
# network.host: 192.168.0.1
#
network.host: "[xxx:xxx:x:a::12]"
http.port: 9200
transport.host: "[xxx:xxx:x:b::12]"
transport.tcp.compress: true

#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
# For more information, consult the discovery and cluster formation module documentation.
discovery.seed_hosts: ["[xxx:xxx:x:b::11]", "[xxx:xxx:x:b::10]", "[xxx:xxx:x:b::9]", "[xxx:xxx:x:b::8]", "[xxx:xxx:x:b::12]"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
cluster.initial_master_nodes: ["[xxx:xxx:x:b::12]", "[xxx:xxx:x:b::11]", "[xxx:xxx:x:b::10]", "[xxx:xxx:x:b::9]", "[xxx:xxx:x:b::8]"]

#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
gateway.expected_nodes: 5
gateway.expected_master_nodes: 5
gateway.expected_data_nodes: 5
gateway.recover_after_master_nodes: 3
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemcert_filepath: certs/wildcard.xxx.de_GeoTrust.crt
plugins.security.ssl.transport.pemkey_filepath: certs/wildcard.xxx.de_GeoTrust.key
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/wildcard.xxx.de_GeoTrust.ca.crt
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certs/wildcard.xxx.de_GeoTrust.crt
plugins.security.ssl.http.pemkey_filepath: certs/wildcard.xxx.de_GeoTrust.key
plugins.security.ssl.http.pemtrustedcas_filepath: certs/wildcard.xxx.de_GeoTrust.ca.crt
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.nodes_dn: 'C = DE,ST = Hamburg,L = Hamburg,O  =xxx AG,CN = \*.xxx.de'

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

pablo · March 25, 2022, 7:15pm

@SPrinz I think your plugins.security.nodes_dn: has incorrect order. I know it appears like that when you run openssl x509 command. However, dn starts with CN and ends with C.

Try below instead in opensearch.yml file.

plugins.security.nodes_dn: 
  - 'CN= \*.xxx.de,O=xxx AG,L=Hamburg,ST = Hamburg,C=DE'

SPrinz · March 28, 2022, 11:39am

@pablo We changed the plugins.security.nodes_dn to the correct order but we still have the same error. Nothing Changed.

plugins.security.nodes_dn:
  - 'CN = \*.xxx.de,O = xxx AG,L = Hamburg,ST = Hamburg,C = DE'

With and without the Spaces around the “=”, nothing changed

pablo · March 28, 2022, 11:55am

@SPrinz Would you mind removing backslash from CN? According to documentation, it is not needed.

pablo · March 28, 2022, 1:37pm

@SPrinz Also, please remove all the spaces around the"=" sign.

SPrinz · April 28, 2022, 1:01pm

@pablo Ty for all the Help . Sorry i didnt write in so long. We have a functional Cluster now. We did what you told us and used a new Wildcard Certificate. So ty and this Thread is resolved

Topic		Replies	Views
Docker OpenSearch : Wildcard SSL TLS handshake issue Security troubleshoot	14	1379	April 14, 2022
Illegal parameter in http or transport request found Security	1	4729	August 4, 2019
Caused by: org.elasticsearch.ElasticsearchException: Illegal parameter in http or transport request found Open Source Elasticsearch and Kibana	1	1091	May 8, 2020
Opendistro_security.nodes_dn configuration error Security	2	1488	April 15, 2021
Security.nodes_dn incorrect configured Security	1	1005	February 7, 2023

Transport SSL Certificate Error

Related topics