Problem after adding ssl tls certificates

maxim · June 29, 2023, 1:28pm

hi, there was a problem after adding ssl tls certificates. The certificates are signed by a CA and are definitely valid. Tell me what could be the problem? OS version 2.8.0.
I attach the coordinator node settings, analog config for the node manager. + logs
connection with these certificates was checked by curl command
curl -v --cacert ca.pem --cert .pem --key .key -XGET https://hostname1-2node:9300
handshake succsess
client-server certificates
alt names dns and ip
what could be the problem?

Configuration:

cluster.name: os-cluster

node.name: os-coordinator-node
node.roles: [ coordinating ]

path.data: /var/lib/opensearch

path.logs: /var/log/opensearch

network.host: ip

http.port: 9200

discovery.seed_hosts: ["ip", "ip", "ip", "ip", "ip", "ip"]

cluster.initial_cluster_manager_nodes: ["ip", "ip"]

############################################################################
plugins.security.ssl.transport.pemcert_filepath: cert.pem
plugins.security.ssl.transport.pemkey_filepath: cert.key
plugins.security.ssl.transport.pemtrustedcas_filepath: ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: cert.pem
plugins.security.ssl.http.pemkey_filepath: cert.key
plugins.security.ssl.http.pemtrustedcas_filepath: ca.pem
plugins.security.nodes_dn:
  - 'C=la,O=lalala\ lala\ of\ the\ lala\ la,OU=la,CN=coordnodename'
  - 'C=la,O=la\ la\ of\ the\ la\ la,OU=la,CN=managernodename'


plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

Relevant Logs or Screenshots:
master logs
[2023-06-29T16:25:01,393][ERROR][o.o.s.t.SecurityRequestHandler] [os-manager-data-node] OpenSearchException[Transport client authentication no longer supported.]

coord logs
[2023-06-29T16:24:57,392][WARN ][o.o.d.HandshakingTransportAddressConnector] [os-coordinator-node] handshake failed for [connectToRemoteMasterNode[:9300]]
org.opensearch.transport.RemoteTransportException: [os-manager-data-node][masterIP:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported

Gsmitt · June 29, 2023, 10:58pm

Hey @maxim

The node in questioned is the configuration above? If so it seam it can not make a connection to you master node. Does your certs have a wild card or did you configure them individually? On each node, did the certificate/s get install in the keystore?

Doubel check the certificate with the configuration as shown below.

maxim · June 30, 2023, 7:23am

hey, that’s right, but I didn’t upload them to “opensearch.keystore”, is that the problem? if yes, how to do it?

I looked at the certificate installed for an example, it has the following parameter in alt names
what is this Registered ID parameter. and could this be the cause of this problem?
X509v3 Subject Alternative Name:
Registered ID:1.2.3.4.5.5, DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1

pablo · June 30, 2023, 3:29pm

@maxim If the CN of the OpenSearch node certificate doesn’t match any entry in node_dn then SAN is scanned against DNS, IP and RID (OID). If RID is present and has a value 1.2.3.4.5.5 then the node will be treated as a valid node certificate.

Otherwise, all nodes certificates’ DNs in the cluster must be listed in node_dn in opensearch.yml in every node.
In OpenSearch 2.x the Client authorization/authentication in the Transport layer has been deprecated. It means that if the node’s certificate doesn’t match the node_dn and has no RID(OID) configured it will be treated as a client certificate and client connection.

pablo · June 30, 2023, 3:33pm

As per the documentation, you should revert the order in the nodes_dn.

maxim · July 7, 2023, 8:14am

hello, unfortunately it didn’t work in my case, I filled in the nodes dn parameter in opensearch.yml in the reverse order, but the error remained… in my case, due to security requirements, I cannot issue a certificate with the specified rid.
today returning to the setup, I noticed a file that is located here
/etc/opensearch/opensearch-security/nodes_dn.yml should this file be populated in parallel with opensearch.yml?

pablo · July 7, 2023, 6:15pm

@maxim Could you share your opensearch.yml with the correct order in node_dn from both nodes?

Also, please share the certificate’s subject from both nodes.

openssl x509 -in <node_certificate> -subject -noout

maxim · July 10, 2023, 8:21am

@pablo opensearch.yml node 1

cluster.name: testik
node.name: node-1
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 10.0.0.1
http.port: 9200
discovery.seed_hosts: ["10.0.0.1", "10.0.0.2"]

plugins.security.ssl.transport.pemcert_filepath: cert.pem
plugins.security.ssl.transport.pemkey_filepath: key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: cert.pem
plugins.security.ssl.http.pemkey_filepath: key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de
plugins.security.nodes_dn:
  - 'CN=example.hostname.node1.domain.com,OU=00AA,O=Example Example,C=RU'
  - 'CN=example.hostname.node2.domain.com,OU=00AA,O=Example Example,C=RU'

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models"]
node.max_local_storage_nodes: 3

opensearch.yml node 2

cluster.name: testik
node.name: node-2
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 10.0.0.2
http.port: 9200
discovery.seed_hosts: ["10.0.0.1", "10.0.0.2"]

plugins.security.ssl.transport.pemcert_filepath: cert.pem
plugins.security.ssl.transport.pemkey_filepath: key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: cert.pem
plugins.security.ssl.http.pemkey_filepath: key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de
plugins.security.nodes_dn:
  - 'CN=example.hostname.node1.domain.com,OU=00AA,O=Example Example,C=RU'
  - 'CN=example.hostname.node2.domain.com,OU=00AA,O=Example Example,C=RU'


plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models"]
node.max_local_storage_nodes: 3

output openssl command

[root@hostname.node1 /etc/opensearch]# openssl x509 -in cert.pem -subject -noout
subject=C = RU, O = Example Example, OU = 00AA, CN = example.hostname.node1.domain.com

[root@hostname.node2 /etc/opensearch]# openssl x509 -in cert.pem -subject -noout
subject=C = RU, O = Example Example, OU = 00AA, CN = example.hostname.node2.domain.com

maxim · July 10, 2023, 10:13am

@pablo @Gsmitt
Thanks for the help, I managed to figure it out.
when issuing the certificate in the “OU” field, I had the name of the organization with spaces. I re-issued the certificate without spaces in this parameter, corrected the config and everything worked

maxim · July 10, 2023, 10:26am

the error does not say at all which way to look … I would like to see more specifics in such cases in the logs(

Topic		Replies	Views
Can't connect two nodes with tls + ssl Security troubleshoot	2	689	March 28, 2022
Installing tls ssl certificates Security troubleshoot , configure	3	448	June 27, 2023
TLS configuration with organizational certs Security troubleshoot , configure	2	503	January 21, 2022
Certificate configuration Error Security troubleshoot , configure	5	1239	June 26, 2023
Unable to configure newly generated SSL certificates on a 3 node opensearch cluster Security configure	3	531	April 20, 2022

Problem after adding ssl tls certificates

Related Topics