Details of the issue and a reproducer are in the [BUG]There is no complete mTLS handshake when REST client connects to mTLS enabled opensearch ES · Issue #2067 · opensearch-project/security · GitHub but I was asked to post it here.
I would appreciate if someone post here working sample of mTLS enabled OpenSearch REST based client
@lsa5521 In your example you’re using the following command to test.
curl --key conf/devpclcl1-pclmtls-client-cert-secret_key.key --cert conf/devpclcl1-pclmtls-client-cert-secret_cert.crt -XGET https://deves.pcl.psr.us-ashburn-1.ocp.oraclecloud.com:9200 -u 'admin:admin' -v
This command uses two types of authentication: basic and client certificate.
If the cert authentication is second and basic first then successful basic authentication (-u admin:admin) will ignore certificate authentication.
I’d like to suggest the following command to test mTLS in the OpenSearch.
curl --key conf/devpclcl1-pclmtls-client-cert-secret_key.key --cert conf/devpclcl1-pclmtls-client-cert-secret_cert.crt -XGET https://deves.pcl.psr.us-ashburn-1.ocp.oraclecloud.com:9200 -v
@lsa5521 Just to be clear. The output of the curl to the OpenSearch cluster described in the README file is the expected mTLS communication.
However, according to your tests, the java REST client doesn’t use mTLS.
Could you provide tcpdumps of your REST Client tests?
@pablo Here is below the output of the SSLDUMP tool, which I used to see mTLS handshake sequence on the OpenSearch server side.
In the case of the CURL client:
ssldump -i ens3 -dnq ‘port 9200’
New TCP connection #1: 100.100.123.173(28570) <-> 100.100.123.176(9200)
1 1 0.1586 (0.1586) C>S Handshake ClientHello
extensions
Short read: -2571 bytes available (expecting 2)
1 2 0.1798 (0.0211) S>C Handshake ServerHello
extensions
extended_master_secret
renegotiation_info
ja3s string: 771,49200,23-65281
ja3s fingerprint: ae4edc6faf64d08308082ad26be60767
Certificate
ServerKeyExchange
Not enough data. Found 294 bytes (expecting 32767)
CertificateRequest
Not enough data. Found 592 bytes (expecting 32767)
ServerHelloDone
1 3 0.1957 (0.0159) C>S Handshake Certificate
ClientKeyExchange
CertificateVerify
Not enough data. Found 258 bytes (expecting 32767)
1 4 0.1957 (0.0000) C>S ChangeCipherSpec
1 5 0.1957 (0.0000) C>S Handshake
1 6 0.2111 (0.0153) S>C ChangeCipherSpec
1 7 0.2111 (0.0000) S>C Handshake
1 8 0.2119 (0.0008) C>S application_data
1 9 0.2159 (0.0040) S>C application_data
1 10 0.2168 (0.0009) C>S Alert
1 0.2169 (0.0000) C>S TCP FIN
1 11 0.2174 (0.0005) S>C Alert
1 0.2178 (0.0003) S>C TCP FIN
The key things here related to the client to look at are:
CertificateRequest (server asks client to send the client certificate);
CertificateVerify (server verifies certificate sent by client).
There is no such info in the SSLDUMP, which was collected for the OpenSearch REST with mTLS client:
ssldump -i ens3 -dnq ‘port 9200’
New TCP connection #1: 10.98.138.71(50085) <-> 100.100.123.176(9200)
1 1 0.3006 (0.3006) C>S Handshake ClientHello
extensions
Short read: -324 bytes available (expecting 2)
1 2 0.3220 (0.0213) S>C Handshake ServerHello
extensions
supported_versions
key_share
ja3s string: 771,4866,43-51
ja3s fingerprint: 15af977ce25de452b96affa2addb1036
1 3 0.3220 (0.0000) S>C ChangeCipherSpec
1 4 0.3220 (0.0000) S>C application_data
1 5 0.4851 (0.1630) C>S ChangeCipherSpec
1 6 0.8149 (0.3298) C>S application_data
1 7 0.8183 (0.0034) C>S application_data
1 8 0.8198 (0.0014) S>C application_data
1 9 1.2064 (0.3865) S>C application_data
1 1.4125 (0.2061) C>S TCP FIN
1 1.4127 (0.0002) S>C TCP FIN
This looks strange because if server (OpenSearch ES in this case ) has mTLS enabled then the mTLS client should not be able to connect. But ,as I have mentioned, it works fine functionally and only concern is that there is no expected steps of the mTLS handshake in the TCPDUMP. Could it be result of some caching…?
@lsa5521 I’ve noticed that you’ve used username and password authentication along with certificates in the RESTClient script. Have you tried to run without it, and use only certificates?
@pablo TLS handshake and authentication happen on different communication layers (OCI Model).
So, the authentication would not affect the handshake.
Anyways, I have removed the setting for authentication and results was the same - no complete mTLS handshake.
TCPDUMP on the server side is below.
[root@slinetsk-psrcorp ssldump-0.9b3]# /usr/sbin/ssldump -i ens3 -dnq ‘port 9200’
New TCP connection #1: 10.74.106.233(65340) <-> 100.100.123.176(9200)
1 1 0.3426 (0.3426) C>S Handshake ClientHello
extensions
Short read: -324 bytes available (expecting 2)
1 2 0.3978 (0.0551) S>C Handshake ServerHello
extensions
supported_versions
key_share
ja3s string: 771,4866,43-51
ja3s fingerprint: 15af977ce25de452b96affa2addb1036
1 3 0.3978 (0.0000) S>C ChangeCipherSpec
1 4 0.3978 (0.0000) S>C application_data
1 5 0.7499 (0.3520) C>S ChangeCipherSpec
1 6 0.8393 (0.0893) C>S application_data
1 7 0.8425 (0.0032) C>S application_data
1 8 0.8491 (0.0066) S>C application_data
1 9 0.8520 (0.0028) S>C application_data