@michael.anderton I don’t see volume mapping for config.yml so I assume that you’re using basic authentication.
If that’s true then I don’t understand why your DC connects to the OpenSearch cluster. Is that where you execute curl commands or access OpenSearch Dashboards?
@pablo
I was assuming that it was because the opensearch cluster was reaching out to verify the certificate.
Update! Both nodes are running with my certs. I can log in via https://fqdn:5601 and the browser shows a valid certificate. When viewing the logs of the nodes, I am still seeing “bad_certificate” on node1. Node2 is not showing any errors. The current error is below.
[ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [opensearch-node1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
The bad_certificate error is related to the nodes talking between each other, not what’s shown on the WebGUI. For some reason, the second node has some kind of bad certificate and can’t communicate. Can you try maybe just copying with a different name and see if that works. Otherwise, I’d suggest making a totally separate certificate for node 2 and try that. The bad_certificate error should clear OpenSearch can validate 1) the subject of the certificate matches what’s put in plugins.security.nodes_dn and 2) that the certificate is signed by the cert listed in plugins.security.ssl.transport.pemtrustedcas_filepath.
@pauchamb
Could you clairify “copying with a different name” please?
Absolutely - sorry about that!
Can you copy cert.pem to cert2.pem and key.pem to key2.pem. Then change your docker compose to look like this under the opensearch-node2 volumes section:
volumes:
- opensearch-data1:/usr/share/opensearch/data
- ./config/opensearch.yml:/usr/share/opensearch/config/opensearch.yml
- ./cert.pem:/usr/share/opensearch/config/cert2.pem
- ./key.pem:/usr/share/opensearch/config/key2.pem
@pauchamb I don’t think this will work as you expected. @michael.anderton is using the same opensearch.yml file for both containers.
This would require a separate opensearch.yml file for node2.
I’ve tested a single certificate in my lab and it worked. I have the same certificates and single node_dn entries in both nodes.
@michael.anderton In regards to your DC and certificate verification, that wouldn’t initiate the connection from the DC node to the OpenSearch cluster.
Maybe you have some service that is trying to connect to the OpenSearch cluster.
@pauchamb
I get “unauthorized” when running the command in your first image. Below is what I get from the second.
I think this was meant for @pablo.
To your point @pablo, I definitely had a type which is that I wanted to actually put:
volumes:
- opensearch-data1:/usr/share/opensearch/data
- ./config/opensearch.yml:/usr/share/opensearch/config/opensearch.yml
- ./cert2.pem:/usr/share/opensearch/config/cert.pem
- ./key2.pem:/usr/share/opensearch/config/key.pem
That above config within the docker-compose file would allow him to have the same opensearch.yml file.
@michael.anderton for the first command, could you try curl --insecure -u admin:admin -XGET https://localhost:9200/_plugins/_security/api/ss/certs?pretty instead?
@pauchamb This curl requires certificate authentication. Admin user is not authorized to see results.
@michael.anderton Do you have any admin_dn defined in the opensearch.yml?
That command says
"error" : "no handler found for uri [/_plugins/_security/api/ss/certs] and method [GET]"
I have implemented your above volumes block into my docker-compose.yml file. While I think this is a good idea, I dont know if it had any effect.
The only error I am seeing now is the “bad_certificate” from above where the remote address is the DC.
@michael.anderton It should be /_plugins/_security/api/ssl/certs
Did you check for any services/scripts/health checks that could be triggered against the OpenSearch cluster?
How often do you see that error? Is it a fixed time?
Roughly every 20-30 seconds. I am not aware of any services/scripts/health checks that are happening. I do know we have our own cert revocation list present on the DC as well.