What CA is used by notifications channels?

Plugins Alerting
, ,
1

Versions :
Opensearch + Dashboards 2.0.1, Chrome, Docker on Ubuntu 20.04

Describe the issue:
I am trying to get the Notification channel working for an https webhook endpoint - however i get the following error:

[status_exception] {“event_status_list”: [{“config_id”:“some-notification-channel”,“config_type”:“webhook”,“config_name”:“notification-channel”,“email_recipient_status”:,“delivery_status”:{“status_code”:“500”,“status_text”:“Failed to send webhook message PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”}}]}

Which suggests to me that the server does not trust the same CA as the webhook api servers certificate on the other end.

Configuration:
I have added the CA to the trust on:

  • The docker host server
  • /usr/share/opensearch/config/root-ca.pem (Opensearch)
  • /usr/share/opensearch_dashboards/config/root-ca.pem (Opensearch Dashboards)

Opensearch config:

plugins.security.ssl.transport.truststore_filepath: truststore.jks
...
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
...
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem

So now i’m wondering - what CA does the Notification channel use? Why am i not able to send test messages using an unverified certificate? And why is there no obvious way to configure the CA for each channel?

2

I have verified using CURL that the root-ca.pem file does include trust for the webhook servers tls certificate.

curl -X POST https://webhookserver/api/endpoint --cacert ./root-ca.pem -vv

Which succeeds.

3

Checking the java truststore for trusted certificates with:

keytool -list -keystore config/truststore.jks

Gives me a list of my CA’s that were mapped in using the configs as explained above. The fingerprints match the working CA, so you’d think Java knows the CA, but it still does not connect.

4

SOLVED

Solved the issue after a good deal of trial and error.

The notification channels outgoing httpclient uses the Java Runtime trust store, and NOT the opensearch truststore for verifying connections.

This is important, as it means you will have to trust your CA with Javas cacerts truststore as well.

This can be done by:

  • Access your container as root, in order to have sufficient privileges: docker exec -u root -it opensearch_container_name bash

  • Read your existing truststore and verify if you already have trust with the CA you’re expecting: keytool -list -keystore "$JAVA_HOME/lib/security/cacerts"

  • Ensure that you have mapped in the CA you wish to add to your list of trusted CA’s.

  • Mapping in the new CA you wish to trust (i use the FQDN as alias here, but you do what makes sense. It has to be unique): keytool -import -noprompt -trustcacerts -alias some_alias -file /usr/share/opensearch/some-new-ca.crt -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit Here you must use the password for the truststore. Normally it is “changeit”.

  • Finally, verify that the CA has been added: keytool -keystore "$JAVA_HOME/lib/security/cacerts" -storepass changeit -list | grep some_alias

  • Your mileage might vary, and maybe there is a better way, but for me it required a restart of the container to take.

3 Likes