Custom webhook from OpenSearch Dashboard to AlertManager

LHozzan · July 3, 2023, 2:25pm

Versions: OpenSearch 2.8.0

Describe the issue:
I attempting to sent custom notification from Alerting plugin to the AlertManager in same Kubernetes cluster. AlertManager have his endpoint secured with HTTPS and custom CA certificate.

If I attempting to sent test message to the AlertManager, in OSDash I see error:

Error is also logged:

main {"type":"error","@timestamp":"2023-07-03T12:52:46Z","tags":[],"pid":456,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:143:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:97:19)\n    at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:92:17)\n    at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:164:34)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http://opensearch.localcluster.internal/api/notifications/test_message/0gOdC4kBtwMlZllVSTKb","message":"Internal Server Error"}

Error is persistent even if I mount the CA certificate to the OSDash pod and load it via opensearch.ssl.certificateAuthorities parameter.

BUT

In same time in one OpenSearch pod with role client I see the cause error:

{"type": "logging", "timestamp": "2023-07-03T12:52:47,087Z", "level": "WARN", "component": "o.o.n.a.PluginBaseAction", "cluster.name": "oststclstr", "node.name": "opensearch-client-87b7645dc-v2pgk", "message": "notifications:OpenSearchStatusException:", "cluster.uuid": "UkQ3GL0lTWmJVlCDFNgRdg", "node.id": "fXO8tzs0S7G3nwZJI_iIFw" , 
"stacktrace": ["org.opensearch.OpenSearchStatusException: {\"event_status_list\": [{\"config_id\":\"0gOdC4kBtwMlZllVSTKb\",\"config_type\":\"webhook\",\"config_name\":\"AlertMananagerWebhook\",\"email_recipient_status\":[],\"delivery_status\":{\"status_code\":\"500\",\"status_text\":\"Failed to send webhook message PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\"}}]}",
"at org.opensearch.notifications.send.SendMessageActionHelper.executeRequest(SendMessageActionHelper.kt:99) ~[?:?]",
"at org.opensearch.notifications.send.SendMessageActionHelper$executeRequest$1.invokeSuspend(SendMessageActionHelper.kt) ~[?:?]",
"at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]",
"at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]",
"at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]"] }
{"type": "logging", "timestamp": "2023-07-03T12:52:47,089Z", "level": "ERROR", "component": "o.o.n.a.SendTestNotificationAction", "cluster.name": "oststclstr", "node.name": "opensearch-client-87b7645dc-v2pgk", "message": "notifications:SendTestNotificationAction-send Error:OpenSearchStatusException[{\"event_status_list\": [{\"config_id\":\"0gOdC4kBtwMlZllVSTKb\",\"config_type\":\"webhook\",\"config_name\":\"AlertMananagerWebhook\",\"email_recipient_status\":[],\"delivery_status\":{\"status_code\":\"500\",\"status_text\":\"Failed to send webhook message PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\"}}]}]", "cluster.uuid": "UkQ3GL0lTWmJVlCDFNgRdg", "node.id": "fXO8tzs0S7G3nwZJI_iIFw"  }
{"type": "logging", "timestamp": "2023-07-03T12:52:47,090Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "oststclstr", "node.name": "opensearch-client-87b7645dc-v2pgk", "message": "path: /_plugins/_notifications/feature/test/0gOdC4kBtwMlZllVSTKb, params: {config_id=0gOdC4kBtwMlZllVSTKb}", "cluster.uuid": "UkQ3GL0lTWmJVlCDFNgRdg", "node.id": "fXO8tzs0S7G3nwZJI_iIFw" , 
"stacktrace": ["org.opensearch.OpenSearchStatusException: {\"event_status_list\": [{\"config_id\":\"0gOdC4kBtwMlZllVSTKb\",\"config_type\":\"webhook\",\"config_name\":\"AlertMananagerWebhook\",\"email_recipient_status\":[],\"delivery_status\":{\"status_code\":\"500\",\"status_text\":\"Failed to send webhook message PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\"}}]}",
"at org.opensearch.notifications.send.SendMessageActionHelper.executeRequest(SendMessageActionHelper.kt:99) ~[?:?]",
"at org.opensearch.notifications.send.SendMessageActionHelper$executeRequest$1.invokeSuspend(SendMessageActionHelper.kt) ~[?:?]",
"at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]",
"at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]",
"at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]",
"at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]"] }

And here starting my confusion: Who communicate with AlertManager? OpenSearch Dashboard or OpenSearch Node? Who must know about custom CA cert? It seems, that OpenSearch Node.

Unfortunately, I not find any usable information in the documentation.

If I attempt to call same webhook (https://alertmanager.prometheus.svc.cluster.local:9093/api/v1/alerts) via CURL, everything is working. Yes, I have the CA certificate available for CURL. So, problem is really in the trust between OpenSearch and AlertManager.

I would like to ask you to help me with the problem. My goal is have notification from OSDash Alerts shown in the AlertManager.

Any suggestions or tips will be valued as well.

Thank you!

pablo · July 3, 2023, 6:30pm

@LHozzan I understand you’ve created a Monitor in OpenSearch Alerting with AlertManager webhook as destination.

The OpenSearch node executes the Monitor and OpenSearch will also handle the connection to the target defined in the Monitor (Channels).

I assume that OpenSearch doesn’t have AlertManager’s self-signed certificate in its keystore.
I’m not aware of any certificate settings for Alerting plugin. However, please check this link.

github.com/opensearch-project/alerting

Allowing additional CA certificates in custom webhook destinations

opened 09:41PM - 02 Jun 21 UTC

closed 04:18PM - 06 Apr 22 UTC

aditjind

enhancement

<a href="https://github.com/srilumpa"><img src="https://avatars.githubuserconten…t.com/u/3243772?v=4" align="left" width="96" height="96" hspace="10"></img></a> **Issue by [srilumpa](https://github.com/srilumpa)** _Thursday Jun 18, 2020 at 12:58 GMT_ _Originally opened as https://github.com/opendistro-for-elasticsearch/alerting/issues/216_ ---- **Is your feature request related to a problem? Please describe.** I am trying to set up alerting destination to a custom webhook using HTTPS. Unfortunately, the destination is configured with a certificate signed by a custom CA and I am not able to include this CA's certificate in the validation chain. The following error is log whenever the trigger quicks off: ``` [2020-06-18T10:28:40,326][ERROR][c.a.o.a.d.f.CustomWebhookDestinationFactory] [srv-ec4-03p.infra.dc2.intra] Exception publishing Message: DestinationType: CUSTOMWEBHOOK, DestinationName:tools-notify-uservice, Ur l: https://my.web.service/notify, scheme: https, Host: , Port: -1, Path: null, Message: {"source":{"monitor_name":"My monitor"},"attachments": [{"t itle": "0 alerts between 2020-06-18T07:28:38.842Z and 2020-06-18T08:28:38.842Z","fields": [{"title": "Hits","value": "0"}]}]} javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:?] [snip] ``` **Describe the solution you'd like** Have a way to include one or several certificate in OpenDistro's truststore or equivalent. **Describe alternatives you've considered** I included the CA's certificate in the Java default truststore but with no changes in the error.

LHozzan · July 11, 2023, 2:39pm

@pablo
Thank you. I try to use it.
Best regards.

Topic		Replies	Views
Unable to add custom webhook to opensearch alert OpenSearch Dashboards troubleshoot	3	339	May 2, 2024
Issue using custom webhook to write to index in OpenSearch Alerting troubleshoot	2	1390	August 1, 2023
Curl to dashboard pod gives {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."} with centos image OpenSearch Dashboards install	7	2246	March 28, 2023
Sending data from opensearch alerts to API OpenSearch troubleshoot	1	281	December 20, 2023
Error when edit notification.core.http.host_deny_list configuration OpenSearch alerting	2	89	September 11, 2024

Custom webhook from OpenSearch Dashboard to AlertManager

Related topics