Is mTLS possible for webhook alerts?

Plugins Alerting
1

Versions :Latest

Describe the issue: I couldn’t find any documentation or discussions around this mTLS support opensearch alerts? Is it supported and someone point me in the right direction. I’m adding some configs and logs that i got while experimenting with this.

Configuration:
I tried setting up these configs.
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.keystore_type: PKCS12
plugins.security.ssl.transport.keystore_filepath: certs/server-keystore.p12
plugins.security.ssl.transport.keystore_password: changeit
plugins.security.ssl.transport.truststore_type: PKCS12
plugins.security.ssl.transport.truststore_filepath: certs/server-truststore.p12
plugins.security.ssl.transport.truststore_password: changeit
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.ssl.transport.resolve_hostname: trueExceptions:
Relevant Logs or Screenshots:

\[2025-08-14T11:31:54,063\]\[WARN \]\[o.o.n.a.PluginBaseAction \] \[opensearch-node1\] notifications:OpenSearchStatusException:
org.opensearch.OpenSearchStatusException: {"event_status_list": \[{"config_id":"UmdEqJgBbZmp5cjtkRza","config_type":"webhook","config_name":"secure-webhook-proper-mtls","email_recipient_status":\[\],"delivery_status":{"status_code":"500","status_text":"Failed to send webhook message Failed: {\\"status\\":\\"error\\",\\"message\\":\\"Valid client certificate required for mTLS authentication\\",\\"ip\\":\\"127.0.0.1\\"}"}}\]}
        at org.opensearch.notifications.send.SendMessageActionHelper.executeRequest(SendMessageActionHelper.kt:101) \~\[?:?\]
        at org.opensearch.notifications.send.SendMessageActionHelper$executeRequest$1.invokeSuspend(SendMessageActionHelper.kt) \~\[?:?\]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) \[kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)\]
        at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32) \[kotlinx-coroutines-core-jvm-1.4.3.jar:?\]
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113) \[kotlinx-coroutines-core-jvm-1.4.3.jar:?\]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) \[kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)\]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) \[kotlinx-coroutines-core-jvm-1.4.3.jar:?\]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) \[kotlinx-coroutines-core-jvm-1.4.3.jar:?\]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) \[kotlinx-coroutines-core-jvm-1.4.3.jar:?\]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) \[kotlinx-coroutines-core-jvm-1.4.3.jar:?\]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) \[kotlinx-coroutines-core-jvm-1.4.3.jar:?\]
2

@abinabu Could you share your full alert definition?

3

Thanks @pablo for the response.

I have localhost web-server which expects mtls.
If the opensearch supports mtls for webhooks could you guide to some right documention.
Also, let me know if i’m doing something wrong with this configs.

Here is the webhook alert configuration.

{
  "config": {
    "name": "secure-webhook-proper-mtls",
    "description": "webhook destination",
    "config_type": "webhook",
    "is_enabled": true,
    "webhook": {
      "url": "https://host.docker.internal:8443/webhook",
      "method": "POST",
      "header_params": {
        "Content-Type": "application/json",
        "User-Agent": "OpenSearch-Alerting-mTLS"
      },
      "host_deny_list": []
    }
  }
}

Monitor config:

{
  "type": "monitor",
  "name": "OpenSearch mTLS Webhook Monitor",
  "monitor_type": "query_level_monitor", 
  "enabled": true,
  "schedule": {
    "period": {
      "interval": 1,
      "unit": "MINUTES"
    }
  },
  "inputs": [
    {
      "search": {
        "indices": ["test-index"],
        "query": {
          "size": 100,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "from": "now-2m",
                      "to": null,
                      "include_lower": true,
                      "include_upper": true,
                      "boost": 1.0
                    }
                  }
                }
              ],
              "adjust_pure_negative": true,
              "boost": 1.0
            }
          },
          "sort": [
            {
              "@timestamp": {
                "order": "desc"
              }
            }
          ]
        }
      }
    }
  ],
  "triggers": [
    {
      "query_level_trigger": {
        "id": "webhook_trigger",
        "name": "mTLS Webhook Alert Trigger",
        "severity": "1",
        "condition": {
          "script": {
            "source": "ctx.results[0].hits.total.value > 0",
            "lang": "painless"
          }
        },
        "actions": [
          {
            "id": "webhook_action",
            "name": "Send mTLS Webhook Alert",
            "destination_id": "6OfYrJgBgtpb7qJukHvs",
            "message_template": {
              "source": "mTLS Monitor {{ctx.monitor.name}} triggered. Found {{ctx.results.0.hits.total.value}} new documents in the last 2 minutes. This alert was sent via secure mTLS connection!",
              "lang": "mustache"
            },
            "throttle_enabled": false,
            "subject_template": {
              "source": "OpenSearch mTLS Alert: {{ctx.monitor.name}}",
              "lang": "mustache"
            }
          }
        ]
      }
    }
  ]
}
4

@abinabu Thank you for clarification. Unfortunately the current webhook client in the OpenSearch doesn’t support mTLS authentication.

You could consider using Nginx between OS and the Webhook server.

5

Thank you @pablo for the update.
Are you suggesting to have kind of ngnix proxy between the opensearch and the mTLS webserver?

Opensearch alerts → ngnix proxy → mTLS webserver

But this still means the proxy will be unsecure, even if it has basic auth, those credentials will be store as plain text and is visible to anyone who has the access?

And thanks again for your support!!

6

@abinabu Nginx is just a suggestion. You can use anything else that can terminate SSL from OpenSearch and create a new mTLS connection to your Webhook server.

You don’t have to expose the man in the middle (i.e. Nginx) to the world, but only to the OpenSearch cluster and only specific IP addresses.

1 Like