Notifications API Create Channel 502 Error

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch v2.11

Describe the issue:
The Notifications API feature to create a notifications channel appears to not work in 2.11. I have the same issue in two clusters on 2.11. This feature is documented here: API - OpenSearch Documentation

The “create channel” API request times out with a HTTP 502 error:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>

I can upgrade to 2.12 if this is fixed in that version, but I didn’t see anything about this issue in the release notes.

2

@crazyadmin How did you execute your API? Could you share it?
I’ve just tested 2.11.0 and Dev Tools using an example API Channel from the OpenSearch Docs.

POST /_plugins/_notifications/configs/
{
  "config_id": "sample-id",
  "name": "sample-name",
  "config": {
    "name": "Sample Slack Channel",
    "description": "This is a Slack channel",
    "config_type": "slack",
    "is_enabled": true,
    "slack": {
      "url": "https://sample-slack-webhook"
    }
  }
}

response

{
  "config_id": "sample-id"
}

GET _plugins/_notifications/configs

{
  "start_index": 0,
  "total_hits": 1,
  "total_hit_relation": "eq",
  "config_list": [
    {
      "config_id": "sample-id",
      "last_updated_time_ms": 1708986627847,
      "created_time_ms": 1708986627847,
      "config": {
        "name": "Sample Slack Channel",
        "description": "This is a Slack channel",
        "config_type": "slack",
        "is_enabled": true,
        "slack": {
          "url": "https://sample-slack-webhook"
        }
      }
    }
  ]
}
3

I’ve tried using Dev Tools and through curl on both clusters, same issue. The example with a Slack channel from the documentation doesn’t work, either. Dev Tools and curl have both worked in the past, but I noticed it had broken shortly after upgrading to 2.11. I have a lot of other API calls scripted using curl, including other Notifications API calls, but this call doesn’t work.
The Dev Tools method is just a copy/paste from the documentation example. Here’s the curl command if you’re interested:

  curl -nk -X POST "https:/0.0.0.0:9200/_plugins/_notifications/configs" \
      -H 'osd-xsrf: true' \
      -H 'Content-Type: application/json' \
      --data @<(cat $JSONfilename)
4

@crazyadmin You’re missing authentication.

curl -nk -X POST "https:/0.0.0.0:9200/_plugins/_notifications/configs" \
      -u admin:admin \
      -H 'osd-xsrf: true' \
      -H 'Content-Type: application/json' \
      --data @<(cat $JSONfilename)
5

The ‘-n’ flag is handling authentication here without needing to specify the credentials in the command line. This method is used about a dozen times in the same script and works for the other API calls it does. Regardless, the Dev Tools way should still work.

6

Here’s the cluster log when trying to use the documented example in Dev Tools. This sequence repeats over and over until failure:

[2024-02-28T11:06:57,714][INFO ][o.o.n.i.ConfigIndexingActions] [test-logdash] notifications:NotificationConfig-create
[2024-02-28T11:07:27,713][WARN ][o.o.n.a.PluginBaseAction ] [test-logdash] notifications:IllegalStateException:
kotlinx.coroutines.TimeoutCancellationException: Timed out waiting for 60000 ms
        at kotlinx.coroutines.TimeoutKt.TimeoutCancellationException(Timeout.kt:186) ~[?:?]
        at kotlinx.coroutines.TimeoutCoroutine.run(Timeout.kt:156) ~[?:?]
        at kotlinx.coroutines.EventLoopImplBase$DelayedRunnableTask.run(EventLoop.common.kt:497) ~[?:?]
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274) ~[?:?]
        at kotlinx.coroutines.DefaultExecutor.run(DefaultExecutor.kt:69) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2024-02-28T11:07:27,714][WARN ][r.suppressed             ] [test-logdash] path: /_plugins/_notifications/configs/, params: {pretty=true}
org.opensearch.OpenSearchStatusException: Timed out waiting for 60000 ms
        at org.opensearch.notifications.action.PluginBaseAction$doExecute$1.invokeSuspend(PluginBaseAction.kt:97) [opensearch-notifications-2.11.1.0.jar:2.11.1.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
        at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
7

@crazyadmin I’m not getting any of these errors in my test lab. Is it 2.11.0 or 2.11.1?

How did you deploy your cluster? Is an upgrade or fresh installation?

8

2.11.1. It was an upgrade from v2.7 in both cases.