Versions (relevant - OpenSearch/Dashboard/Server OS/Browser)
OpenSearch 2.6
Describe the issue:
I recently upgraded OpenSearch from 1.2.4 to 2.6. I use the Alerting plugin to send email alerts via the new Notifications plugin. After the upgrade to 2.6 lots of Monitors started failing to trigger alerts and send emails. The error message is:
[2023-04-18T09:58:39,364][WARN ][o.o.n.a.PluginBaseAction ] [test-logdash] notifications:OpenSearchStatusException:
org.opensearch.OpenSearchStatusException: {"event_status_list": [{"config_id":"U6pT-n4Byiy6gsxIWQNx","config_type":"email","config_name":"email crazyadmin","email_recipient_status":[{"recipient":"emailaddress@email.com","delivery_status":{"status_code":"413","status_text":"Email size larger than 10000"}}],"delivery_status":{"status_code":"413","status_text":"Email size larger than 10000"}}]}
at org.opensearch.notifications.send.SendMessageActionHelper.executeRequest(SendMessageActionHelper.kt:99) ~[?:?]
at org.opensearch.notifications.send.SendMessageActionHelper$executeRequest$1.invokeSuspend(SendMessageActionHelper.kt) ~[?:?]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
“Email size larger than 10000” is the message that you can also see in the web interface trying to manually send an email. I never encountered this error in OpenSearch 1.2.4, maybe it’s something new to the Notifications plugin?
I’ve tried Googling for this error but can’t find anything out there. The error code “413” made me think it was an HTTP 413, which would indicate the internal API call was getting maxed out. Elasticsearch docs tell me the default size on that is 100 mb, which wouldn’t be getting hit by a 10000 “something” limit. I still tried increasing the API HTTP request size limit to 512 mb with “http.max_content_length”, but that didn’t fix it. The emails I’m trying to send wouldn’t be that large anyway, but they’re hitting some 10000 size limit.
The limits in search return count and CSV line export are also 10000, so I’m wondering if this arbitrary size limit has also been introduced into the Notifications plugin somewhere. Is there a way to configure that? 10000 characters (if it’s chars) is not a very high limit and is really hindering our OpenSearch use case. Thank you for any help anybody has.