# Acknowledge alert times out

**URL:** <https://forum.opensearch.org/t/acknowledge-alert-times-out/18952>\
**Category:** OpenSearch\
**Tags:** troubleshoot, alerting, security-issue\
**Created:** [April 19, 2024, 10:07pm UTC](https://forum.opensearch.org/t/acknowledge-alert-times-out/18952 "2024-04-19T22:07:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![CarolynR](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@CarolynR](https://forum.opensearch.org/u/CarolynR)\
**Post date:** [April 19, 2024, 10:07pm UTC](https://forum.opensearch.org/t/acknowledge-alert-times-out/18952/1 "2024-04-19T22:07:45Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):  
Opensearch 2.11.1 on RHEL 9 any browser

**Describe the issue** :  
I am running Opensearch inside a docker container with security turned on through the security plugin.  
I have created a monitor which has triggers an alert.

When I select the alert and click Acknowledge, nothing happens for a while. Eventually, I see a message that says the “Failed to acknowledge the alert Request Timeout after 30000ms”.

I have pulled the container log and see:  
{“type”: “server”, “timestamp”: “2024-04-19T22:01:31,180Z”, “level”: “ERROR”, “component”: “o.o.b.OpenSearchUncaughtExceptionHandler”, “cluster.name”: “docker-cluster”, “node.name”: “datastore”, “message”: “uncaught exception in thread [DefaultDispatcher-worker-4]”, “cluster.uuid”: “2WjR\_3\_cQGq4fa4jC-sSjg”, “node.id”: “jHwAj9aHSTK9eyuH4jOLrQ” ,  
“stacktrace”: [“org.opensearch.alerting.util.AlertingException: Filter by user backend roles is enabled with security disabled.”,  
“at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[?:?]”,  
“at org.opensearch.alerting.transport.SecureTransportAction$DefaultImpls.validateUserBackendRoles(SecureTransportAction.kt:82) ~[?:?]”,  
“at org.opensearch.alerting.transport.TransportGetMonitorAction.validateUserBackendRoles(TransportGetMonitorAction.kt:50) ~[?:?]”,  
“at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:79) ~[?:?]”,  
“at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:50) ~[?:?]”,  
“at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-2.11.1.jar:2.11.1]”,  
“at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) ~[?:?]”,  
“at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.11.1.jar:2.11.1]”,  
“at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) ~[?:?]”,  
“at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.11.1.jar:2.11.1]”,  
“at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:324) ~[?:?]”,  
“at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) ~[?:?]”,  
“at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.11.1.jar:2.11.1]”,  
“at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.11.1.jar:2.11.1]”,  
“at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.11.1.jar:2.11.1]”,  
“at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.11.1.jar:2.11.1]”,  
“at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.11.1.jar:2.11.1]”,  
“at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) ~[opensearch-2.11.1.jar:2.11.1]”,  
“at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1$getMonitorResponse$1.invoke(TransportAcknowledgeAlertAction.kt:99) ~[opensearch-alerting-2.11.1.0-SNAPSHOT.jar:2.11.1.0-SNAPSHOT]”,  
“at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1$getMonitorResponse$1.invoke(TransportAcknowledgeAlertAction.kt:92) ~[opensearch-alerting-2.11.1.0-SNAPSHOT.jar:2.11.1.0-SNAPSHOT]”,  
“at org.opensearch.alerting.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:152) ~[alerting-core-2.11.1.0-SNAPSHOT.jar:?]”,  
“at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1.invokeSuspend(TransportAcknowledgeAlertAction.kt:92) ~[opensearch-alerting-2.11.1.0-SNAPSHOT.jar:2.11.1.0-SNAPSHOT]”,  
“at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]”,  
“at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]”,  
“at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]”,  
“at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]”,  
“at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]”,  
“Caused by: java.lang.Exception: org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled.”,  
“… 27 more”] }  
uncaught exception in thread [DefaultDispatcher-worker-4]  
AlertingException[Filter by user backend roles is enabled with security disabled.]; nested: Exception[org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled.];  
at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70)  
at org.opensearch.alerting.transport.SecureTransportAction$DefaultImpls.validateUserBackendRoles(SecureTransportAction.kt:82)  
at org.opensearch.alerting.transport.TransportGetMonitorAction.validateUserBackendRoles(TransportGetMonitorAction.kt:50)  
at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:79)  
at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:50)  
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)  
at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39)  
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)  
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118)  
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)  
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:324)  
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165)  
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)  
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)  
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)  
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)  
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)  
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476)  
at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1$getMonitorResponse$1.invoke(TransportAcknowledgeAlertAction.kt:99)  
at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1$getMonitorResponse$1.invoke(TransportAcknowledgeAlertAction.kt:92)  
at org.opensearch.alerting.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:152)  
at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1.invokeSuspend(TransportAcknowledgeAlertAction.kt:92)  
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)  
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)  
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)  
at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)  
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742)  
Caused by: java.lang.Exception: org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled.  
… 27 more

I don’t have a problem with any of the other alerting functions.

After putting some debug statements in the log, it appears that the userid that I am logged in with get forgotten.

Is there a code fix in newer code for this? Is there some configuration setting that I am missing?

**Configuration** :

**Relevant Logs or Screenshots** :

---

<div class="post-metadata">

**Author:** ![CarolynR](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@CarolynR](https://forum.opensearch.org/u/CarolynR)\
**Post date:** [April 20, 2024, 2:10pm UTC](https://forum.opensearch.org/t/acknowledge-alert-times-out/18952/2 "2024-04-20T14:10:02Z")

</div>

I believe this problem is caused by what is documented in

> <https://github.com/opensearch-project/security/issues/2487>
>
> \*\*Is your feature request related to a problem?\*\*
> 
> The usage of ThreadContext.…stashContext() is widely used in OpenSearch plugins to stash the current thread context and assume a "local direct" mode which escalates the plugin's privileges to perform any action on an OpenSearch cluster. Plugin developers often wrap code paths that interact with system indices to skip privilege evaluation and write documents into the plugin's own indices. When stashing the context, information that was previously populated into the threadcontext from the security plugin is removed and information about the original caller is unavailable within the new context. 
> 
> Stashing the ThreadContext should be deprecated and a new way of evaluating actions that originate from plugins developed. Plugins should only be able to interact with their own system indices.
> 
> \-----
> 
> Below is a detailed trace of the code path to understand how stashing the context makes it so that plugins can assume a "local direct" mode to skip privilege evaluation. 
> 
> From my understanding of how stashContext works in a block like this (\[Example in AD\](https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/transport/AnomalyDetectorJobTransportAction.java#L96-L121)):
> 
> \`\`\`
> try (ThreadContext.StoredContext context = client.threadPool().getThreadContext().stashContext()) {
> resolveUserAndExecute(user, detectorId, method, listener, (detector) -\> adExecute(request, user, detector, context, listener));
> } catch (Exception e) {
> LOG.error(e);
> listener.onFailure(e);
> }
> \`\`\`
> 
> is that \`resolveUserAndExecute\` in this context will run with no threadcontext headers unless you explicitly take values out before stashing and add them back in within this block.
> 
> When a REST request hits a cluster with the security plugin installed, the security plugin wraps the REST handler to authenticate the request and upon successful authentication the security plugin will populate the threadcontext headers like this:
> 
> \`\`\`
> ThreadContext
> 
> transientHeaders:
> 
> \_opendistro\_security\_user: \<User obj\>
> 
> headers:
> 
> \_opendistro\_security\_origin: REST
> \`\`\`
> 
> then after the request is authenticated it goes from the REST-layer -\> Transport-layer at which point the action filters are applied. The security plugin performs authorization at this level and within the SecurityFilter it populates another ThreadContext header with the user object serialized for plugins to read from with common-utils. That looks like:
> 
> \`\`\`
> ThreadContext
> 
> transientHeaders:
> 
> \_opendistro\_security\_user: \<User obj\>
> 
> headers:
> 
> \_opendistro\_security\_origin: REST
> \_opendistro\_security\_user\_info: username|backend\_role1,backend\_role2 // This is a pipe-delimited serialized representation of a User
> 
> \`\`\`
> 
> There are a few other headers the security plugin will use, but these are 3 of the major ones. When a plugin registers a new REST handler with a corresponding REST Action and Transport Action, these headers will already be populated by the security plugin when you are inside the transport action for the plugin. When plugin's need to bypass privilege evaluation they can stash the context and operate without these headers in place.
> 
> When that happens, the SecurityFilter is still applied before the transport action is executed but the SecurityFilter at that point is pass-through because the action is implicitly trusted as its considered a "local direct" request:
> 
> \- Local means the request is not originated from the REST-layer: This is populated by the SecurityFilter itself if the \_opendistro\_security\_origin header is null: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L182-L184
> \- Direct means that \_opendistro\_security\_channel\_type is null: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/support/HeaderHelper.java#L42-L46
> \- Altogether that means it will hit this block which is located before the privilege evaluation so the plugin is operating in effect as a superuser: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L315-L322
> 
> \*\*What solution would you like?\*\*
> 
> Actions originating from plugins should be identified and authorized accordingly. Reliance on stashing the context should be deprecated and eventually removed/forbidden.

> <https://github.com/opensearch-project/security/issues/2909>
>
> When plugins need to run actions outside of the currently authenticated user's c…ontext, they will call on \`ThreadContext.stashContext\` to assume a fresh context where they can run transport actions in a trusted manner. There are a few issues in the Security backlog that talk about this process in detail:
> 
> \- https://github.com/opensearch-project/security/issues/2487
> \- https://github.com/opensearch-project/security/issues/2860
> 
> An example block looks like:
> 
> \`\`\`
> try (ThreadContext.StoredContext context = client.threadPool().getThreadContext().stashContext()) {
> // Run code in this block with fresh context which allows a plugin to enter a trusted mode
> // PrivilegeEvaluation is skipped in this block as the SecurityFilter exits early here: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L315-L322
> } catch (Exception e) {
> LOG.error(e);
> listener.onFailure(e);
> }
> \`\`\`
> 
> As part of \[Centralized Scheduled Job Identity Management\](https://github.com/opensearch-project/security/issues/2905) the security plugin would need access to the currently authenticated user even if the thread context has been stashed. There needs to be a mechanism (SecurityContext or something similar) where there is a carrier of Security information that is available even if the ThreadContext has been stashed. For Centralized Scheduled Job Identity Management, that would mean that Job Scheduler can listen in on index operations into plugins jobs indices and call on the security plugin to associate the newly created job to the currently authenticated user automatically. When the threadcontext is stashed, the security plugin currently has no idea who the currently authenticated user is. As a result, plugin developers have resorted to using \_roles injection\_ where they will read the user from the threadcontext before stashing and save a copy of the user with the job. At job execution time they inject the roles back into the threadcontext. Ideally, plugin developers do not need to do that work as it is the responsibility of the identity/security system.

Is there any sample code that I can look at that addresses this or a direct fix for the alert acknowledge problem?

---

<div class="post-metadata">

**Author:** ![hm21](https://avatars.discourse-cdn.com/v4/letter/h/8dc957/32.png) [@hm21](https://forum.opensearch.org/u/hm21)\
**Post date:** [April 21, 2024, 11:35am UTC](https://forum.opensearch.org/t/acknowledge-alert-times-out/18952/3 "2024-04-21T11:35:24Z")

</div>

you may try version 2.13.0  
it may have been fixed already

---

<div class="post-metadata">

**Author:** ![mark.boyd](https://avatars.discourse-cdn.com/v4/letter/m/35a633/32.png) [@mark.boyd](https://forum.opensearch.org/u/mark.boyd)\
**Post date:** [June 24, 2026, 4:29pm UTC](https://forum.opensearch.org/t/acknowledge-alert-times-out/18952/4 "2026-06-24T16:29:12Z")

</div>

Hey @CarolynR - I ran into the same issue with acknowledging alerts timing out when filtering by user backend roles is enabled. I believe that this issue will be resolved by my PR: [Add filter by backend roles setting by markdboyd · Pull Request #2034 · opensearch-project/alerting · GitHub](https://github.com/opensearch-project/alerting/pull/2034)