Acknowledge alert times out

CarolynR · April 19, 2024, 10:07pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.11.1 on RHEL 9 any browser

Describe the issue:
I am running Opensearch inside a docker container with security turned on through the security plugin.
I have created a monitor which has triggers an alert.

When I select the alert and click Acknowledge, nothing happens for a while. Eventually, I see a message that says the “Failed to acknowledge the alert Request Timeout after 30000ms”.

I have pulled the container log and see:
{“type”: “server”, “timestamp”: “2024-04-19T22:01:31,180Z”, “level”: “ERROR”, “component”: “o.o.b.OpenSearchUncaughtExceptionHandler”, “cluster.name”: “docker-cluster”, “node.name”: “datastore”, “message”: “uncaught exception in thread [DefaultDispatcher-worker-4]”, “cluster.uuid”: “2WjR_3_cQGq4fa4jC-sSjg”, “node.id”: “jHwAj9aHSTK9eyuH4jOLrQ” ,
“stacktrace”: [“org.opensearch.alerting.util.AlertingException: Filter by user backend roles is enabled with security disabled.”,
“at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[?:?]”,
“at org.opensearch.alerting.transport.SecureTransportAction$DefaultImpls.validateUserBackendRoles(SecureTransportAction.kt:82) ~[?:?]”,
“at org.opensearch.alerting.transport.TransportGetMonitorAction.validateUserBackendRoles(TransportGetMonitorAction.kt:50) ~[?:?]”,
“at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:79) ~[?:?]”,
“at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:50) ~[?:?]”,
“at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-2.11.1.jar:2.11.1]”,
“at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) ~[?:?]”,
“at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.11.1.jar:2.11.1]”,
“at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) ~[?:?]”,
“at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.11.1.jar:2.11.1]”,
“at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:324) ~[?:?]”,
“at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) ~[?:?]”,
“at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.11.1.jar:2.11.1]”,
“at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.11.1.jar:2.11.1]”,
“at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.11.1.jar:2.11.1]”,
“at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.11.1.jar:2.11.1]”,
“at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.11.1.jar:2.11.1]”,
“at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) ~[opensearch-2.11.1.jar:2.11.1]”,
“at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1$getMonitorResponse$1.invoke(TransportAcknowledgeAlertAction.kt:99) ~[opensearch-alerting-2.11.1.0-SNAPSHOT.jar:2.11.1.0-SNAPSHOT]”,
“at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1$getMonitorResponse$1.invoke(TransportAcknowledgeAlertAction.kt:92) ~[opensearch-alerting-2.11.1.0-SNAPSHOT.jar:2.11.1.0-SNAPSHOT]”,
“at org.opensearch.alerting.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:152) ~[alerting-core-2.11.1.0-SNAPSHOT.jar:?]”,
“at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1.invokeSuspend(TransportAcknowledgeAlertAction.kt:92) ~[opensearch-alerting-2.11.1.0-SNAPSHOT.jar:2.11.1.0-SNAPSHOT]”,
“at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]”,
“at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]”,
“at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]”,
“at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]”,
“at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]”,
“Caused by: java.lang.Exception: org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled.”,
“… 27 more”] }
uncaught exception in thread [DefaultDispatcher-worker-4]
AlertingException[Filter by user backend roles is enabled with security disabled.]; nested: Exception[org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled.];
at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70)
at org.opensearch.alerting.transport.SecureTransportAction$DefaultImpls.validateUserBackendRoles(SecureTransportAction.kt:82)
at org.opensearch.alerting.transport.TransportGetMonitorAction.validateUserBackendRoles(TransportGetMonitorAction.kt:50)
at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:79)
at org.opensearch.alerting.transport.TransportGetMonitorAction.doExecute(TransportGetMonitorAction.kt:50)
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39)
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118)
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:324)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165)
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476)
at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1$getMonitorResponse$1.invoke(TransportAcknowledgeAlertAction.kt:99)
at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1$getMonitorResponse$1.invoke(TransportAcknowledgeAlertAction.kt:92)
at org.opensearch.alerting.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:152)
at org.opensearch.alerting.transport.TransportAcknowledgeAlertAction$doExecute$1$1.invokeSuspend(TransportAcknowledgeAlertAction.kt:92)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742)
Caused by: java.lang.Exception: org.opensearch.OpenSearchStatusException: Filter by user backend roles is enabled with security disabled.
… 27 more

I don’t have a problem with any of the other alerting functions.

After putting some debug statements in the log, it appears that the userid that I am logged in with get forgotten.

Is there a code fix in newer code for this? Is there some configuration setting that I am missing?

Configuration:

Relevant Logs or Screenshots:

CarolynR · April 20, 2024, 2:10pm

I believe this problem is caused by what is documented in

github.com/opensearch-project/security

[FEATURE] Deprecate the usage of ThreadContext.stashContext in plugins

opened 05:25PM - 27 Feb 23 UTC

cwperks

enhancement triaged

**Is your feature request related to a problem?** The usage of ThreadContext.…stashContext() is widely used in OpenSearch plugins to stash the current thread context and assume a "local direct" mode which escalates the plugin's privileges to perform any action on an OpenSearch cluster. Plugin developers often wrap code paths that interact with system indices to skip privilege evaluation and write documents into the plugin's own indices. When stashing the context, information that was previously populated into the threadcontext from the security plugin is removed and information about the original caller is unavailable within the new context. Stashing the ThreadContext should be deprecated and a new way of evaluating actions that originate from plugins developed. Plugins should only be able to interact with their own system indices. ----- Below is a detailed trace of the code path to understand how stashing the context makes it so that plugins can assume a "local direct" mode to skip privilege evaluation. From my understanding of how stashContext works in a block like this ([Example in AD](https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/transport/AnomalyDetectorJobTransportAction.java#L96-L121)): ``` try (ThreadContext.StoredContext context = client.threadPool().getThreadContext().stashContext()) { resolveUserAndExecute(user, detectorId, method, listener, (detector) -> adExecute(request, user, detector, context, listener)); } catch (Exception e) { LOG.error(e); listener.onFailure(e); } ``` is that `resolveUserAndExecute` in this context will run with no threadcontext headers unless you explicitly take values out before stashing and add them back in within this block. When a REST request hits a cluster with the security plugin installed, the security plugin wraps the REST handler to authenticate the request and upon successful authentication the security plugin will populate the threadcontext headers like this: ``` ThreadContext transientHeaders: _opendistro_security_user: <User obj> headers: _opendistro_security_origin: REST ``` then after the request is authenticated it goes from the REST-layer -> Transport-layer at which point the action filters are applied. The security plugin performs authorization at this level and within the SecurityFilter it populates another ThreadContext header with the user object serialized for plugins to read from with common-utils. That looks like: ``` ThreadContext transientHeaders: _opendistro_security_user: <User obj> headers: _opendistro_security_origin: REST _opendistro_security_user_info: username|backend_role1,backend_role2 // This is a pipe-delimited serialized representation of a User ``` There are a few other headers the security plugin will use, but these are 3 of the major ones. When a plugin registers a new REST handler with a corresponding REST Action and Transport Action, these headers will already be populated by the security plugin when you are inside the transport action for the plugin. When plugin's need to bypass privilege evaluation they can stash the context and operate without these headers in place. When that happens, the SecurityFilter is still applied before the transport action is executed but the SecurityFilter at that point is pass-through because the action is implicitly trusted as its considered a "local direct" request: - Local means the request is not originated from the REST-layer: This is populated by the SecurityFilter itself if the _opendistro_security_origin header is null: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L182-L184 - Direct means that _opendistro_security_channel_type is null: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/support/HeaderHelper.java#L42-L46 - Altogether that means it will hit this block which is located before the privilege evaluation so the plugin is operating in effect as a superuser: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L315-L322 **What solution would you like?** Actions originating from plugins should be identified and authorized accordingly. Reliance on stashing the context should be deprecated and eventually removed/forbidden.

github.com/opensearch-project/security

Security Information is removed from the TheadContext when plugins call on ThreadContext.stashContext

opened 04:04PM - 27 Jun 23 UTC

cwperks

bug triaged

When plugins need to run actions outside of the currently authenticated user's c…ontext, they will call on `ThreadContext.stashContext` to assume a fresh context where they can run transport actions in a trusted manner. There are a few issues in the Security backlog that talk about this process in detail: - https://github.com/opensearch-project/security/issues/2487 - https://github.com/opensearch-project/security/issues/2860 An example block looks like: ``` try (ThreadContext.StoredContext context = client.threadPool().getThreadContext().stashContext()) { // Run code in this block with fresh context which allows a plugin to enter a trusted mode // PrivilegeEvaluation is skipped in this block as the SecurityFilter exits early here: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L315-L322 } catch (Exception e) { LOG.error(e); listener.onFailure(e); } ``` As part of [Centralized Scheduled Job Identity Management](https://github.com/opensearch-project/security/issues/2905) the security plugin would need access to the currently authenticated user even if the thread context has been stashed. There needs to be a mechanism (SecurityContext or something similar) where there is a carrier of Security information that is available even if the ThreadContext has been stashed. For Centralized Scheduled Job Identity Management, that would mean that Job Scheduler can listen in on index operations into plugins jobs indices and call on the security plugin to associate the newly created job to the currently authenticated user automatically. When the threadcontext is stashed, the security plugin currently has no idea who the currently authenticated user is. As a result, plugin developers have resorted to using _roles injection_ where they will read the user from the threadcontext before stashing and save a copy of the user with the job. At job execution time they inject the roles back into the threadcontext. Ideally, plugin developers do not need to do that work as it is the responsibility of the identity/security system.

Is there any sample code that I can look at that addresses this or a direct fix for the alert acknowledge problem?

hm21 · April 21, 2024, 11:35am

you may try version 2.13.0
it may have been fixed already

Topic		Replies	Views
I get a response timeout 30000 only with the alerting plugin Alerting	1	334	April 22, 2024
Opensearch alerting plugin is throwing exception Filter by user backend roles is enabled by security disabled Alerting	7	795	June 8, 2023
OpenSearch benchmark not working OpenSearch	3	361	April 25, 2024
SMTP sender Timeout while TLS handshake Alerting troubleshoot	12	1725	May 22, 2023
Changing settings via GUI/DevTools always gives RequestTimeout OpenSearch Dashboards troubleshoot	6	274	April 16, 2024

Acknowledge alert times out

Related topics