Unable to create monitor in opensearch 2.5

anubisg1 · February 1, 2023, 4:47pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

opensearch 2.5.0, dashboards 2.5.0, running in containers (deployed on kuberentes using the k8s operator

Describe the issue:

I’m trying to create a per-document monitor but i always hand up with errors of all kinds.
Notice that the same thing with opensearch 2.3.0 worked fine. (didn’t try 2.4)

example of errors

[2023-02-01T16:34:43,914][WARN ][r.suppressed             ] [opensearch-cluster-masters-0] path: /_plugins/_alerting/destinations, params: {startIndex=0, searchString=, size=200, destinationType=ALL}
org.opensearch.alerting.util.AlertingException: all shards failed
	at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
	at org.opensearch.alerting.transport.TransportGetDestinationsAction$search$1.onFailure(TransportGetDestinationsAction.kt:162) [opensearch-alerting-2.5.0.0.jar:2.5.0.0]
	at org.opensearch.action.support.TransportAction$1.onFailure(TransportAction.java:122) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.action.ActionListener$5.onFailure(ActionListener.java:275) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:689) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:663) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:698) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:471) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:294) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:104) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:74) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:753) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.TransportService$6.handleException(TransportService.java:794) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:312) [opensearch-security-2.5.0.0.jar:2.5.0.0]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1414) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:420) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.InboundHandler.handleException(InboundHandler.java:418) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:410) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:158) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) [transport-netty4-client-2.5.0.jar:2.5.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) [netty-handler-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) [netty-handler-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) [netty-handler-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.86.Final.jar:4.1.86.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.Exception: org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
	... 55 more

here are my indexes

green open security-auditlog-2023.02.01           3pQELCdIR62zQZlRqDeEug 1 1     457  0   3.1mb   1.6mb
green open .opendistro-job-scheduler-lock         Z-9_SoTnTXaft7bvqDqmjQ 1 1       4 52 207.4kb 111.7kb
green open syslog-2023.02.01                      b20IFasaSMmjESC5mS8H6A 1 1   12696  0   8.5mb   4.1mb
green open .opendistro_security                   TL2M7ZdyQMS5aVOra6f8Sg 1 2      10 81 600.4kb   241kb
green open .kibana_1                              CJRdpYqVSAu5W0FxuIkN5w 1 1    1322  2 874.5kb 437.2kb
green open .opensearch-notifications-config       0vXqp8lIRSG9uzafIVfKvg 1 2       1  0  52.7kb  22.1kb
green open .opendistro-alerting-config            hyYjthB-SkauGe-78DKk4w 1 1       1  0  10.4kb   5.2kb
green open elastiflow-flow-ecs-8.0-2.1-2023.02.01 rEA_8YdUQ5m_-SebP0rlIQ 1 1  949499  0 711.3mb 370.2mb
green open filebeat-7.12.1-2023.02.01             tZaL228wRr2IyHAby0SJ6A 1 1 9639447  0   9.4gb   4.6gb

anubisg1 · February 1, 2023, 5:06pm

other errors i get from opensearch nodes… of course i’m logged in as admin

[2023-02-01T17:03:44,313][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-masters-0] uncaught exception in thread [DefaultDispatcher-worker-3]
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/create]
Caused by: org.opensearch.ResourceAlreadyExistsException: index [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] already exists
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:214) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1184) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:382) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:437) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:343) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:867) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.5.0.jar:2.5.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
uncaught exception in thread [DefaultDispatcher-worker-3]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
RemoteTransportException[[opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/create]]; nested: ResourceAlreadyExistsException[index [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] already exists];
Caused by: [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] ResourceAlreadyExistsException[index [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] already exists]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:214)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1184)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:382)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:437)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:343)
	at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
	at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:867)
	at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424)
	at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
	at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
	at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190)
	at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(Thread.java:833)
[2023-02-01T17:03:44,364][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-cluster-masters-0] Detected cluster change event for destination migration
[2023-02-01T17:03:44,385][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-masters-0] PluginService:onIndexModule index:[.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg]
[2023-02-01T17:03:44,408][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-cluster-masters-0] Detected cluster change event for destination migration
[2023-02-01T17:03:44,462][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-cluster-masters-0] Detected cluster change event for destination migration
[2023-02-01T17:03:44,572][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-cluster-masters-0] Detected cluster change event for destination migration
uncaught exception in thread [DefaultDispatcher-worker-3]
[2023-02-01T17:03:44,630][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-masters-0] uncaught exception in thread [DefaultDispatcher-worker-3]
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/create]
Caused by: org.opensearch.ResourceAlreadyExistsException: index [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] already exists
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:214) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1184) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:382) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:437) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:343) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:867) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.5.0.jar:2.5.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
RemoteTransportException[[opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/create]]; nested: ResourceAlreadyExistsException[index [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] already exists];
Caused by: [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] ResourceAlreadyExistsException[index [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] already exists]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:214)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1184)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:382)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:437)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:343)
	at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
	at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:867)
	at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424)
	at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
	at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
	at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190)
	at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(Thread.java:833)
[2023-02-01T17:03:44,639][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-masters-0] uncaught exception in thread [DefaultDispatcher-worker-3]
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/create]
Caused by: org.opensearch.ResourceAlreadyExistsException: index [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] already exists
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:214) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1184) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:382) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:437) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:343) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:867) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190) ~[opensearch-2.5.0.jar:2.5.0]
uncaught exception in thread [DefaultDispatcher-worker-3]
	at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.5.0.jar:2.5.0]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.5.0.jar:2.5.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
RemoteTransportException[[opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/create]]; nested: ResourceAlreadyExistsException[index [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] already exists];
Caused by: [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] ResourceAlreadyExistsException[index [.opensearch-alerting-queries-000001/BK2aCNFORiCeF7Vfp_poKg] already exists]
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:214)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1184)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:382)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:437)
	at org.opensearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:343)
	at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
	at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:867)
	at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424)
	at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
	at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
	at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190)
	at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(Thread.java:833)
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

and from the dashboard

{"type":"response","@timestamp":"2023-02-01T17:04:10Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/api/alerting/monitors/_search","method":"post","headers":{"host":"opensearch-dev.cust.com","x-request-id":"7ada88430c81e266c26c15ca45bd38da","x-real-ip":"172.19.136.8","x-forwarded-for":"172.19.136.8","x-forwarded-host":"opensearch-dev.cust.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","content-length":"100","sec-ch-ua":"\"Not_A Brand\";v=\"99\", \"Google Chrome\";v=\"109\", \"Chromium\";v=\"109\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","osd-version":"2.5.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://opensearch-dev.cust.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://opensearch-dev.cust.com/app/alerting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,it;q=0.8,cs;q=0.7","securitytenant":""},"remoteAddress":"10.244.4.25","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","referer":"https://opensearch-dev.cust.com/app/alerting"},"res":{"statusCode":200,"responseTime":20,"contentLength":9},"message":"POST /api/alerting/monitors/_search 200 20ms - 9.0B"}
Alerting - MonitorService - createMonitor: StatusCodeError: [alerting_exception] [opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/mapping/put]
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (events.js:412:35)
    at IncomingMessage.emit (domain.js:475:12)
    at endReadableNT (internal/streams/readable.js:1333:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  status: 500,
  displayName: 'InternalServerError',
  path: '/_plugins/_alerting/monitors?refresh=wait_for',
  query: {},
  body: {
    error: {
      root_cause: [Array],
      type: 'alerting_exception',
      reason: '[opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/mapping/put]',
      caused_by: [Object]
    },
    status: 500
  },
  statusCode: 500,
  response: '{"error":{"root_cause":[{"type":"alerting_exception","reason":"[opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/mapping/put]"}],"type":"alerting_exception","reason":"[opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/mapping/put]","caused_by":{"type":"exception","reason":"org.opensearch.transport.RemoteTransportException: [opensearch-cluster-masters-2][10.244.6.183:9300][indices:admin/mapping/put]"}},"status":500}',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}
{"type":"response","@timestamp":"2023-02-01T17:04:10Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/api/alerting/monitors","method":"post","headers":{"host":"opensearch-dev.cust.com","x-request-id":"af539d9e6e0eb0f4c747312b4a3f6e63","x-real-ip":"172.19.136.8","x-forwarded-for":"172.19.136.8","x-forwarded-host":"opensearch-dev.cust.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","content-length":"1662","sec-ch-ua":"\"Not_A Brand\";v=\"99\", \"Google Chrome\";v=\"109\", \"Chromium\";v=\"109\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","osd-version":"2.5.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://opensearch-dev.cust.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://opensearch-dev.cust.com/app/alerting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,it;q=0.8,cs;q=0.7","securitytenant":""},"remoteAddress":"10.244.4.25","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","referer":"https://opensearch-dev.cust.com/app/alerting"},"res":{"statusCode":200,"responseTime":55,"contentLength":9},"message":"POST /api/alerting/monitors 200 55ms - 9.0B"}
Alerting - MonitorService - executeMonitor: StatusCodeError: Request Timeout after 30000ms
    at /usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:397:9
    at Timeout.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:429:7)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7) {
  status: undefined,
  displayName: 'RequestTimeout',
  body: undefined
}
{"type":"response","@timestamp":"2023-02-01T17:03:44Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/api/alerting/monitors/_execute","method":"post","headers":{"host":"opensearch-dev.cust.com","x-request-id":"b5419ca667678242d48374b590b9c7f7","x-real-ip":"172.19.136.8","x-forwarded-for":"172.19.136.8","x-forwarded-host":"opensearch-dev.cust.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","content-length":"803","sec-ch-ua":"\"Not_A Brand\";v=\"99\", \"Google Chrome\";v=\"109\", \"Chromium\";v=\"109\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","osd-version":"2.5.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://opensearch-dev.cust.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://opensearch-dev.cust.com/app/alerting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,it;q=0.8,cs;q=0.7","securitytenant":""},"remoteAddress":"10.244.4.25","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","referer":"https://opensearch-dev.cust.com/app/alerting"},"res":{"statusCode":200,"responseTime":30015,"contentLength":9},"message":"POST /api/alerting/monitors/_execute 200 30015ms - 9.0B"}
Alerting - MonitorService - executeMonitor: StatusCodeError: Request Timeout after 30000ms
    at /usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:397:9
    at Timeout.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:429:7)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7) {
  status: undefined,
  displayName: 'RequestTimeout',
  body: undefined
}
{"type":"response","@timestamp":"2023-02-01T17:03:44Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/api/alerting/monitors/_execute","method":"post","headers":{"host":"opensearch-dev.cust.com","x-request-id":"48f39b6448a2709df859d760330754e0","x-real-ip":"172.19.136.8","x-forwarded-for":"172.19.136.8","x-forwarded-host":"opensearch-dev.cust.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","content-length":"807","sec-ch-ua":"\"Not_A Brand\";v=\"99\", \"Google Chrome\";v=\"109\", \"Chromium\";v=\"109\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","osd-version":"2.5.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://opensearch-dev.cust.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://opensearch-dev.cust.com/app/alerting","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,it;q=0.8,cs;q=0.7","securitytenant":""},"remoteAddress":"10.244.4.25","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","referer":"https://opensearch-dev.cust.com/app/alerting"},"res":{"statusCode":200,"responseTime":30012,"contentLength":9},"message":"POST /api/alerting/monitors/_execute 200 30012ms - 9.0B"}
Alerting - MonitorService - executeMonitor: StatusCodeError: Request Timeout after 30000ms
    at /usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:397:9
    at Timeout.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:429:7)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7) {
  status: undefined,
  displayName: 'RequestTimeout',
  body: undefined
}

ashisagr-amzn · February 1, 2023, 10:23pm

Him thanks for calling this out. We currently have a PR to address this issue: [BUG] ExecuteMonitor inserting metadata doc during dry run by petardz · Pull Request #758 · opensearch-project/alerting · GitHub.

Have you tried to just create the monitor without running a dry run? If that blocks you, temporarily, you can use the API to create the alerting monitor by following this: API - OpenSearch documentation

anubisg1 · February 1, 2023, 10:57pm

I’ve tried to create it only from the dashboard. I believe that the dashboard automatically executes the queries as I write the monitor itself.

I’ll try with the API to see if anything changes, but to be honest, I’m barely able to do it via the UI

Topic		Replies	Views
[Error] Failed to load destinations, Failed to create monitor Alerting troubleshoot	2	1140	April 17, 2023
Unable to Create Dashboard or Visualization Observability	4	755	July 25, 2023
OpenSearch Alerting failed to update the Monitor Alerting	3	231	April 28, 2024
Error message for orphaned monitor OpenSearch	2	264	May 3, 2023
DNS SAP monitor throws error on create Security Analytics troubleshoot	8	355	July 7, 2023

Unable to create monitor in opensearch 2.5

Related topics