# Possible issue?

**URL:** <https://forum.opensearch.org/t/possible-issue/2101>\
**Category:** Index Management\
**Created:** [January 16, 2020, 2:42pm UTC](https://forum.opensearch.org/t/possible-issue/2101 "2020-01-16T14:42:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jasonrojas](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/jasonrojas/32/4902_2.png) [@jasonrojas](https://forum.opensearch.org/u/jasonrojas)\
**Post date:** [January 16, 2020, 2:42pm UTC](https://forum.opensearch.org/t/possible-issue/2101/1 "2020-01-16T14:42:07Z")

</div>

Hi, we have a indexing heaving cluster that for the last few weeks since upgrading to 1.3.0 has been experiencing timeouts a few times a week. During the period where indexing stops I see this in the logs:

> org.elasticsearch.transport.RemoteTransportException: [master][1.1.1.1:9300][cluster:admin/ism/update/managedindexmetadata], Caused by: org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (opendistro-ism) within 30s, at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0(MasterService.java:134) ~[elasticsearch-7.3.2.jar:7.3.2], at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?], at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:133) ~[elasticsearch-7.3.2.jar:7.3.2], at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.3.2.jar:7.3.2], at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?], at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?], at java.lang.Thread.run(Thread.java:835) ~[?:?]

Not sure if this is anything important but the state of the managed ism indices was “running”.  
For a test I removed all policies and managed indices from ISM to see if there is any improvement.  
Has anyone else seen issues like this?

---

<div class="post-metadata">

**Author:** ![dbbaughe](https://avatars.discourse-cdn.com/v4/letter/d/bbe5ce/32.png) [@dbbaughe](https://forum.opensearch.org/u/dbbaughe)\
**Post date:** [January 16, 2020, 5:14pm UTC](https://forum.opensearch.org/t/possible-issue/2101/2 "2020-01-16T17:14:14Z")

</div>

Hi @jasonrojas,

How many managed indices do you have in ISM?  
When you removed the policies/managed indices from ISM did you see the timeouts stop or were they still happening?

Thanks,  
Drew

---

<div class="post-metadata">

**Author:** ![jasonrojas](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/jasonrojas/32/4902_2.png) [@jasonrojas](https://forum.opensearch.org/u/jasonrojas)\
**Post date:** [January 17, 2020, 2:28pm UTC](https://forum.opensearch.org/t/possible-issue/2101/3 "2020-01-17T14:28:22Z")

</div>

I think I had manually applied the policy to about 30 indices.

The cluster had no issues last night so only time will tell if this was the issue.

---

<div class="post-metadata">

**Author:** ![dbbaughe](https://avatars.discourse-cdn.com/v4/letter/d/bbe5ce/32.png) [@dbbaughe](https://forum.opensearch.org/u/dbbaughe)\
**Post date:** [January 17, 2020, 6:48pm UTC](https://forum.opensearch.org/t/possible-issue/2101/4 "2020-01-17T18:48:19Z")

</div>

@jasonrojas

30 indices wouldn’t be enough to cause master cluster state queue to be backed up purely from ISM.  
If you could also answer these questions, might be able to help us pinpoint and replicate:

- How much data do you have in the cluster
- How many indices/shards do you have
- What type of instances are you using
- What is the ingestion throughput you average

Also is the timeout exceptions only occurring for ISM cluster events or other cluster events too?

Thanks,  
Drew
