This is for opendistro-for-elasticsearch-kibana:0.10.0. Once I am logged in, occasionally it takes 15-20 seconds to switch between the index patterns in the Discover tab. Here is an example showing the request breakdown using the Chrome console:
https://i.stack.imgur.com/XxKsS.png
_msearch is the part where Kibana initiates the ES query to bring the latest 15 minutes worth results for given index pattern, that takes 127ms so I am pretty sure that things get slow internally, inside the Kibana system.
Same Kibana UI, 15 minutes later, no config changes and the same request takes ~120ms:
https://i.stack.imgur.com/V0gqc.png
3 stages this time (no _find?type=index-pattern&per_page=10000 request, no idea why?). I do acknowledge there may be some caching involved but it seems to me things are happening at random here. Kibana verbose logging did not help either, it basically confirms the stages happening as I see those in the Chrome console. What I would like to understand is why those _find and _bulk_get requests take that long.
Welcome @skydancer!
Anything notable about your cluster? Do you have a lot of patterns or indices? My first thought is that it’s just doing a TON of work or doing some sort of GC sweep.
Also, of note, if you’re still using 0.10.0, you should consider upgrading to OpenSearch or at least a newer version of Open Distro. 0.10.0 is years out of date now.
Upgrade is planned by the end of this year.
The issue is totally intermittent, I can see it on a relatively empty cluster (5 index patterns): request takes 15-20s, minutes later it is a matter of mere 100-200ms. So far I have not found a way of replicating the problem, I just see it happening when making one request after another.
Are you memory constrained?
Definitely not: https://i.stack.imgur.com/ydJZ2.png. This is one of the first things I checked. And like I mentioned earlier, this is a relatively quiet cluster, not many folks using it so not much work for Kibana.
Yeah, looks like memory isn’t the issue at all.
Anything going on in the network? There isn’t a lot to go on (probably not from your side either)…
