DevTools autocomplete refresh permanently increase JVM heap old gen

We have strange problem with our ODFE 1.10 , based cluster (7.9.1 ES)

Old gen part of JVM heap on our cluster`s coordinator nodes constantly increasing till OOM.

Memory leaking not related with cluster load but precisely correlated with requests generated by Kibana DevTool (when autocomplete enabled and Console used by admin role)
In cluster audit problem requests looks like
GET /*/_mapping

For some reason such requests is

  1. too heavy for our cluster
  2. data newer collected with GC (stuck at Old Gen)

Is there any way to

  1. Disable autocomplete\refresh for all users of DevTools (including new) ?
    (I tried to set kibana.autocompleteTerminateAfter=1 , but without any luck )

  2. Understand why this “standard Kibana” request so heavy for our cluster ?
    ( too many indecies? too many fields ? )

  3. Understand why ES coordinator nodes never free memory filled with this requests result data ?
    (old gen heap consumption grow till node OOM)

Any thoughts/advice would be great.

@val16384 That’s super interesting. Could you tell us more about your cluster?

Nothing special. Mostly logs indexing load.

Anything particular with size, # of nodes, etc?

28 - data nodes
3 dedicated master
2 coordinators

About 5000 shards

1 Like

Thanks - good context. Trying to triangulate this with some other issues on 7.9.