Problem with RCA - Over 138k files generated

Hello,

We are using kubernetes with opendistro. We started to notice an issue where /var would start filling up and tracked the issue to our pods that are running opendistro 1.9.0. The /tmp directory in the pod is between 16-30Gb in size and contains over 138 000 files with the name “metricsdb”:

Is this related to the newly implemented RCA? Can we somehow disable it?

Thanks

I don’t know the answer however steps to enable/disable the new RCA feature are described here: performance-analyzer-rca/INSTALL.md at main · opendistro-for-elasticsearch/performance-analyzer-rca · GitHub

Let us know what you find out!

Hi @victor,

Can you check the performance-analyzer.properties file located inside
cat /usr/share/elasticsearch/performance-analyzer-rca/pa_config/performance-analyzer.properties (for the docker image) and see if the property,
cleanup-metrics-db-files = true ?

To disable PA, you can use

curl localhost:9200/_opendistro/_performanceanalyzer/cluster/config -d '{"enabled": false}' -H 'Content-Type: application/json'