Performance Analyzer settings problems

I am running into some issue of the performance analyzer plugin.
The first 1 is that every minutes it dump the metrics it has collected to stdout
(I am running open distro 1.9 in docker)
Update: I am getting these logs when I upgrade a existing conatiner from 1.3 to 1.9.
On a fresh install on a new machine (no existing database) these logs dont showup. (default logs settings, no changes to log files)

[2020-08-14T10:16:14,967][INFO ][stats_log                ] [d1fc2515b3d2] ------------------------------------------------------------------------
EndTime=Fri, 14 Aug 2020 10:16:14 UTC
Time=1597400174967 msecs
[2020-08-14T10:16:14,969][INFO ][stats_log                ] [d1fc2515b3d2] ------------------------------------------------------------------------
EndTime=Fri, 14 Aug 2020 10:16:14 UTC
Time=1597400174969 msecs
[2020-08-14T10:16:14,969][INFO ][stats_log                ] [d1fc2515b3d2] ----------------------------------------

I have tried to disabled the info log of the PerformanceAnalyzerPlugin by editing

By settings the log levels to FATAL or by removing the console output references.
But so far no result.

I am also seeing a lot of metricsdb_ files in /tmp.
Currectly after a day around 16000. The files are not that large (100k each). But still previous version did not do this.

-rw-r--r-- 1 elasticsearch elasticsearch 110K Aug 14 10:19 metricsdb_1597400375000
-rw-r--r-- 1 elasticsearch elasticsearch 113K Aug 14 10:20 metricsdb_1597400380000
-rw-r--r-- 1 elasticsearch elasticsearch 106K Aug 14 10:20 metricsdb_1597400385000
-rw-r--r-- 1 elasticsearch elasticsearch 106K Aug 14 10:20 metricsdb_1597400390000
-rw-r--r-- 1 elasticsearch elasticsearch 111K Aug 14 10:20 metricsdb_1597400395000

Does anyone have a any suggestions on how to fix these 2 problems?

Current log4j2.xml

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="FATAL">
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
        <File name="PerformanceAnalyzerLog" fileName="/appl/logs/elasticsearch/nodeName/PerformanceAnalyzer.log" immediateFlush="true" append="true">
            <PatternLayout pattern="%d{yyy-MM-dd HH:mm:ss.SSS} [PA:Reader] [%t] %-5level %logger{36} - %msg%n"/>
        <File name="StatsLog" fileName="/appl/logs/elasticsearch/nodeName/performance_analyzer_agent_stats.log" immediateFlush="true" append="true">
        <Logger name="stats_log" level="FATAL" additivity="false">
            <AppenderRef ref="StatsLog"/>
        <Root level="FATAL">
            <AppenderRef ref="Console" />
            <AppenderRef ref="PerformanceAnalyzerLog"/>

however this is the default that comes with the docker container.

# ======================== Elasticsearch performance analyzer plugin config =========================

# NOTE: this is an example for Linux. Please modify the config accordingly if you are using it under other OS.

# Metrics data location
metrics-location = /dev/shm/performanceanalyzer/

# Metrics deletion interval (minutes) for metrics data.
# Interval should be between 1 to 60.
metrics-deletion-interval = 1

# If set to true, the system cleans up the files behind it. So at any point, we should expect only 2
# metrics-db-file-prefix-path files. If set to false, no files are cleaned up. This can be useful, if you are archiving
# the files and wouldn't like for them to be cleaned up.
cleanup-metrics-db-files = true

# WebService exposed by App's port
webservice-listener-port = 9600

# Metric DB File Prefix Path location
metrics-db-file-prefix-path = /tmp/metricsdb_

https-enabled = false

#Setup the correct path for certificates
certificate-file-path = specify_path

private-key-file-path = specify_path

# WebService bind host; default only to local interface
#webservice-bind-host =

# Plugin Stats Metadata file name, expected to be in the same location
plugin-stats-metadata = plugin-stats-metadata

# Agent Stats Metadata file name, expected to be in the same location
agent-stats-metadata = agent-stats-metadata

I have now upgraded to 1.8.
So far everything looks good and i am not having the same issues as above.

I found the same problems in 1.9. Hosts filesystems went full because of container’s /tmp/metricsdb_* files.

Had to downgrade to 1.8 :neutral_face:

Related issue: Disk full of files metricsdb_* on /tmp dir after disabling plugin. · Issue #178 · opendistro-for-elasticsearch/performance-analyzer · GitHub