Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Ubuntu 22.04 Server
dpkg -l | grep opensearch
ii opensearch 2.8.0 amd64 An open source distributed and RESTful search engine
Describe the issue:
I have OpenSearch installed on this sever, it is paired with Graylog and MongoDB and has the function of receiving and storing UDP syslog messages from a number of routers and switches.
Upon initial installation, I assigned JVM heap space to 8-10 GB. After a while I realized that I don’t need that much memory since there’s no much traffic coming in to this server (about 5 messages per second on average, maybe peaks to a few hundred per second during outages/recovery).
So I decided to change the JVM heap space to 4-4 GB. The server has been doing fine since then, no change in performance whatsoever. However, I noticed a clear jump in network traffic after that change.
I’m just wondering if anyone has an explanation for this? Any guesses/ideas about the correlation between JVM heap space and network traffic?
Configuration:
cat /etc/opensearch/opensearch.yml | grep ^[^#]
cluster.name: graylog
node.name: ${HOSTNAME}
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
discovery.type: single-node
network.host: 0.0.0.0
action.auto_create_index: false
plugins.security.disabled: true
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test, C=de
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models"]
cat /etc/opensearch/jvm.options | grep ^[^#]
-Xms4g
-Xmx4g
8-10:-XX:+UseConcMarkSweepGC
8-10:-XX:CMSInitiatingOccupancyFraction=75
8-10:-XX:+UseCMSInitiatingOccupancyOnly
11-:-XX:+UseG1GC
11-:-XX:G1ReservePercent=25
11-:-XX:InitiatingHeapOccupancyPercent=30
-Djava.io.tmpdir=${OPENSEARCH_TMPDIR}
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/opensearch
-XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/opensearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
18-:-Djava.security.manager=allow
-Dclk.tck=100
-Djdk.attach.allowAttachSelf=true
-Djava.security.policy=file:///etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy
--add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED
Relevant Logs or Screenshots:
