Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch version = 2.5.0
Dashboard = Graylog 5
ServerOS = Debian 11.6 LXC container (Proxmox host OS) with kernel 5.15.102-1-pve #1 SMP PVE 5.15.102-1
Browser = Safari 16.3 (18614.4.6.1.6)
LXC resources: 12GB RAM, 4 cores or a Ryzen 5-2600; ram for opensearch is 7G for xmx and xms, graylog gets 3GB for xmx and xms
Describe the issue:
earlier today I added a second custom index set in graylog and ever since opensearch has been very unstable. Once I got an error message about missing/wrong privileges in the /var/log/opensearch directory, but even after fixing this, it does not run properly. I’m kind of at a loss for the actual cause because the logs are missing any relevant info (see log section).
Configuration:
root@graylog2:~# cat /etc/opensearch/opensearch.yml
Summary
======================== OpenSearch Configuration =========================
NOTE: OpenSearch comes with reasonable defaults for most settings.
Before you set out to tweak and tune the configuration, make sure you
understand what are you trying to accomplish and the consequences.
The primary way of configuring a node is via this file. This template lists
the most important settings you may want to configure for a production cluster.
Please consult the documentation for further information on configuration options:
---------------------------------- Cluster -----------------------------------
Use a descriptive name for your cluster:
cluster.name: graylog
------------------------------------ Node ------------------------------------
Use a descriptive name for the node:
node.name: node-1
Add custom attributes to the node:
#node.attr.rack: r1
----------------------------------- Paths ------------------------------------
Path to directory where to store the data (separate multiple locations by comma):
path.data: /var/lib/opensearch
Path to log files:
path.logs: /var/log/opensearch
----------------------------------- Memory -----------------------------------
Lock the memory on startup:
#bootstrap.memory_lock: true
Make sure that the heap size is set to about half the memory available
on the system and that the owner of the process is allowed to use this
limit.
OpenSearch performs poorly when the system is swapping the memory.
---------------------------------- Network -----------------------------------
Set the bind address to a specific IP (IPv4 or IPv6):
network.host: 192.168.2.210
Set a custom port for HTTP:
http.port: 9200
For more information, consult the network module documentation.
--------------------------------- Discovery ----------------------------------
Pass an initial list of hosts to perform discovery when this node is started:
The default list of hosts is [“127.0.0.1”, “[::1]”]
#discovery.seed_hosts: [“host1”, “host2”]
discovery.type: single-node
action.auto_create_index: false
plugins.security.disabled: true
Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:
#cluster.initial_cluster_manager_nodes: [“node-1”, “node-2”]
For more information, consult the discovery and cluster formation module documentation.
---------------------------------- Gateway -----------------------------------
Block initial recovery after a full cluster restart until N nodes are started:
#gateway.recover_after_nodes: 3
For more information, consult the gateway module documentation.
---------------------------------- Various -----------------------------------
Require explicit names when deleting indices:
#action.destructive_requires_name: true
######## Start OpenSearch Security Demo Configuration ########
WARNING: revise all the lines below before you go into production
#plugins.security.ssl.transport.pemcert_filepath: esnode.pem
#plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
#plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
#plugins.security.ssl.transport.enforce_hostname_verification: false
#plugins.security.ssl.http.enabled: true
#plugins.security.ssl.http.pemcert_filepath: esnode.pem#
#plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
#plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
#plugins.security.allow_unsafe_democertificates: true
#plugins.security.allow_default_init_securityindex: true
#plugins.security.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test, C=de
#plugins.security.audit.type: internal_opensearch
#plugins.security.enable_snapshot_restore_privilege: true
#plugins.security.check_snapshot_restore_write_privileges: true
#plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
#plugins.security.system_indices.enabled: true
#plugins.security.system_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
#node.max_local_storage_nodes: 3
######### End OpenSearch Security Demo Configuration ########
Relevant Logs or Screenshots:
/var/log/opensearch/gc.log.22
Summary
[2023-04-13T13:11:51.342+0000][23366][gc] Using G1
[2023-04-13T13:11:51.363+0000][23366][gc,init] Version: 17.0.5+8 (release)
[2023-04-13T13:11:51.363+0000][23366][gc,init] CPUs: 12 total, 4 available
[2023-04-13T13:11:51.363+0000][23366][gc,init] Memory: 64240M
[2023-04-13T13:11:51.363+0000][23366][gc,init] Large Page Support: Disabled
[2023-04-13T13:11:51.363+0000][23366][gc,init] NUMA Support: Disabled
[2023-04-13T13:11:51.363+0000][23366][gc,init] Compressed Oops: Enabled (Zero based)
[2023-04-13T13:11:51.363+0000][23366][gc,init] Heap Region Size: 4M
[2023-04-13T13:11:51.363+0000][23366][gc,init] Heap Min Capacity: 7G
[2023-04-13T13:11:51.363+0000][23366][gc,init] Heap Initial Capacity: 7G
[2023-04-13T13:11:51.363+0000][23366][gc,init] Heap Max Capacity: 7G
[2023-04-13T13:11:51.363+0000][23366][gc,init] Pre-touch: Disabled
[2023-04-13T13:11:51.363+0000][23366][gc,init] Parallel Workers: 4
[2023-04-13T13:11:51.363+0000][23366][gc,init] Concurrent Workers: 1
[2023-04-13T13:11:51.363+0000][23366][gc,init] Concurrent Refinement Workers: 4
[2023-04-13T13:11:51.363+0000][23366][gc,init] Periodic GC: Disabled
[2023-04-13T13:11:51.364+0000][23366][gc,metaspace] CDS archive(s) not mapped
[2023-04-13T13:11:51.364+0000][23366][gc,metaspace] Compressed class space mapped at: 0x00000007c0000000-0x0000000800000000, reserved size: 1073741824
[2023-04-13T13:11:51.364+0000][23366][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 3, Narrow klass range: 0x800000000
[2023-04-13T13:11:51.448+0000][23366][gc,heap,exit] Heap
[2023-04-13T13:11:51.448+0000][23366][gc,heap,exit] garbage-first heap total 7340032K, used 6146K [0x0000000600000000, 0x00000007c0000000)
[2023-04-13T13:11:51.448+0000][23366][gc,heap,exit] region size 4096K, 2 young (8192K), 0 survivors (0K)
[2023-04-13T13:11:51.448+0000][23366][gc,heap,exit] Metaspace used 4570K, committed 4736K, reserved 1056768K
[2023-04-13T13:11:51.448+0000][23366][gc,heap,exit] class space used 358K, committed 448K, reserved 1048576K
/var/log/opensearch/gc.log.22 (END)
/var/log/opensearch/gc.log
[2023-04-13T13:11:51.506+0000][23213][gc] Using G1
journalctl -xe
Summary
Apr 13 15:11:49 graylog2 systemd[1]: Starting OpenSearch…
░░ Subject: A start job for unit opensearch.service has begun execution
░░ Defined-By: systemd
░░ Support:
░░
░░ A start job for unit opensearch.service has begun execution.
░░
░░ The job identifier is 1771.
Apr 13 15:12:03 graylog2 systemd[1]: opensearch.service: Main process exited, code=killed, status=9/KILL
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support:
░░
░░ An ExecStart= process belonging to unit opensearch.service has exited.
░░
░░ The process’ exit code is ‘killed’ and its exit status is 9.
Apr 13 15:12:03 graylog2 systemd[1]: opensearch.service: Failed with result ‘signal’.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support:
░░
░░ The unit opensearch.service has entered the ‘failed’ state with result ‘signal’.
Apr 13 15:12:03 graylog2 systemd[1]: Failed to start OpenSearch.
░░ Subject: A start job for unit opensearch.service has failed
░░ Defined-By: systemd
░░ Support:
░░
░░ A start job for unit opensearch.service has finished with a failure.
░░
░░ The job identifier is 1771 and the job result is failed.
Apr 13 15:12:03 graylog2 systemd[1]: opensearch.service: Consumed 9.152s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support:
The unit opensearch.service completed and consumed the indicated resources.
root@graylog2:~# systemctl status opensearch
Summary
● opensearch.service - OpenSearch
Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
Active: failed (Result: signal) since Thu 2023-04-13 15:23:09 CEST; 23s ago
Docs:
Process: 23604 ExecStart=/usr/share/opensearch/bin/systemd-entrypoint -p ${PID_DIR}/opensearch.pid --quiet (code=killed, signal=KILL)
Main PID: 23604 (code=killed, signal=KILL)
CPU: 7.427s
Apr 13 15:22:50 graylog2 systemd[1]: opensearch.service: Failed with result ‘signal’.
Apr 13 15:22:50 graylog2 systemd[1]: Failed to start OpenSearch.
Apr 13 15:22:50 graylog2 systemd[1]: opensearch.service: Consumed 9.257s CPU time.
Apr 13 15:22:51 graylog2 systemd[1]: Starting OpenSearch…
Apr 13 15:23:09 graylog2 systemd[1]: opensearch.service: Main process exited, code=killed, status=9/KILL
Apr 13 15:23:09 graylog2 systemd[1]: opensearch.service: Failed with result ‘signal’.
Apr 13 15:23:09 graylog2 systemd[1]: Failed to start OpenSearch.
Apr 13 15:23:09 graylog2 systemd[1]: opensearch.service: Consumed 7.427s CPU time.
I’m very new to opensearch and due to missing crash reason indicators I have no idea how to troubleshoot this apart from throwing more ram at it. especially since it was running fine over night with the same config it has right now. any ideas would be greatly appreciated. If I missed details, please let me know.