# Nodes crashes, leader check **URL:** \ **Category:** Open Source Elasticsearch and Kibana\ **Created:** [June 22, 2021, 1:38pm UTC](https://forum.opensearch.org/t/nodes-crashes-leader-check/6309 "2021-06-22T13:38:46Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![doc113](https://avatars.discourse-cdn.com/v4/letter/d/8797f3/32.png) [@doc113](https://forum.opensearch.org/u/doc113)\ **Post date:** [June 22, 2021, 1:38pm UTC](https://forum.opensearch.org/t/nodes-crashes-leader-check/6309/1 "2021-06-22T13:38:46Z")
Greetings all! Previously, sorry for my English. Having trouble with ES cluster. Sometimes data-nodes crashes (one or more (rarely)). This can be preceded by different events: force-merge, relocating shards, real-time upload data to indices, creating replicas, but the logs are always the same (I attach the link below to DEBUG-level logs). About real-time bulk-upload data: 1. every 10 minutes, 3 indices (15 shard with no replica, kNN) 2. data size ~100k documents (during rush hour) = ~200-400Mb (depending on the index) 3. upload time spent ~60 seconds (upload + flush), translog = 512Mb 4. in total, 10-15 million documents are collected per day (1 day = 1 index), in amount 270 indices with replicas (3 months, 3 different sources) 5. example index template: > { > “settings”: { > “index”: { > “knn”: “true”, > “number\_of\_shards”: 15, > “number\_of\_replicas”: 0, > “knn.space\_type”: “cosinesimil”, > “knn.algo\_param.m”: 48, > “knn.algo\_param.ef\_construction”: 8192, > “knn.algo\_param.ef\_search”: 8192, > “max\_result\_window”: 1000000 > } > }, > “mappings”: { > “\_source”: { > “excludes”: [ > “my\_vector” > ] > }, > “properties”: { > “detect\_id”: { > “type”: “long” > }, > “cam\_id”: { > “type”: “integer” > }, > “time\_check”: { > “format”: “yyyy-MM-dd HH:mm:ss”, > “store”: True, > “type”: “date” > }, > “my\_vector”: { > “type”: “knn\_vector”, > “dimension”: 320 > } > } > } > } The same indexes are then force\_merged (at night, when the load is small) Cluster formation: 1. 3 master nodes, 4core, RAM=4gb, heap=2gb, configuration file: > cluster.name: open-distro-cluster > node.name: es-masternode001 > node.master: true > node.data: false > node.ingest: false > network.host: 10.250.9.100 > discovery.seed\_hosts: [“10.250.9.100”, “10.250.9.102”, “10.250.9.104”] > cluster.initial\_master\_nodes: [“10.250.9.100”, “10.250.9.102”, “10.250.9.104”] 1. 3 coordinator nodes, 14core, RAM=30gb, heap=15gb, same configuration file (node.master: false) 2. 70 data-nodes, 18core, RAM=368gb, heap=30gb, same configuration file (node.master: false, node.data:True, node.ingest:True) 3. cluster changed parameters: > { > “persistent”: { > “cluster”: { > “routing”: { > “rebalance”: { > “enable”: “none” > }, > “allocation”: { > “allow\_rebalance”: “indices\_all\_active”, > “cluster\_concurrent\_rebalance”: “15”, > “node\_concurrent\_recoveries”: “2”, > “disk”: { > “threshold\_enabled”: “true”, > “watermark”: { > “low”: “80%”, > “high”: “85%” > } > }, > “enable”: “all”, > “node\_concurrent\_outgoing\_recoveries”: “2” > } > }, > “metadata”: { > “perf\_analyzer”: { > “state”: “0” > } > } > }, > “knn”: { > “algo\_param”: { > “index\_thread\_qty”: “4” > }, > “memory”: { > “circuit\_breaker”: { > “limit”: “80%”, > “enabled”: “true” > } > } > } > }, > “transient”: {} > } 1. master-log and data-node log: [ES logs - Google Drive](https://drive.google.com/drive/folders/1Hy36qGh1bIrdLGDxjtfFG1ywqKlwSSqy?usp=sharing) first crash at “2021-06-21T12:12:59,356Z”, manually restart ES service at “2021-06-21T12:47:46,314Z”, normally works at “2021-06-21T12:53:54,577Z”. At the same time, two other nodes broke, near to “2021-06-21T14:14:16,213Z” all nodes in cluster started working normally second crash at “2021-06-21T22:35:17,319Z”, normally started at “2021-06-22T01:22:43,459Z” (without my doing anything) Thoughts about it - it is not trouble with network (because every crash = any actions other than real-time upload data, the cluster can work normally for a week and more). Monitoring in Zabbix shows no anomalies, CPU utilization ~25-30%, RAM utilization on every node depends on size of indices. I can provide logs of other crashes, but in general the situation is the same everywhere - many times “leader check”, then starts working normally… I do not know what the problem may be, I am grateful for any advice! Thanks! ---
**Author:** ![searchymcsearchface](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/searchymcsearchface/32/868_2.png) [@searchymcsearchface](https://forum.opensearch.org/u/searchymcsearchface)\ **Post date:** [June 22, 2021, 2:05pm UTC](https://forum.opensearch.org/t/nodes-crashes-leader-check/6309/2 "2021-06-22T14:05:10Z")
hey @doc113 - what version are you using? ---
**Author:** ![doc113](https://avatars.discourse-cdn.com/v4/letter/d/8797f3/32.png) [@doc113](https://forum.opensearch.org/u/doc113)\ **Post date:** [June 22, 2021, 2:31pm UTC](https://forum.opensearch.org/t/nodes-crashes-leader-check/6309/3 "2021-06-22T14:31:05Z")
I completely forgot to specify - 1.12.0 ---
**Author:** ![colinmatthew](https://avatars.discourse-cdn.com/v4/letter/c/f6c823/32.png) [@colinmatthew](https://forum.opensearch.org/u/colinmatthew)\ **Post date:** [August 19, 2022, 4:57pm UTC](https://forum.opensearch.org/t/nodes-crashes-leader-check/6309/4 "2022-08-19T16:57:12Z")
Hi @searchymcsearchface , @doc113 I just wanted to add that I’m seeing the same issue when using 1.12.0 in aws opensearch service. So far nodes are deterministically disconnecting everytime I do (1) force merge or (2) an index warm up. For both actions I see the issues arise on indices that use a FAISS model with - dimension 512 - method = ivf - encoding = pq - m = 256 (seen the same at 512) - nlist = 2048 - nprobes = 25 Trained on 1M example vectors. The warm up actions are on indices with ~3-10M vectors across 8 primary shards, and end up causing a short disconnect for one or more nodes very consistently. The cluster recovers afterwards and generally a repeated attempt will be successful. My cluster is currently 10 data nodes, each with 32GB ram - but we were seeing the same issue when running 40 nodes in our cluster also. I’ll add that I don’t think this happened when using an index built using nmslib, only FAISS. Nmslib was also much quicker to warm up. Any thoughts since the last time this topic was visited?