# Snapshots Tipping Over Data Nodes

**URL:** <https://forum.opensearch.org/t/snapshots-tipping-over-data-nodes/19400>\
**Category:** General Feedback\
**Created:** [May 15, 2024, 11:58pm UTC](https://forum.opensearch.org/t/snapshots-tipping-over-data-nodes/19400 "2024-05-15T23:58:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![BurntBaboon](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/burntbaboon/32/7531_2.png) [@BurntBaboon](https://forum.opensearch.org/u/BurntBaboon)\
**Post date:** [May 15, 2024, 11:58pm UTC](https://forum.opensearch.org/t/snapshots-tipping-over-data-nodes/19400/1 "2024-05-15T23:58:56Z")

</div>

Howdy!

We’ve noticed a peculiar issue where, rather sporadically, data nodes drop out from our cluster when our nightly snapshot policy executes. The interesting tidbit is that this is only happening for one out of a handful of clusters.

Currently, we haven’t gleaned much from the OpenSearch logs. The only noticeable errors being `GC did not bring memory usage down` and `failed to list shard`. However, if are contributing to the issue, we’re not entirely sure how to rectify the problem and why it’s only happening in one of our clusters.

From,  
DH

---

<div class="post-metadata">

**Author:** ![AmiSMB](https://avatars.discourse-cdn.com/v4/letter/a/b77776/32.png) [@AmiSMB](https://forum.opensearch.org/u/AmiSMB)\
**Post date:** [May 24, 2024, 1:25pm UTC](https://forum.opensearch.org/t/snapshots-tipping-over-data-nodes/19400/2 "2024-05-24T13:25:25Z")

</div>

We are seeing the same thing happening where the backup is starting and then data nodes are dropping and then we have indexes that are unavailable.

 ![Screenshot 2024-05-24 at 14.25.06](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/7/7d41796841924eef66c567d3d2ff3027495fbec6.png)

---

<div class="post-metadata">

**Author:** ![Nilushan](https://avatars.discourse-cdn.com/v4/letter/n/aca169/32.png) [@Nilushan](https://forum.opensearch.org/u/Nilushan)\
**Post date:** [July 4, 2024, 11:20am UTC](https://forum.opensearch.org/t/snapshots-tipping-over-data-nodes/19400/3 "2024-07-04T11:20:09Z")

</div>

@BurntBaboon ,  
I have also experienced this issue with snapshots. This happened because the Java Heap went out of memory when taking snapshots. To confirm if it is the same issue that you are experiencing, could you please provide the following information?

1. Can you share the response body of [Get Snapshot - OpenSearch Documentation](https://opensearch.org/docs/latest/api-reference/snapshots/get-snapshot/) API call?. Need to check the `failures` and `shards` fields as it will provide a clue

2. Do you see `java.lang.OutOfMemoryError: Java heap space` errors in the OpenSearch data node logs? This should get printed after the `GC did not bring memory usage down` errors