Index size spikes during snapshotting

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.12

Describe the issue:

We run a daily snapshot that’s uploaded to a GCS repository. During this snapshot, we see index size increase by about 20%.

What could cause this? Does snapshotting build auxiliary files?

Not sure if this is relevant but we update a large percentage of the indexed docs in the period between snapshots.

Configuration:

OpenSearch 2.12 running on general-purpose SSD-backed nodes on GCP.

Relevant Logs or Screenshots:
Snapshotting is running at 12am. The following is a screenshot of our index size metrics.

image
image1048×250 20.1 KB

2

Maybe it’s caused by the merge operation, merge operation copied the segment files to a new one and then delete the old files, it it’ll increase the disk usage in a short period, so you can check that if the merge operation performs before taking the snapshot.

1 Like
3

Yep, that’s it. We have a force merge operation that starts around the time the snapshot starts. The index size spike coincides with that merge kicking off.

Thanks for the help!