Before snapshot : is an operation "flush" needed?

OpenSearch
1

Versions : latest (non relevant)

Describe the issue: Before taking an index snapshot, I have data in the in-memory ingesting buffer (and the translog too).

Does taking a snapshot trigger a refresh and a flush of this in-memory data before the snapshot actually takes place?

Or do I need to perform a “refresh” operation and a “flush” operation before taking the snapshot?

The goal: I want all data ingested up to time t (whether on disk or still in memory) to be included in the upcoming snapshot.

Thanks.

Configuration: non relevant.

2

Hi @dmdevito,

For a regular snapshot, you do not need to manually run a flush or refresh beforehand.
When OpenSearch takes a shard snapshot, it acquires the latest Lucene index commit with flushFirst enabled. The source code explicitly notes:
// we flush first to make sure we get the latest writes snapshotted
https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/snapshots/SnapshotShardsService.java#L471
A manual refresh is also not required, as the snapshot is based on the Lucene commit, and the flush needed to acquire the latest commit is handled internally by the snapshot process.
I also verified this in a lab by disabling automatic refresh, indexing a document, and creating a snapshot without manually running either refresh or flush. After deleting and restoring the index from the snapshot, the document was successfully restored.

Hope this helps!

3

Hi @Yanlin

Thanks for your response and checking your side. At the same time (more or less), I came to the conclusion that it would be better to check myself with an OpenSearch instance. Of course, the result was the same.

Anyway, my additionnal point is that the behavior of OpenSearch snapshot should be part of the snapshot contract, and then, part of the documentation (and unfortunately, it’s not the case).
Thanks for all
Dominique