We are experiencing bloated OpenSearch (and Lucene) indices that still carry a large number of docs marked for deletion, or docs that have older versions that were not purged yet:
We have (“keep-latest-per-track-id” type of) workflows where we need to support constantly updating documents by _id, or need to delete a doc from one (rolling-date) index and “move” it to another one, which leaves the doc marked for deletion in the old index.
We are seeing that overtime the indices, especially the ones with (medium to large) geo_shape geometries, start to get bloated to the point where we might have only ~10 docs but the index size on disk is huge since it still has thousands of older versions for each of the docs.
I know that eventually (we haven’t seen that yet) the old versions and docs that are marked for deletion should get cleared (purged) and the index will reduce its footprint on disk, but this is not happening for us (perhaps because we keep writing and updating docs on that index).
I was wondering if there are some index settings we can set on these indices to make the clearing/purging/“compacting” process happen or happen more frequently.
One of our main requirements is that the index has to stay “live” during this operation (I called it “comparing” for lack of a better name), as we need to be able to keep writing and reading from the index, while it is “compacting” (deleting tombstones data docs and versions) in the background.
Hi @hanoch,
normally the deletion of this Documents will happen when there is a forcemerge on the Index.
You can trigger this manually with this command:
POST /_forcemerge
With this you will merge the Apache Lucene Segments of your shards.
Please try this and share if this cleaned up some Space for you.
But be careful as this may take some CPU resources so do this outside normal working hours.
Just for better Understanding, why are you deleting some Documents in your Index?
Maybe there is a better Solution we can recommend you.
normally the deletion of this Documents will happen when there is a forcemerge on the Index.
One of our main requirements is that the index has to stay “live” during this operation
Note that the force merge API will block all calls to the index so it will not say “live”.
The key takeaway here is that purging deleted docs occurs during a merge and it is generally discouraged to force a merge unless it’s absolutely necessary (even then only do it during a maintenance period). There are some configurations you could experiment with that might be helpful.
First, the default merge policy has a bias towards merging many small segments and if a segment becomes “large” it will avoid merging altogether. If many of your deleted documents are in these large segments they may not be purged until a second criteria is met (mentioned below). You can start by lowering index.merge.policy.max_merged_segment to something like 1 or 2gb and see if that helps purge deleted documents more often.
Second, the above setting may not help purge deleted documents because of “soft deletes” which retains history of a document until a sequence number is committed on all shards (primaries and replicas) or a retention lease period expires (default 12h). You can try lowering the index.soft_deletes.retention_lease.period setting to something like 2h and see if that (along with the above setting) strikes a good balance between purging deleted documents while ensuring good recovery settings?