# \_cat/indices reports docs.deleted decreasing

**URL:** <https://forum.opensearch.org/t/cat-indices-reports-docs-deleted-decreasing/21021>\
**Category:** OpenSearch\
**Created:** [August 30, 2024, 12:02am UTC](https://forum.opensearch.org/t/cat-indices-reports-docs-deleted-decreasing/21021 "2024-08-30T00:02:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://forum.opensearch.org/u/reshippie)\
**Post date:** [August 30, 2024, 12:02am UTC](https://forum.opensearch.org/t/cat-indices-reports-docs-deleted-decreasing/21021/1 "2024-08-30T00:02:33Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):  
2.14.0

**Describe the issue** :  
We have Logstash creating daily indices of logs collected. I used curl to hit `_cat/indices/$INDEX` twice in about 30 seconds. The first time it reported `docs.count - 24347115` and `docs.deleted - 562865`. The second time it reported `docs.count - 24357773` and `docs.deleted - 560369`.  
How is that possible?

I was initially investigating the fact that we have such a high number of deleted documents for an index that should only ever be added to. I am misunderstanding what `docs.deleted` means?

---

<div class="post-metadata">

**Author:** ![yeonghyeonKo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/yeonghyeonko/32/8202_2.png) [@yeonghyeonKo](https://forum.opensearch.org/u/yeonghyeonKo)\
**Post date:** [August 30, 2024, 8:07am UTC](https://forum.opensearch.org/t/cat-indices-reports-docs-deleted-decreasing/21021/2 "2024-08-30T08:07:43Z")

</div>

Actually, both OpenSearch and Elasticsearch cannot store deleted documents forever. When you try to delete a document, the cluster just checks whether it is to be searched or not because there is no need to **merge** segments in real time. Lucene uses **Bitsets** data structure for marking deletion.  
(The tasks for merging segments need quite high CPU usage and Disk I/O. But you can still test it using **[\_forcemerge API](https://opensearch.org/docs/latest/api-reference/index-apis/force-merge/)**)

Data nodes periodically merge segments in background for optimizing the performance.  
At the same time, deleted documents won’t be moved to new segments. (JUST alive documents will be)

---

<div class="post-metadata">

**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://forum.opensearch.org/u/reshippie)\
**Post date:** [August 30, 2024, 5:28pm UTC](https://forum.opensearch.org/t/cat-indices-reports-docs-deleted-decreasing/21021/3 "2024-08-30T17:28:38Z")

</div>

Are you saying that `docs.deleted` means the number of documents marked for deletion, not the number of documents actually deleted?

---

<div class="post-metadata">

**Author:** ![yeonghyeonKo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/yeonghyeonko/32/8202_2.png) [@yeonghyeonKo](https://forum.opensearch.org/u/yeonghyeonKo)\
**Post date:** [August 31, 2024, 12:32am UTC](https://forum.opensearch.org/t/cat-indices-reports-docs-deleted-decreasing/21021/4 "2024-08-31T00:32:45Z")

</div>

yes, until each data node finish merging its segments. At the time old segments are combined into new one, you can say that the documents marked as to be deleted are completely passed away.

---

<div class="post-metadata">

**Author:** ![reshippie](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@reshippie](https://forum.opensearch.org/u/reshippie)\
**Post date:** [August 31, 2024, 1:03am UTC](https://forum.opensearch.org/t/cat-indices-reports-docs-deleted-decreasing/21021/5 "2024-08-31T01:03:22Z")

</div>

If that’s the case, then why does calling `_forcemerge` on an index with the value of `docs.deleted` in the hundreds of thousands not have any effect on that value?  
I can guarantee that nothing is being written to that index.  
I can also guarantee that there is no way hundreds of thousands of documents were deleted from that index in the first place.

---

<div class="post-metadata">

**Author:** ![yeonghyeonKo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/yeonghyeonko/32/8202_2.png) [@yeonghyeonKo](https://forum.opensearch.org/u/yeonghyeonKo)\
**Post date:** [September 10, 2024, 12:59am UTC](https://forum.opensearch.org/t/cat-indices-reports-docs-deleted-decreasing/21021/6 "2024-09-10T00:59:38Z")

</div>

Can you show the number and the size of indices?  
Also, your indices might be read-only after a day, right? (as you explained Logstash collects logs daily.) The reason why I ask you whether the indices are read-only or not is that making force-merge an index receiving writes can cause very large segments to be produced and make snapshots more expensive than before. ([Force merge API | Elasticsearch Guide [8.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html#forcemerge-api-desc))

When you try calling **\_forcemerge API** , i recommend you to attach **wait\_for\_completion=false** query parameter(asynchronously) so the task can still be running in the background, not falling into a connection loss.

---

<div class="post-metadata">

**Author:** ![yeonghyeonKo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/yeonghyeonko/32/8202_2.png) [@yeonghyeonKo](https://forum.opensearch.org/u/yeonghyeonKo)\
**Post date:** [September 10, 2024, 2:50am UTC](https://forum.opensearch.org/t/cat-indices-reports-docs-deleted-decreasing/21021/7 "2024-09-10T02:50:30Z")

</div>

**TieredMergePolicy** which is allowed to merge non-adjacent segments (sorted by size) is a default merging policy in Lucene.

In my opinion(and i’m not sure), force-merge API doesn’t always allow us to delete documents.

As you see the below graph([Changing Bits: Visualizing Lucene's segment merges](https://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html)), the dark-grey band on top of each segment bar increases i.e the proportion of deletions in a segment is incrementing until TieredMergePolicy determines an optimizing merge based on “ **budget** ”.

> TieredMergePolicy first computes the allowed “budget” of how many segments should be in the index, by counting how many steps the “perfect logarithmic staircase” would require given total index size, minimum segment size (floored), mergeAtOnce, and a new configuration maxSegmentsPerTier that lets you set the allowed width (number of segments) of each stair in the staircase. This is nice because it decouples how many segments to merge at a time from how wide the staircase can be.

![Lucene segment merges](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/7/74fe0bee8ceb65088517fdef396829b10a9bb299.gif)
