Our indexes look like “entity-yyyy-mm” where yyyy is year and mm is month. The application implements multi tenancy, so every document has a field tenantId. Tenant is not a part of index name, so many tenants share the same index.
I need to implement a retention policy for tenants data: every tenant should be able to define how long we keep its data (in days).
I was looking into Index State Management, but it works on index level and will not allow to drop only a subset of documents. I need a retention policy that would work on a document level.
Is there anything that I can use to implement this?
If you co-locate tenants in the same index, you can’t remove a particular index from the series unless all tenants have only expired data in that index. Which effectively means that the tenant with the longest retention dictates when the index is deleted.
You could delete a tenant’s data from within an index using Delete By Query, but I don’t think you can do that with ISM. Plus, it’s much more expensive to delete by query than to delete an index. But if tenants have wildly different retention policies, it might be worth it.
