What is the index rotate logic? how to clean otel-* index

Data Prepper
,
1

I am new to opensearch trace analytic data, currently, I deployed my data-prepper pod in my eks, and i can see the index sent by data-prepper in my AWS Opensearch.

what I noticed is that instead of date format suffix, its index-pattern is based on sequence , such as otel-v1-apm-span-000008，otel-v1-apm-span-000007… But I don’t know the rotate logic between those indexes e.g. when data-prepper will rotate index from otel-v1-apm-span-000007 to otel-v1-apm-span-000008, I have spent 2 hours on digging the rotating logic, but no luck. As i need to know how and when to clean those indexes base on the rotating logic, it’s very important.

So, it’s very appreciated that who could kindly provide some help, thanks in advance

2

Hi @wwwlll2001,

The OpenSearch Sink reserves otel-v1-apm-span-* as index pattern for Raw Span Trace Analytics.
A new index is rolled out only in case of “min_size”: “50gb” or “min_index_age”: “24h”.

Here’s the link to the raw span policy: data-prepper/raw-span-policy-with-ism-template.json at main · opensearch-project/data-prepper · GitHub

Hope this helps.

3

Thank you for your kindly input, I will check it , and i remember that one index could span 3 days which means maybe the duration limitation is not 24h, I will double confirm it.

Thanks again.