I have created AWS Managed Opensearch 2.11 and ingesting my application logs to this opensearch domain through logstash.
I have one index per one log file and ingestion works great.
But I am quite confused on how to roll the index over. Initially, I thought of setting up a policy that can delete old docs when index age passed certain threshold. but through policy, it’s not possible. I also tried giving ILM settings in my conf file of logstash, but figured that ILM settings won’t work for opensearch output.
Any suggestions on clearing old documents when index age reach certain threshold ? or any way to roll over indexes in opensearch that are ingested through logstash ?
I couldn’t find direct ways to rollover logs ingested through LogStash.
Instead, used ISM to set policy for all matching index templates that are created monthly : to move them to read-only then transition to delete after n number of days.
This way, new indexes will be created every month and data will be written to these new indexes ignoring the old indexes, and the these old indexes will be moved to read only and get deleted as per threshold.
@gaobinlong , I have scheduled a weekly report generation, I can see and download the reports through opensearch UI. Any suggestion on how to download them through API calls/curl commands.
