Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
I’m using AWS OpenSearch (ie OpenSearch 2.3 (latest), R20221114-P1 (latest) from AWS Console)
Describe the issue:
I’m migrating from Elastic Cloud to AWS Opensearch, mostly using the reindex API … it’s fine, slow, coming from ES ver 8+
I’m trying to setup ISM … I believe I’m getting confused / don’t understand / getting an error.
I want an ISM policy for my application logs. All app logs go to logstash instances, and from there to Opensearch (I have already put in the correct OSS version of logstash, that supports Opensearch) … my output is:-
opensearch {
hosts => ["${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
user => "${ES_USERNAME}"
password => "${ES_PASSWORD}"
index => "logs-%{+YYYY.MM.dd}"
ssl => true
ssl_certificate_verification => false
action => "create"
}
I am using Opensearch Data Streams for my logs, which I assume is best practice?
I would like an ISM policy that keeps logs:-
- HOT: 3 days
- WARM: 30 days
- COLD: forever
(or other suggestions for best practice for logging policy / cost effectiveness)
I’m confused about when/where to do a rollover (if at all)
Perhaps I should not be creating an index-per-day at all, and just have an index called “logs”
Any advice or samples would be appreciated?