New to OpenSearch Management; ISM Hot/Cold Template + Graylog

[04_996_C2]

Opensearch Version 2.14

Greetings:

I run Graylog with Opensearch handling the indexing. Graylog handles the administration of opensearch but, since I use the Graylog Open version, I not have an archiving feature.

Currently my setup has Graylog utilizing a Time Size Optimizing rotation strategy. Essentially, Graylog handles the shards for an index and rotates between 30 - 40 days age depending on whats best for shard distribution. When it is rotated, the index is closed.

I would like to use the ISM plugin to handle index management once Graylog has rotated it out. Basically, rather than have Graylog close the index, have it rotate it out and do nothing further with that index.

It is at this point I would want ISM to take over. I would want a policy that will look for any index that is 41 days or older (so as not to interfere with Graylog’s management of indices aged 30 - 40 days), move it to cold storage, and then close it. Also, if it were possible, I would want Opensearch to delete the associated replica shards.

Optimally, this ISM would also force all new indices to be built on/in the Hot nodes.

My stack is as follows:
3 x Management Only OS Nodes LXC
4 x Data Nodes (3 Hot, 1 Cold) LXC

Graylog Components:
1 x Mongodb LXC
1 x Graylog Server LXC

Thank you!

[Gsmitt]

Hey @04_996_C2

I’m not sure it will work with Graylog, you could try using an API but since Graylog control the index management it might not work.

[04_996_C2]

I’ve used the Snapshot plugin in Opensearch Dashboard to successfully create a snapshot policy for the indices in Graylog but I suppose this is a bit more complex.

I am just growing a bit concerned with the direction in which Graylog Open is going. With 6.0, index retention defaults to “delete” with having to select a deprecated retention strategy if you want anything else.