The index management feature is available in ES Basic version as well as in OpenDistro with OSS .
Is there a benefit for users to use OD’s index management over Basic?
Do we have a detailed documentation for the features available in OD’s Index State Management(ISM)?
Redirecting… is the only doc I could find for ISM.
Hi @dinesh,
One key difference is the ILM feature in ES Basic version is not Apache 2.0 while the Index Management feature in ODFE is. From a functional stand point, the Index Management plugin in ODFE allows more control since the policy is configured by the user allowing things such as cycles between states. We’ll work on explaining the benefits more clearly in the documentation. But, if you notice anything missing functionality wise that you would like to see, just let us know.
Thanks
Thanks for replying @dbbaughe,
it will be really helpful to have explanations of the benefits more clearly in the documentation. Thank you for considering that.
Regarding your point below:
From a functional stand point, the Index Management plugin in ODFE allows more control since the policy is configured by the user allowing things such as cycles between states.
I believe Basic version of ES also does support this (where from kibana we can create the policies using a template and then create the index using this policy and template). Please correct me or add some more details if I am wrong.
Also can we use OD with Basic version to get the best of both worlds? Are there any challenges/issues using ES Basic+OD plugins ?
Hello @dinesh,
I’m referring to the point that a user can create any number of states with their own choice of actions inside of each of those states and they can loop indefinitely between states if their use case requires. Admittedly I haven’t followed the ILM development so if it can also do this that’s great, but from previous articles it seems like it was forced into a hot → warm → cold → delete type workflow and restricted which actions can be executed in which state.
Regardless, we’d love to know what further features you’d want out of the ODFE Index Management so we can work on them!
As for the Basic + OD plugins, that might work and probably does for quite a few of our plugins, but not something we explicitly support so if issues do occur you would possibly be reliant on the support of the community.
Hello @dbbaughe
It will really help to have few features of index, snapshot & restore
- Index lifecycle management
- snapshot life cycle management
- provision to have HOT, WARM, COLD and finally a delete option to better manage indexes suitable for various situations
- Taking snapshot & restoring them in same or different clusters
Hi @learntech07,
Could you clarify what you mean by needing index lifecycle management?
Our alternative to ILM is ISM which is available right now: Redirecting…
This does have a delete action, and the allocation/snapshot are in PR and will require a bit more work.
great this answers my question.
Are the below features also planned to release in future versions?
Hi @learntech07,
Snapshot Lifecycle Management is currently not planned. You could create a feature request on our repository for us to track requests for it though.
Can you explain a bit more about provision to have HOT, WARM, COLD ...? Do you mean being able to allocate indices to different nodes? If so that is currently in PR. The delete action is already supported. You can delete indices after certain conditions like size, age, document count, etc.
Snapshot is currently not supported, but also in PR, will require a bit more work though. The first iteration is unlikely to allow restoring though. Can you explain a bit more about your use case for automating restoring snapshots? We mostly see users wanting to take a snapshot of their data either on an automated scheduled basis or right before deleting the index.
Hi @dbbaughe
Thanks for your clarification and quick response
Can you share how to raise for feature requests?
two use cases I mean
- categorizing of log indexed data as per requirements in terms of HOT for few weeks or months or days, WARM again goes same and COLD again goes same way - this will help keep only highly needed index in HOT category etc…
- If i want to restore same category indexed data along with any of the hidden indexes at destination remote cluster then it should also have similar bucketing at destination remote cluster - this will help to have very similar index structure & its category a like primary cluster
Is see these are already available @ Take and Restore Snapshots - Open Distro Documentation. Have not try this, but it only has snapshot and do not see any repeated snapshot options in docs at least - like every day or week or hour or based on index type rather than time can also be an addition etc…
Till we have a cross replication or a DR type solution released, a feature which has automatic restore function activated by providing required remote cluster details along with its Auth/Authz permissions in its settings that can restore index in an automated way for every xyz time based on the snapshots taken at source/primary cluster. Hope I was able to explain my request.
Hi @learntech07,
If you have a feature request for a specific plugin you can make it directly on the plugin repository:
And if you have a more general feature request you can make it on the community repository:
That snapshot link is about the general snapshot APIs available. Index Management is about the usage of Index Management related features such as Index State Management where you can automate the management of indices. ISM will eventually support the usage of the snapshot APIs in an automated fashion.
And still a bit confused about your automated snapshot restore. ISM is used to fully automate management of indices. So this would be the equivalent of you knowing ahead of time when and what you want to restore. Is that possible in your use case? Or are you saying in your second cluster you want to automate the restore of all indices that are pushed to the repository from your original cluster to always have a backup ready to switch to?
Yep this is what I expect. Adding an AI layer on top of auto index restore of all indexes along with its dependencies from source/original cluster to backup cluster to be in a ready state to switch