# Managing Index Setups & Configuration in Microservices Environment **URL:** \ **Category:** DevOps\ **Tags:** discuss\ **Created:** [March 3, 2022, 3:24pm UTC](https://forum.opensearch.org/t/managing-index-setups-configuration-in-microservices-environment/8815 "2022-03-03T15:24:58Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ralph](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/ralph/32/1194_2.png) [@ralph](https://forum.opensearch.org/u/ralph)\ **Post date:** [March 3, 2022, 3:24pm UTC](https://forum.opensearch.org/t/managing-index-setups-configuration-in-microservices-environment/8815/1 "2022-03-03T15:24:58Z")
**Background** We have a bit of a complicated setup, so let me introduce that one first: - everything runs on kubernetes - we have a bunch of [self contained systems (SCS)](https://scs-architecture.org/). - [for the sake of simplicity](https://xkcd.com/2587/) we’ll focus on only one in this discussion here as it just means that we’ll do the same for all of them - if an SCS has a use-case where they need opensearch they’ll have an opensearch cluster (i.e. different SCS don’t share a common opensearch cluster to avoid coupling) - functionality in the SCS is built as microservices - these microservices are scalable, so it is well possible that more than a single pod of the service is running - our applications are not run by the teams building them (of which i’m a part) but instead by other companies (i.e. there are dozens of deployments out there, with varying versions of different services in use) and accordingly we also have 0 access to the production system - downtimes for maintenance windows should be avoided if ever possible, in some future use-cases they might be unacceptable it is thus imperative that all actions are fully automated and fail-safe (i.e. re-runable). no human interaction must be needed for anything but unrecoverable errors (which in turn shouldn’t happen). **Setup** - the opensearch cluster being deployed in an SCS starts out with a minimal configuration needed just to get it online - some of the minimal setup is customer/landscape specific (e.g. authentication realms) - the rest of the setup is use-case specific: - setup of indices - setup of roles and role mappings for these indices - the actual data is fed in asynchronously once the setup is done (and there’s a constant flow of data afterwards) in a naïve implementation the use-case specific setup can be handled by the data ingestion system by just doing it whenever it starts. but that means that if a new pod starts (e.g. scaling from 1 to 2 pods) it’ll try to do it again. a slightly more advanced system can build a setup-management system around this and version the changes, storing the version information in opensearch so that it only applies the change if it hasn’t been done before (think “liquibase for opensearch”). however, this still leaves the issue if multiple pods start at the same time (in general they’re a `Deployment` and not a `StatefulSet`, i.e. all will start together and not one after the other). we also considered having a single configuration service (which runs as a singleton per SCS) through which we could handle the config updates (i.e. the other systems call this one with the required config changes) so that it could queue the config updates and ensure that they are run sequentially. note that we can’t package the configuration with opensearch itself as we don’t know which use-cases will end up running against it (it must be possible to update a use-case specific component/service without updating opensearch and vice-versa). we do have a versioning & dependency management system in place to ensure that if an update of a component requires a newer version of another component this gets pulled in and deployed as well. **Question** i somehow presume that we’re not the first ones facing this issue - how have others solved this? what are your recommendations? thanks a lot for your feedback! ---
**Author:** ![ralph](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/ralph/32/1194_2.png) [@ralph](https://forum.opensearch.org/u/ralph)\ **Post date:** [August 30, 2022, 6:32am UTC](https://forum.opensearch.org/t/managing-index-setups-configuration-in-microservices-environment/8815/2 "2022-08-30T06:32:43Z")
it’s been a long time that i’ve posted this question. it still stands - is there any chance that somebody might have some feedback here? ---
**Author:** ![nateynate](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/nateynate/32/4913_2.png) [@nateynate](https://forum.opensearch.org/u/nateynate)\ **Post date:** [October 25, 2022, 5:55pm UTC](https://forum.opensearch.org/t/managing-index-setups-configuration-in-microservices-environment/8815/3 "2022-10-25T17:55:38Z")
Hey @ralph ! I wanted to thank you for joining our community meeting today and speaking up about your issue here. I’ve sent up a flare to some of the developers on the project hoping to get some suggestions. To me it sounds like this would be an awesome extensibility option - ‘index version management’ or some kind of ‘index migration’ plugin where before/after mappings can be defined, very much like a Ruby on Rails migration. The cluster would perform some kidn of self check at regular intervals and perform these migrations on any indices that need to ‘roll forward’ so to speak. I personally think it’s an awesome idea. We should file an issue on this if there’s no best practice or some kind of index state management option that could be used. Nate ---
**Author:** ![ralph](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/ralph/32/1194_2.png) [@ralph](https://forum.opensearch.org/u/ralph)\ **Post date:** [October 31, 2022, 4:43pm UTC](https://forum.opensearch.org/t/managing-index-setups-configuration-in-microservices-environment/8815/4 "2022-10-31T16:43:14Z")
as discussed in last weeks meeting i’ve now raised the following issue: > > > \*\*Is your feature request related to a problem? Please describe.\*\* > when deployi…ng OpenSearch as part of a larger application fleet in an environment (in our case: kubernetes) where any installation/update must be 100% hands-off (i.e. fully automated) and esp. when the connected applications are microservices (i.e. lots of them, various versions of the same in parallel due to canary upgrades or just in general rolling upgrades) it's very hard to actually set up the proper index structures & general settings on OpenSearch: > \* this cannot be done with the deployment of OpenSearch itself as it doesn't know anything about its consumers (it can only ship basic configuration like TLS, authentication, etc. but doesn't know about any indices, specific roles, etc.) > \* if the consumer application takes care of an update there's an issue if multiple replicas of the same application are running in parallel - they might all try to do the setup/update and then either block or break each other > \* this is both about managing indices as well as related settings (e.g. roles and role mappings) > \* 24/7 operations must be supported, i.e. scaling down everything, doing the upgrade, applying the config updates and then starting back up is not an option > > \*\*Describe the solution you'd like\*\* > there should be a way for consumer applications to manage opensearch indices in a similar way as can be done with \[liquibase\](https://www.liquibase.org/) for RDBMS (SQL-based relational DBs). there it's possible to define upgrade scripts and liquibase then keeps track of what has already been applied and what hasn't (by storing that information in dedicated table(s) on the DB). this can be used both for DDL (data definition language; e.g. changing tables) as well as DML (data manipulation language; e.g. migrating data) and any mixture of the two (e.g. changing an existing table schema and migrating the data in the process). > > \*\*Describe alternatives you've considered\*\* > \* manually triggering any action is not an option as the installation of any component (or upgrade of any component) - once started - must happen 100% automated > \* running the updates from outside (e.g. by the installation process) is not an option as it doesn't know when exactly which version of which application is booting up (this is handled by kubernetes) > > \*\*Additional context\*\* > \* more information can be found \[in this forum post\](https://forum.opensearch.org/t/managing-index-setups-configuration-in-microservices-environment/8815) > \* this had been discussed in the open round part of the \[community meeting on 25.10.2022\](https://forum.opensearch.org/t/opensearch-community-meeting-2022-1025/10949/2) > > > note: while this ticket has now been opened in the main OpenSearch repository i'm not sure whether the actual solution for this will be part of this repository. i could well imagine that the solution would be a dedicated application or an OpenSearch plugin. ---
**Author:** ![ralph](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/ralph/32/1194_2.png) [@ralph](https://forum.opensearch.org/u/ralph)\ **Post date:** [November 9, 2022, 5:01pm UTC](https://forum.opensearch.org/t/managing-index-setups-configuration-in-microservices-environment/8815/5 "2022-11-09T17:01:37Z")
as a small update: i’ve now come up with a solution proposal and am looking for feedback on it, please see this comment on the ticket: > > > \*\*Is your feature request related to a problem? Please describe.\*\* > when deployi…ng OpenSearch as part of a larger application fleet in an environment (in our case: kubernetes) where any installation/update must be 100% hands-off (i.e. fully automated) and esp. when the connected applications are microservices (i.e. lots of them, various versions of the same in parallel due to canary upgrades or just in general rolling upgrades) it's very hard to actually set up the proper index structures & general settings on OpenSearch: > \* this cannot be done with the deployment of OpenSearch itself as it doesn't know anything about its consumers (it can only ship basic configuration like TLS, authentication, etc. but doesn't know about any indices, specific roles, etc.) > \* if the consumer application takes care of an update there's an issue if multiple replicas of the same application are running in parallel - they might all try to do the setup/update and then either block or break each other > \* this is both about managing indices as well as related settings (e.g. roles and role mappings) > \* 24/7 operations must be supported, i.e. scaling down everything, doing the upgrade, applying the config updates and then starting back up is not an option > > \*\*Describe the solution you'd like\*\* > there should be a way for consumer applications to manage opensearch indices in a similar way as can be done with \[liquibase\](https://www.liquibase.org/) for RDBMS (SQL-based relational DBs). there it's possible to define upgrade scripts and liquibase then keeps track of what has already been applied and what hasn't (by storing that information in dedicated table(s) on the DB). this can be used both for DDL (data definition language; e.g. changing tables) as well as DML (data manipulation language; e.g. migrating data) and any mixture of the two (e.g. changing an existing table schema and migrating the data in the process). > > \*\*Describe alternatives you've considered\*\* > \* manually triggering any action is not an option as the installation of any component (or upgrade of any component) - once started - must happen 100% automated > \* running the updates from outside (e.g. by the installation process) is not an option as it doesn't know when exactly which version of which application is booting up (this is handled by kubernetes) > > \*\*Additional context\*\* > \* more information can be found \[in this forum post\](https://forum.opensearch.org/t/managing-index-setups-configuration-in-microservices-environment/8815) > \* this had been discussed in the open round part of the \[community meeting on 25.10.2022\](https://forum.opensearch.org/t/opensearch-community-meeting-2022-1025/10949/2) > > > note: while this ticket has now been opened in the main OpenSearch repository i'm not sure whether the actual solution for this will be part of this repository. i could well imagine that the solution would be a dedicated application or an OpenSearch plugin.