Kubernetes operator support for the fork

I tried to get some internal interest in building something. So far the best one I have seen is this one from Zalando. I could reach out and see if they are willing to collaborate on that, but I think we need a stable release before we can think about that.

2 Likes

if you could do that (i presume you have some contacts there?), that’d be great!

i’m not so sure about that: i would presume that - for now at least - the operations aspects of opensearch will stay as they are (and as they were with elasticsearch) - of course with “elasticsearch” renamed to “opensearch” in all the config options & files. since the renaming is already done i’d presume that an operator which gets adapted to work with opensearch now would still work with opensearch 1.0.0?

and i think adoption rate of opensearch would be higher if there’s an operator - maybe i’m wrong, but i’d presume that nowadays most “professional” setups are run on a k8s cluster and there an operator makes live a lot easier. so companies (i presume few here just run their own k8s cluster on their private NAS just for fun? i know that i don’t :slight_smile:) probably used ECK so far and now won’t be able to move to elasticsearch until a new operator is available since ECK won’t work with opensearch. those using managed services obviously are not affected by this (and those running the manged service have their own operator - where i was maybe hoping that somebody might decide to share theirs here).

@jkowall: do you already have specific plans on how you’ll be running opensearch at logz.io? i presume now with elasticsearch you’re either using ECK or possibly ECE (i don’t know your exact setup?

do we have anyone from redhat here in the forum? since you publicly voiced support for opensearch i presume you’ll integrate it in an upcoming release of openshift? if so, do you already have a plan on how you’ll be operating it there? will you continue to use your existing operator (modified for opensearch) with a pure focus on your logging stack or would you be interested in moving to a generic operator which works for all use-cases? (added advantage: anyone running openshift doesn’t need to deal with deploying / upgrading the operator - it’s already packaged by you and we can just set up the custom resources to get the opensearch clusters up & running :slight_smile:)

1 Like

I had contacts there, but they are outdated now. I need to make a new contact. I will ask them.

Today we run ElasticSearch (soon OpenSearch) on EC2 instances, similarly, we run Kafka on but I think in the future we want to move both OpenSearch and Kafka to K8S. We run most of our other services on K8S already today. I doubt we would migrate to OpenSearch and to K8S at once, as this would be a lot of risk for the size of clusters we are running.

There are folks from RedHat I am in touch with. They are using the RedHat Operator for K8S, but they don’t run large customers on OpenShift. I can email them if you would like @ralph. They are also not sure if they want to use OpenSearch longer term as it’s quite heavy and complex to run, but there is not a good alternative right now. They are investigating things like ClickHouse, but of course you’ll give up Kibana with that approach.

Hey Ralph - Amazon Elasticsearch Service for what it is worth manages Elasticsearch directly on EC2. But I am hoping there will be a lot folks with experience using K8S in production that want to work on operators for OpenSearch and OpenSearch Dashboards. That would be great.

3 Likes

just to point it out here: @jkowall has created a ticket with zalando to check if they’re interested in contributing their operator to OpenSearch:

also worth mentioning is the fact that their operator is currently under the MIT license. i’m not sure if this’d be an issue if it were contributed to the project or if they’d have to re-license it under the Apache 2.0 license to avoid license-mixture under the OpenSearch brand?

1 Like

Maybe it would make sense to base an official OpenSearch operator on ECK (the official Elasticsearch and Kibana operator made by Elastic) (GitHub - elastic/cloud-on-k8s: Elastic Cloud on Kubernetes). I gave it a spin last weekend just with a trivial test on my local machine and it worked great. It was able to scale up and down one data node at a time while keeping cluster health from going red.

The most recent version of it doesn’t have an open source license. But perhaps an older version of it was published with an open source license that could be forked? Or, its design could be used as inspiration?

EDIT:

Oops, didn’t see the original message here:

using ECK does not seem to be an option (both from a license point of view and also from a technical point of view, see: Does elastic eck operators support open distro for Elasticsearch )

Sounds like it was never published with an open source license. But what makes it unusable from a technical perspective? It uses StatefulSets right now. Is that problematic?

1 Like

I think an ES operator needs to use statefulSet. But the ES one is not open source. I.e. a dead-end.

1 Like

I have been recently working on K8s. I can pick this up but will be needing an additional hand from the community as I don’t have in depth understanding about OpenSearch.

1 Like

@TheAlgo That’s great news. What kind of help specifically do you need?

I am currently working on creating a Helm Chart for OpenSearch. This is helping me understand it better. Once the PR is merged I can start on the operator.

2 Likes

Are you all looking for a Helm based operator or anything specific? I have just completed creation of basic Helm Charts for OpenSearch and OpenSearch Dashboards.
@searchymcsearchface @jkowall

I know the RedHat team was asking about a Helm Chart. Personally, I think the operator would be more useful. I really like the Zalando one. I hope we adopt OpenSearch on k8s in the future.

Here is a tracking issue from May: Interest in contributing to OpenSearch · Issue #169 · zalando-incubator/es-operator (github.com)

1 Like

I have already created basic Helm charts for deploying official docker images of OpenSearch and OpenSearch Dashboards[PR is in review]. I will try going through how Zelando have built it and try doing a POC for OpenSearch to kickstart.

we’re definitely looking for an operator.

we’re not using helm charts for various reasons and have built our own k8s deployment descriptors for now as a quick-win (sorry, nothing i can easily share since it contains various company-internal things as well for our specific way of rolling things out).

we’d want an operator to have all the nice things which it usually brings you: properly managing rolling upgrades (as pointed out in How To: Upgrade from Open Distro to OpenSearch · OpenSearch you e.g. need to set some cluster settings before starting it & at the end), auto-scaling, etc. and it’d just in general make our life a lot easier if we can just create a custom resource and then get a full cluster deployed.

i’m still hoping that Zalando can be convinced to move their license to Apache 2.0 so that either their operator can be forked for OpenSearch or that they might also move to OpenSearch themselves and thus the operator itself is directly moved to support OpenSearch.

building a complete new operator from scratch is a lot more effort than re-using their existing, battle-proven operator. and “just having a look” at what they’re doing is a bit of a gray-zone from a legal point of view (different license, so copying would be re-licensing without their approval and also w/o attribution). but i’m no lawyer and hate all things licenses (well, actually by now all things lawyer-related :smiley:), just wanted to point out that there’s also a legal component to the whole thing.

Just curious, can I know the reasons why you are not using helm chart for deployments?

we don’t have a classical devops setup where we build something and have one production environment - we’re running software in various setups (ranging from SaaS (with multiple production environments (for legal reasons)) to on-prem - with more than 100 production environments, most of which are not managed by us), in various different versions (you won’t get an on-prem customer to upgrade every 5min even if you’d release that often :slightly_smiling_face:) and various setups (not every customer has the same set of features and thus not the same set of applications) and various different configurations. furthermore, our software product offering is quite vast (we’re not just talking about 10-20 microservices here).
e.g. we have / will potentially have multiple opensearch clusters running in the same k8s namespace, depending on the use-cases - but they should of course still share most config-related things (i.e. i don’t want to build up the k8s resources manually for each of them - i just want the name to be different).

helm charts were not offering this capability when we started on this journey with k8s (quite a while ago) - they might be now (to be honest i’m not sure, haven’t looked at them in detail) but we’re quite heavily invested in our current approach. so while we might switch at some point i’d probably even then still not be able to use standard opensearch helm charts.

2 Likes

I get your use-case, I was just thinking you can leverage Helm Charts itself for configuring different types of clusters in your production environments but the issue lies on sharing config related things which I guess might not be possible with a Helm Chart as it scaffolds a new release altogether once you install a chart.

Helm charts only help with provisioning, which is just part of the lifecycle. An Operator can do a lot more. In the case of the Zalando operator you can scale based on metrics or other data. The operator can configure cross cluster and how the cluster is setup. The operators also does rolling restarts.

Many other operators do things like backups, data management, upgrades, and chaos/failure testing.

2 Likes

This isn’t strictly true. Helm charts can manage post provisioning lifecycle events using chart hooks.

I think the OpenSearch community would benefit from having a helm chart as well as an operator. It can take quite a bit of detailed operational experience to implement a complex operator and often these details are learned over time. Starting with a helm chart would make a good stop gap and inform the development of a more complex operator.

1 Like

Added a tracking issue in the repo, and hope the community can help resolve: [Help Needed] OpenSearch/Dashboards in Kubernetes Operator · Issue #13 · opensearch-project/opensearch-devops · GitHub

3 Likes