Upgrade path to OpenSearch

OpenDistro
1

I realize we don’t have a thread on the upgrade path to OpenSearch. For context, I am looking into various ODFE plugins and their potential upgrade path. What I’d like to hear about on this thread is what’s important to you for migration/upgrade?

We have ElasticSearch + Kibana (aka ES), Open Distro for Elasticsearch (OSS Elasticsearch+Kibana+OSS plugins, aka ODFE) migrating to OpenSearch + OpenSearch Dashboards + OpenSeach plugins (aka OpenSearch).

Some questions I can think of:

  • Should OpenSearch be 100% backwards compatible with ES?
  • Is it important to you that OpenSearch be a drop-in replacement for ES? For ODFE?
  • For completeness, would you be ok if there was no upgrade path at all, because maybe you only plan to do new deployments with OpenSearch?

Your thoughts and additional questions would be much appreciated.

Thanks,
-dB.

4 Likes
2

We use only Elasticsearch itself, no plugins (other than discovery-ec2 plugin) or dashboards, but we run tens of ES clusters so it’s quite important for us to be able to upgrade seamlessly to OpenSearch. We are currently running ES 6.8 and are testing ES 7.9 but we most likely skip directly to OpenSearch if the upgrade is reasonably cheap - ideally for free.

So to your questions:

  • 100% backward compatibility? - Yes please.
  • Drop in replacement? - Yes, we need it to be able to read ES 6 or 7 data structures. Pointing it to ES 6/7 data directory should just work.
  • No upgrade path at all? - Not in our case. We would have to reindex TBs of data during upgrade which is just not feasible.

But I think it’s OK if OpenSearch diverges in later versions and the upgrade path would be only ES 7 → OpenSearch 1.0 → OpenSearch 2.0 → OpenSearch X.0. The important thing is to allow current ES users to have painless upgrade process and once they are on OpenSearch you are in much better position to control upgrades to future versions.

4 Likes
3

Same as what Katulus said would be true for us at Sematext, too.

Otis

1 Like
4

At VMware, we’ve embedded Elasticsearch and Kibana into our products, and are considering replacing them with ES, so in answer to your questions … we need backwards compatibility, a drop-in replacement, and upgrade path. Our interest is mostly around ES, rather than the plugins.

1 Like
5

we have the same use-case, i.e. we (plan to) ship Elasticsearch and want to use OpenSearch as a drop-in replacement. we do need a security plugin and for a few use-cases maybe 1-2 additional plugins where we haven’t checked yet what their plans are regarding OpenSearch.
since we’re not running it ourselves in most cases it needs to be a smooth upgrade/replacement (we are running on kubernetes/openshift). obviously the nicest thing would be if i could do an online-upgrade of an Elasticsearch 7.10.x cluster to OpenSearch 1.0.0, but i’d also be ok with having a short down-time if the cluster wouldn’t be happy about that). picking up the existing data-files would be very helpful as it’d allow an easy transition without any index migrations / re-indexing.

6

I’d like to add a little to what @dblock is asking - for those of you needing seamless upgrades: what other tooling do you use with your current ES clusters or embedded environment?

Logstash? Beats? Fluentbit? Custom software?

7

custom software (ours & 3rd party) in our case (both for read & write).
note that most of them will be using spring-data-elasticsearch, so this must stay compatible. see also Spring Data for ElasticSearch support / collaboration · Issue #542 · opensearch-project/OpenSearch · GitHub for this.

1 Like
8

We use:

  • ODFE embedded into our products
  • ODFE “standalone” (usually installed in docker / Kubernetes)
  • Amazon Elasticsearch Service

We really want to migrate to OpenSearch once ready, but we cannot reindex all the data, so the index migration must be seamless. We can accept a minimum downtime.
We also need that OpenSearch/ODFE and Amazon Elasticsearch Service remain aligned, because we have the same products running on different clouds (using AES on AWS and ODFE elsewhere)

About the tools that we use:

1 Like
9

In Aiven we support:
OS ES + Kibana and additional open source plugins
Besides that we use our own plugins which support ZSTD compression and client-side encryption for S3, GCS and Azure

Thanks,
Andrey

1 Like
10

I’ve opened OpenSearch#638 to help us work backwards from the assumption of maximum and most seamless drop-in replacement. We have a dozen ODFE plugins to test out various strategies and I know different developers have different ideas about how we could accomplish this technically - let’s start putting them into there. Would love to hear from some plugin authors that are considering supporting OpenSearch, outside of ODFE, too.

11

Our most important requirement for upgrade is no outage. There should be an upgrade path allowing for upgrade without any outage, preferably while keeping the cluster in read/write mode. Read-only upgrade would be considered to be a last resort, but still feasible.

2 Likes
12

Thanks everyone for your feedback. It’s pretty clear that customers want a rolling upgrade without downtime. I opened Support rolling upgrades to OpenSearch 1.0 from ODFE 1.13 and ES 7.10.x · Issue #671 · opensearch-project/OpenSearch · GitHub and Support restart upgrades to OpenSearch Dashboards 1.0 from Kibana v7.10.x · Issue #334 · opensearch-project/OpenSearch-Dashboards · GitHub to track related work-items.

2 Likes