+1 to move 3.0 to 2023. Since Segment Replication and Remote Index can be delivered in minor versions, this also will give more time for developers to leverage the new inclusive usages before removing deprecated usages.
2 Likes
I can definitely see both perspectives, but I lean towards a longer time between majors (option 2). This will help users have more time to ramp onto 2.0 and not have to think through potential breaking changes across two major versions. And +1 to @nknize’s point on not feeling locked on delivering new features. If there are ways to make things additive, but not change the default, we can minimize breaking changes while still helping users adopt new capabilities.
One thing to consider is that OpenSearch Dashboards will need a 3.0 release before April 30th, 2023 in order to upgrade Node.js from v14 to a newer version (v16 or v18) before the end of LTS. Releases | Node.js
2 Likes
I bet we can decouple OpenSearch from OpenSearch Dashboards releases before April 2023. 
2 Likes
I think not considering the elastic compatibility and client issues I link to is a big miss if we don’t consider a 3.0 release sooner than later. Maintainers will be the ones pulling the magic to keep the house of cards standing and users will be the ones continuing to endure pain until they decide to either wait for 3.0 or bail on the project completely. I urge not to ignore the implementation details here in favor of the simple idea of not having to release another major until next year. Some short term pain just might alleviate long term chronic pain.
Maybe we should put this discussion on GitHub so other maintainers don’t miss out on weighing in before a decision is made without their input?
Another thing to consider. If we release 3.0 in September with Lucene 9.x, the REST Version API, and Elastic 7.x version removal, users can perform a direct 1.x to 3.0 rolling upgrade without having to go to 2.0 first; and client compatibility (including beats and Logstash fixed version compatibility) will be fixed with full bwc support. This means a one time client upgrade, instead of upgrading to a wonky 2.0 first, and then upgrading to a 3.0.
If I’m an end user going through the client compatibility issues we’re seeing today I’d love the option of directly upgrading to 3.0 instead of being forced to upgrade my clients to 2.0 and then again to 3.0 because we waited until Lucene 10.
2 Likes
I would prefer we don’t bake too many exciting new features for too long, and thus would prefer an earlier schedule (version 1). My biggest concern is that by extending the schedule we are just deferring assembling 3.0 till later, instead of getting serious about it in parallel with 2.x. The time to care about 3.0 is now regardless of when it ships.
I am 100% with @nknize on needing to get to a state where we are not constantly introducing breaking changes. We were pretty good with bringing features into 1.x that didn’t break backwards compatibility, and we should continue going the extra mile while building features in a backward-compatible way. Our goal is to get to a state where we can go years without a major release, all while adding a ton of value continuously.
Personally, I was leaning towards 2nd option before we brought back spoof hacks for compatibility with Elasticsearch clients . Essentially, we still cannot break ties from Elasticsearch as @nknize mentioned, sooner we do that - better it is going to be for both projects and communities. In these regards, 1st option is looking like a better path forward now.
1 Like
+1 to all your comments, especially moving the Remote Index work to 2.x
I am concerned with option 1. It seems like we are doing another major version and introducing breaking changes earlier than we need to for the sake of breaking. I agree with @dblock that we should be assembling 3.0 in parallel with 2.x, but I don’t think we need to push to launch 3.0 early if we have ways to continue to deliver new functionality in 2.x without introducing breaks. That’s why I’m still leaning with option 2. I think we are asking a lot of end users to plan for multiple major version jumps in a ~1.5 year period if they want the latest features. (e.g., 7.10 → OpenSearch 1.x → OpenSearch 2.x → OpenSearch 3.x).
2 Likes
If we release 3.0 in September with Lucene 9.x, the REST Version API, and Elastic 7.x version removal, users can perform a direct 1.x to 3.0 rolling upgrade without having to go to 2.0 first; and client compatibility (including beats and Logstash fixed version compatibility) will be fixed with full bwc support.
That is very useful context! Thanks @nknize! Do you know what the planned breaking changes that would push the major to version 3?
The removal of backwards compatibility logic with Elasticsearch and spoof hacks for compatibility with Elasticsearch clients.
1 Like
Thx! Makes sense. Would the API versioning then handle backwards compatibility problem with Logstash/Beats?
Yes, exactly…
If moving to 3.0 sooner will allow us to scale client support, reduce versioning headaches, and make it easy to move straight from 1.x to 3.x, then I am for option 1 as that is a win for the code and community.
Hey Nick;
Here are the things that are confusing me:
It seems like the primary argument is that moving to quickly to 3.0 makes it easier for for folks to upgrade to 3.0. But if we’re continuing to release on the 2.x line, I can’t see the benefit of doing that earlier?
As for clients, I think the horses are already out of the barn – We’re the ones who did the deprecation which broke them most recently-- so we’re also in control of not breaking them again (for the most part). For clients not controlled by us, AFAIK, we’ve pinned to a particular version (example: 7.17.1 for logstash here), so they should not be a source of breaking changes.
I agree that REST version API is definitely the way to go, but AFAIK, it’s not actively being worked on (it’s also not a breaking change, so we could add it as part of the 2.x line). I could absolutely be wrong about that, but I’m just not seeing anything happening right now. So I couldn’t promise anything would be ready for an August code freeze (unless you’re working on it?)
Finally, while I personally can see the benefit of having a clean break from Elastic, I don’t think that’s a priority for folks using the software. I’d hate to release a major version and cause disruption for something that doesn’t directly solve a problem for someone using the software. To ask another way: I understand the maintainer pain, but what is the pain for the end users on 2.x that would be removed if we moved to 3.x?
Thanks,
/C
1 Like
“The sooner we can migrate users away from these hacky integrations the better the OpenSearch experience will be; and this will be a breaking change that will require a major release.”
“Easier to upgrade to 3.0” is just a positive side effect of doing it w/ Lucene 9 instead of waiting to release 3.0 with Lucene 10.
I actually think we should release REST API in a major instead of a minor. Releasing in a minor makes it harder to discern between 2.x-PreAPI and 2.x-PostAPI without more mid-stream version juggling which has caused many of the bugs we’re seeing in 2.x today.
The ongoing bug reports and patch releases to fix bugs introduced due to version juggling to maintain Elastic and Client cross compatibility.
I think this is the main blocker. I started this work but my plate has been full; and most of the development effort is on long term architecture changes / new features and less on quickly fixing near term pain points like this. That makes it hard to move quickly even if we wanted to so I’m not sure we could make the penciled in August deadline.
Another quick observation: I haven’t seen many users chime in here on their preference of releasing 3.0 this year or next. In my experience, if a user doesn’t have a critical need for a new feature they typically prefer waiting until at least a minor or two release before bumping to the next major version. I speculate that’s part of what we’re seeing here. And if so, releasing 3.0 in October 2022 vs early 2023 may be moot to the users anyway. I’d love to hear otherwise.
Just wanted to chime in as a user on what would make this decision easier:
A more detailed pros and cons regarding other features.
Breaking compatability with Elasticsearch is major, many of our clients still use the Elasticsearch NEST client as the C# client isn’t released yet.
Even if it is, it will take some time until trust is gained, so for now, they still use NEST.
This is just one client, but the supporting community just isn’t there yet. It has started to move there, but many things (especially 3rd party) are not supported yet.
However, with all that said, killer features which will happen much sooner if 3.0 is released on schedule could move many of our clients.
So what exactly is likely to be delayed? What isn’t?
Do take note, “likely” is definitely enough. But a more thorough understanding of the downsides except “breaking from the Elastic chain” (which I can sympathise with, Elastic sucks as far as we’re concerned) could help sway our clients better here.