We missed the March 9th freeze date for 2 of the plugins. This prevented us from starting full end-to-end testing of the bundle.
We’re currently projecting a slip from March 15th to March 17th
1.4
Originally scheduled for April 26th
With the upcoming 2.0 freeze date currently scheduled for April 18th, we are proposing to cancel 1.4
Are features in 1.4 the community would want to adopt on 1.x where shipping them in 2.0 will prohibit adoption
2.0
We are approximately 2 weeks behind getting all Lucene changes merged into core due to the depth and breadth of the changes
Originally scheduled for preview on March 31st
To allow plugins to do integration against the mainline of Core we increase their timeline from 10 to 28 days
This would make the 2.0 release date overlap with 1.4, so we are at this time evaluating removing the 1.4 release and moving from a preview to a release candidate
Release
Feature Freeze
Release Date
1.3
March 13
March 17
1.4
April 20
April 26
2.0 RC
March 21
April 25
2.0
May 2
May 12
We would appreciate the community’s feedback on this by Monday, March 21st.
Thank you
None of the new features in 1.4 would make a difference to us to have them in 1.4 vs 2.0. I think the path put forth makes sense and we are excited to see what’s to come for 2.0!
The nomenclature/dates confuse me. 2.0 RC is feature frozen on March 21, then is released on April 25 … but then there was a May 12 date thrown around for GA. Is the expectation that users/customers would find bugs in RC, and we would submit bugs/issues before May 12?
Thanks for the feedback @everbeck32 - we’re excited about 2.0 as well!
We’ve added a bit more to the date table that should clear things up. Essentially, targeting a time period between April and early May for feedback on the RC release. (working through details of what this will mean)
here’s a vote in favour of having another 1.x release:
there are two features for the security plugin which we need which are not yet in 1.3 (one has been merged to master a few days ago, the other one is still lacking a PR which will however hopefully come soon). having them in 2.x means that we’d have to do an upgrade from elasticsearch 7.10.2 to opensearch 2.x which contains breaking changes, so we might also run into issues with 3rd party libraries and products (not all support opensearch yet, e.g. spring-data-elasticsearch).
having a 1.4 release containing these features would give us a way to already migrate from elasticsearch 7.10.2 to opensearch as a first (and important) step. from there we can then properly plan the upgrade with breaking changes (which is a bit more complicated for us, as outlined in this thread).
this is, admittedly, a very selfish view as others might not depend on these features… so here’s a more general view:
the more general problem with a 2.x release (i.e. a release with breaking changes) now is that there are still gaps when it comes to clients (esp. e.g. the mentioned spring-data-elasticsearch) and until this is resolved it’s not possible to move to a 2.x release for users of such libraries. and during that time it should be possible to still have 1.x releases to have a clear upgrade path and important new features.
maybe it’d make sense to gather a list of gaps (missing features) which currently prevent people from migrating to opensearch and ensure that they are filled before abandoning the “stepping stone” 1.x release? this will include some features in opensearch (or, more likely, opensearch plugins) and some client work, maybe also some things around (think e.g. the kubernetes operator as some might currently be using ECK and waiting for that to migrate over).
specifically e.g. for spring-data-elasticsearch they plan to only add proper opensearch support in a new major version: Prepare for the "next generation" clients · Issue #1853 · spring-projects/spring-data-elasticsearch · GitHub
so that means that users of this library can use it for now (in a slightly outdated version which uses a RestHighLevelClient w/o the version/license check) with ES 7.10.2 and opensearch 1.x but will then have to first upgrade (and roll out) their client applications to use the new major release of spring-data-elasticsearch before they can start upgrading their cluster(s). this already creates a dependency chain which has to be managed.
also, some people might not yet be able to migrate because their distribution is not yet supported, this board seems to give an overview: OpenSearch distributions · GitHub
Evaluating what’s currently on the roadmap (https://github.com/orgs/opensearch-project/projects/1) for 1.4, it looks to me to make more sense to hold these off to the 2.x branch, and go full steam ahead under 2.x. That said - you folks may want think about a smaller and earlier 2.1.
As I mentioned in the meeting last week, rapid-fire releases create a lot of extra work. If there is going to be a short period between 1.4 and 2.0 (which is what the schedule would indicate), we probably wouldn’t deploy 1.4 even if it was produced – unless you managed to sneak something in there that is critical for us in the short term.
what does this means for those which currently don’t have an upgrade path from ES 7.10.2 to OpenSearch 1.x? will there be a direct upgrade path (rolling cluster upgrade; not re-indexing!) from ES 7.10.2 to OpenSearch 2.x.
That’s a big concern in my head too. The data should be compatible (Lucene 9 can read ES 7.10 Lucene 8 just fine), but the devil is in the details and heterogeneous clusters should work. Personally, I want to see it and the 2.0.0 branch is getting to a build state where it’s possible to try a number of migration scenarios. Seeing this migration should to does in my mind. All this testing will occur before 2.0.0 comes out.
Keep in mind OpenSearch 2.0.0 is a breaking change from ES 7.10.x / OpenSearch 1.x with regards to the removal of mapping types. So it’s responsible to test to make sure your pipelines / tools / scripts don’t use these features. The deprecation of mapping types pre-dates OpenSearch, so it shouldn’t be a surprise.
if no direct upgrade ES 7.10.2 to OpenSearch 2.x will be possible, can you please create a 1.4 release with any missing features people need (with contributed PRs, of course) to give this stepping stone?
of course such breaking changes are to be expected and i think they’re also ok (same as going from ES 7 to ES 8). is there already a list of known breaking changes (which can also still be work in progress) so applications can be checked against this?
btw, one important thing regarding client compatibility: as mentioned there are still various clients which are not supporting OpenSearch directly. some of these might also have issues with OpenSearch reporting a different version number compared to Elasticsearch. will compatibility.override_main_response_version still be present and supported in 2.x (it’s currently still present in the main branch) and maybe also 3.x? i vaguely remember that it was once stated somewhere that this would only be around for 1.x as a stepping stone and then be dropped, but as it stands now it’ll still be needed on at least 2.x and i don’t see any ticket discussing the removal of this flag.
of course, client compatibility will be an issue for things where the API changed (which is presumably mainly in the administrative area where pure wording changes were done?), but for many use-cases it might still work.
Humm. I don’t think 1.4 (in any form) is now in the plan. You’re asking for backporting, right? I think you’d need to convince a lot of people that there is a pressing need.
As far as list of changes, not yet, but I think everyone is clear that there needs to be crisp run down of every breaking change. I believe there are people working on that.
I have real mixed feelings on reporting a version that isn’t compatible. Seems like runtime error tears waiting to happen. But your point is taken - do you have specific clients/tools which are incompatible?
FWIW reporting as 7.10.2 was added pre-1.0 and was useful for a while. Then there were license check built into a bunch of ES clients which made it less useful. Most of the clients were forked (granted: not all yet). So, in my mind, a lot of the usefulness of this feature has lapsed, or at least narrowed significantly. Feel free to tell me I’m wrong here.
Also, the C# client which still isn’t supported. Probably a few other languages too.
We’ll personally managed (although as said through PM we’re barred from contributing to my great dismay), but I am guessing quite a lot of devs still don’t have support as seen on the clients topic