Does anyone have any status on the RPM distributions? If I remember correctly, this was supposed to be a GA (1.0) feature. It was then pushed to 1.1, but then 1.0.1 got released at the 1.1 planned date and 1.1 pushed further to the right. Now it looks like even the ‘1.1’ tag got removed from the RPM issue in GitHub (https://github.com/opensearch-project/opensearch-build/issues/27). Is there any information that can be shared about when we will get a RPM for OpenSearch similar to what has consistently been available for ElasticSearch? Thanks in advance for your insights.
Oh, this was disappointing news to me I was waiting for the 1.1 to release before upgrading from OpenDistro because the rpm support https://github.com/orgs/opensearch-project/projects/1
x64 RPM is in 1.2 on Nov 9. ARM RPM is going to be in 1.3. As @oscark noted, it’s on the road map.
A lot of this was affected by needing to fork the clients and produce an output plugin for Logstash (see: OpenSearch Roadmap & Release Update). It’s nothing technical, it’s just a bottleneck in capacity. Good news is the team is automating at they go along so it should pay off later with the ability more rapidly release.
But I do hear you both - RPM and DEB are super vital to serious use for a lot of folks.
3 Likes
x64 deb will be added in 1.2 @searchymcsearchface?
@yuvaraj Yup. DEB x64 is in 1.2.
https://github.com/orgs/opensearch-project/projects/1?card_filter_query=deb
Can you please give us an update about the delivery plan of RPM?
Looks like this topic is moved to 1.3 release in Jan.22?
If this is the case, can you please provide a step-by-step manual how to create the RPM on local machine. (in my case for RHEL as target platform)
This is equally disappointing to me as it was two months ago. IMO it undermines the trust in OpenSearch to make changes to the version objectives a couple of days before it is supposed to release (twice!).
Basically I have postponed upgrading from Open Distro because I thought that the rpm will be release in the “next version” basically since OpenSearch was released, I will not wait for rpm to be released this time. I would have upgraded to 1.0 straight away if I had known that it would take ~6 months for rpms to get release.
There’s an old saying in Tennessee — I know it’s in Texas, probably in Tennessee — that says, fool me once, shame on — shame on you. Fool me — you can’t get fooled again.”
1 Like
Hey folks. I hear you on this one - I get it that it can be frustrating. OpenSearch works on a release train model, so it looks like the RPM didn’t make it based on the timeline.
You might want to keep track of this via the issue if you aren’t already:
https://github.com/opensearch-project/opensearch-build/issues/27
2 Likes
Let me first say that I find your communications great and from near as I can see on this board, you seem to be a great guy. Nothing of the below is all that important in the grand scheme of the universe and it certainly isn’t intended to be an attack or personal.
From Sep:
From Nov:
I was really excited at the notion of OpenSearch and I thought the actions of Elastic were disrespectful to the community that was a big part of their success. However, my enthusiasm for this fork has waned and continues to do so. With the continued prioritizations that can be seen, it seems like the name OpenSearch is a bit of a misnomer; AmazonSearch might be a better description.
Now, there is nothing wrong with that - clearly there are a lot of Amazon people contributing to the project, but I think when a feature as critical as RPM/installers, which you acknowledge is ‘super vital to serious use for a lot of folks’ are booted from 1.0, to 1.1, to 1.2, to 1.3, (to 1.never???), the project is saying that the open source community isn’t the target for this project. This isn’t some small feature about an idiosyncrasy of some off-the-beaten-path behavior - this is the primary means open source packages have been maintained and installed for decades, including from your fork of ElasticSearch.
The continued lack of priority for something as fundamental as a RPM/DEB installation for the largest platform (x86-64) is disappointing on a micro-level and as a commentary as to the state of the fork. At least in my admittedly limited experience, the responsiveness of the devs has been underwhelming (I did as you suggested and posted some comments and items in the referenced ticket back in June, but was ignored). I am sure there is a ton to communication happening inside the Amazon systems, but my experience is that it is absent in the open source facing tools. I sincerely hopes this changes for the better.
1 Like
@justme OK- you make some really valid points here, and yeah, I think it needs further explanation. I wasn’t personally involved in the decision making but I was personally helping with the spec files for RPM last Thursday, so A LOT changed since that time.
I followed up with some folks closer to the decision and I was forwarded to a design review doc which was 12 pages (
) from 2 days ago. Let me try to summarize - the original idea was that it would just reuse the ODFE process which was based on gradle and a plugin called ospackage. It wasn’t ideal but made more sense when we were relying on the ES artifact vs our own artifact as we could easily bring that in. It also causes all the plugins to maintain their own workflow to create the plugin package for each release. Short/short: using the old process would be carrying forward technical debt and slowing the overall release process down until that technical debt was resolved.
Instead, the solution would be to move to a different tool, which would move the project in the net right direction and pay dividends in release velocity down the road.
There are a couple of more approvals to go through (open source reviews to confirm license compatibility of tools, etc.) but then we’ll publish the entire direction to the build repo. We’re also working on a markdown file that will let you generate your own RPM artifacts as a stop gap which will be put in the repo.
3 Likes
I appreciate the response and your genuine effort to try and give some context. However, after reading your post, I can’t help but ask a few follow-up questions, if you don’t mind me probing. I have two areas of concern that pop into my head:
Area 1: Can I ask where this 12 page design document is? I have been trying to keep tabs on this, and with the exception of a couple of posts to the github issue in the last week (which made very little sense without the context in which they were making comments) there is no activity in that issue other than when they remove the label for the next release when it misses it. I am not accusing anyone of doing this in secret, but it appears there is significant direction happing behind the Amazon curtain and coordination of only Amazon employees. These types of behaviors will only serve to reinforce the ‘AmazonSearch’ instead of ‘OpenSearch’ moniker that the project at least stated it wanted to have and be. I suspect this is a symptom and not the only instance where this type of thing is occurring. If I am oblivious to the public forum/distribution list/whatever, then I apologize in advance for my ignorance. Is this something that you think can or will change, or has there been some rethinking on behalf of Amazon management and its role in this project?
Area 2: A little more uncomfortable, but how is it a feature that was supposed to ship months ago only has a design document written two days before the third release it was targeted for? And then to be written in a way that has significant technical debt? My general impression was that this kept getting slipped because you needed to do it correctly, which seems like a good technical call, but your description of the state of it two days before release is worrying. Again, I am not as concerned about this specific issue as what it may say about the trajectory of the project and prioritization of issues. I realize that everyone is trying their best against the list of priorities that are given to them to hammer out, but my point is that the priorities seem a little bent out of shape as I would expect for an open source project with the goals and aspirations of OpenSearch.
Admittedly, my two areas of concern are somewhat rhetorical. You seem to grasp some of the worrying signs from the perspective of the community and I perceive that you are a good intermediary into the Amazon side of the effort. I think you will more likely than not try to communicate such and try to find meaningful changes that could be made on both sides of the equation. I am relatively sure you will respond, but the progress and changes in the project over the next 3-6 months will speak louder, although I fully realize you may not be a decision maker in that process. I appreciate the opportunity to have an intelligent discourse on the issue. In any case, as I assume you are in the US, I wish you a good holiday next week!
Area 1: Yup, you’ll see a consolidated version posted as a markdown document - I was actually referring to that in the previous message. If there are things we can’t do (e.g. include an incompatible dependency), those parts are just dropped from the final version. You’re right context is important, so just dropping something in the repo would be puzzling. However, it’s a fair callout that more openness in these processes is better and I can honestly say I see more important decisions being made on github as the project has evolved. It’s not perfect yet, but the phrase “could this decision be made on github” is common in meetings.
Area 2: This was a moment where a final review didn’t go as planned. Generally there is a direction and then a final review. I’m sure you’ve had those moments where things look good initially and you’re just not satisfied with the end result.
Anyhow - RPM is a very active area of work. Glad to discuss it - I do know it’s important to a range of people.
Also, I’m not in the US, I’m in Canada. I appreciate the sentiment but my turkey day was well over a month ago. I’m here all next week 
1 Like
Bringing up this thread as RPMs were again planned and eventually got out of scope for 1.3.0. Reading from 1.3.0 release notes:
RPM & Debian Distributions
As of yesterday (March 16), the team identified the roadmap for 1.3.0 included references to RPM and Debian distributions. In late January, the distribution efforts started to be tracked on a separate board as distributions were decoupled from version releases. In this transition, the cards for RPM and Debian distributions were never removed from the 1.3.0 roadmap. The team regrets this error and have put measures into place to ensure that the roadmap will be more accurate moving forward.
Having a look on that separate board I can see relevant tickets but there’s no timeline or some target release. I understand that this work takes time and totally respect all efforts done, however particularly because of how crucial and demanded this is, it would be nice to have some timeline so that we can also plan accordingly internally. Especially since OpenDistro 1.13.2 will reach its EOL on May 2022 (along with ES 7.10.2) and the lack of RPMs has been consistently reported as a blocker for the jump from OD to OpenSearch.
3 Likes
I wonder what witty explanation @searchymcsearchface will whip us this time (I do feel bad for him as he seems like a generally nice guy and he has to put the PR face on and explain this one again). I guess that ‘final review’ from November is still being kicked around over four months later. So much for that promise that ‘RPM is a very active area of work’. The most worn out thing about the RPM is the perpetual movement from 1.0, to 1.1, to 1.2, to 1.3, to well - never! It must be very tiring for that poor ticket to have to keep enduring the movement and relabeling.
At least that is one thing they did fix: it will never miss a release again since they essentially put it into neverland with no target. Here we are over nine months late and we are still only 6 of 21 tasks compete, per #27 in the build repo. That definitely speaks volumes for who Amazon’s intended audience here is. I guess that is their prerogative; I am sure you can easily get the most recent release available to you if you were to pay AWS to get an AmazonSearch instance allocated. The communication is still exceedingly poor. I couldn’t even get a clarification as to what the latest comment in meant with respect to the release.
I just wish Amazon was more honest at the beginning of the project to make clear this isn’t a replacement for ElasticSearch, but rather that this is needed to replace ElasticSearch in AWS. Then, at least the community could realize that a gap still existed. It is a shame. I wish I could say I was disappointed, but I came to expect this with the perpetual movement of this essential task. At this point, my enthusiasm for this project is fully dead. I wish I could blame Amazon, but really, I should have expected this from the beginning. I was a fool to believe this would be any better than Elastic’s behavior and disregard for the community.
Hello to @justme, @spapadop and the others that have chimed in on this thread.
You’re right — We have missed this requested support several releases in a row, which is not acceptable. Not only did we miss our stated promise, but we also didn’t communicate clearly enough about what was causing the delay and what the next steps were.
So I wanted to give the community an update from the engineering front. I’ll do my best to explain how we got to this point, the current plan to provide RPM/Debian support, and how we’re going to handle this going forward into 2.0 and beyond.
A little historical context as to what happened:
We supported Debian and RPM in OpenDistro for Elasticsearch (ODFE), and our initial plan upon creating OpenSearch was to to re-use the same packaging design. But once we dug in, the team documented several issues with the old DEB/RPM distribution packaging process. For example, we were using fpm to create packages. Overall, FPM is a fantastic tool for quickly building an RPM without a spec file but its not recommended for creating production grade RPMs.
We collected community feedback, and concluded that a new packaging process needed to be created for for OpenSearch. The primary motivator for this change was quality; we believed that we owed the users of this project a better installation/configuration experience, and the resulting delays have been primarily related to that. We continued to update the issue to add clarity to what had become a complex development challenge.
At the same time, we also decoupled the distributions from the main project. Releasing a new distribution shouldn’t need to wait for a bundle release, and seeing a future in which even more distro support will be needed, we broke distribution into its own roadmap.
Which brings us to 1.3. During the run up to the 1.3 Kyle caught that although we’d created a new roadmap for distributions, we had not removed them from the main software roadmap (good catch Kyle!). We could have chosen to delay the 1.3 release, but because we wanted to decouple distribution and software development, we chose to launch 1.3 and started working on a plan to support RPM and Deb as soon as possible.
Today, we have resources working on RPM and Debian Support, and we will provide that support in 1.3 as soon as we can. The 1.x codebase will not be closed prior to providing this support. This work is ongoing; once we have a firm commit on a date, I will follow up with the community in this thread.
We have already started changing our internal processes for reviewing roadmap items as a result of this issue — we should have caught that the distributions were still in the software roadmap much sooner. We are also looking for better ways to communicate changes on the roadmap — if you have suggestions, let me know. And we are also going to shift more resources on to RPM/Distro support so we can accelerate those projects. If we hit issues again, we will do better about communicating them. Please continue to hold us (and me) accountable, and if you have questions or concerns, raise them.
Finally, a call for help: We are actively trying to tackle this challenge. If this is an area you’re passionate about, we’d love to build this out with you — feel free to jump in on any issue listed here. We also have open positions for people looking to solve big challenges like these.
Thanks,
/C
2 Likes
I was wondering how I was going to respond and I had to read your message a couple of times to make sure I absorbed it, but when I came to the discuss to reply, it all became clear right from the discuss user interface:
It’s been a while since we’ve seen CEHENKLE — their last post was 7 months ago.
Forgive me for being so blunt, but is this really the person who is genuinely asking how you can communicate with the community better? There seems to be a systemic issue with communication. The lack of involvement of the actual performers of work in threads such as this one is evidence of it. A start might be that people interacting with the project actually communicating those things in a forum where people can see them instead of behind Amazon servers? Why not just tell everyone (including Amazon employees) that if the topic pertains to this project, this forum (or some other designated public place) is the place to have that conversation? Isn’t that how most successful open source projects work? Wasn’t it in Amazon’s own press release about OpenSearch that the following was stated (in April 2021):
We are truly excited about the potential for OpenSearch to be a community endeavor, where anyone can contribute to it, influence it, and make decisions together about its future. Community development, at its best, lets people with diverse interests have a direct hand in guiding and building products they will use; this results in products that meet their needs better than anything else.
It is crystal clear that the vast majority of communication regarding this project isn’t happening in public. How can you be uncertain why there is poor communication if people aren’t using open discussion forums to discuss elements important to the product? How can we achieve the stated aim to ‘make decisions together’ when basic information isn’t being discussed as a community?
I also have some trouble fully accepting your explanation about FPM. Although shockingly little has been shared with the community, it appears that the FPM issue was known about, at the very least, in November 2021. It seems that was what @searchymcsearchface was referring to in his posts there. Here is what I am referring to:
(Again, of course, since all this happened in private, it is really tough to sort out what happened here - I am sure you have the access one would need to figure it out). By the way, I am still waiting for the 12 page design document that @searchymcsearchface told us was coming, unless I missed its publication somewhere:
I will also note that I tend to believe @searchymcsearchface when he said (in September 2021):
This also seems to be confirmed by dblock in the github issue, speaking about the RPM function for 1.0 release:
Nobody was working on RPM then, but we should have been able to answer signing questions in general, apologies.
Just look at what all the OpenSearch team has achieved since the announcement in April 2021 and GA in July of 2021. And just look at what you are planning in less than ten months: Two first-digit releases! Are you really suggesting that a team that is capable of that much isn’t capable of releasing an RPM? First, let me assure you I am not trying to dismiss the craft of release engineering. It is a discipline and portion of software engineering that is often paid short shift. It isn’t the most alluring part of the process, by most engineers measure, but absolutely essential. @searchymcsearchface himself repeated the importance of the issue to community repeatedly, such as (in Sep 2021):
But, how can a reasonable person looking from afar and the information on the table come back with anything other than the conclusion that this isn’t really a feature that is important to the OpenSearch team? We all deal with limited resources and a lot of worthy, competing elements needing attention. The OpenSearch team has continuously deprioritized this issue, almost exclusively by communicating days AFTER a release target has been missed how important this issue was and how some totally unexpected thing made it impossible to deliver.
I also have to beg to differ about your statements about ‘collecting community feedback’ and ‘updating the issue’. These statements seem to be aimed at showing how good the efforts were engaging the community. However, please note that even though release was aimed for 1.0, that the issue you referenced wasn’t even opened until January 2022. What about the litany of basic questions that were simply ignored on that thread or the original #27:
Will packages be signed with SHA-256?
What repo will they be served out of?
Even just a basic: “Does that mean we won’t actually be seeing RPMs at the opensearch 1.3 release?” wasn’t answered.
This might be the right time to say that your closing comment really took me by surprise:
I think you might have said the quiet part out loud: You seem to be conflating the ‘we’ with ‘Amazon’, not OpenSearch. Where is that community effort that was being sought after? I have said before, and I will say again, that it is totally Amazon’s prerogative how they spend money/resources/effort. If Amazon wanted to take all the effort/code/outputs they are expending behind their curtain, that is certainly within their rights. I am not a opensource fanatic/zealot screaming we are entitled to those products. However, please also don’t make statements (such as the one in your January 2021 press release I referenced earlier) that you don’t intend to keep. If you really want community involvement and a tool useful to the community then you will have to let that happen. If that is no longer what Amazon (or this project) desires, then put out a press release that says that, shut down this site, and let the community respond to the hole that exists (Just as you responded when Elastic made its announcement). But please, don’t tease the community with a solution to a problem and then leave it hanging for at a bare minimum 10 months. Which leads me to my next point:
In reading your message, I can discern no project commitments, nor timelines to release this critical feature. In fact, moving it to a milestone which has no target doesn’t seem like the move of a project that sees this feature as important. Look at the length of this thread alone! This can’t have been a ‘first heard’ just now. I am confused why even after all the focus the project is so uncertain as to the completion that you can’t provide any information as to the time of release? And, please, don’t take this the wrong way, but in a sense I don’t care about what you say in your response about the timeline. The actions or inactions of the project have, and will continue to, speak far louder than any of the words I or you type on this thread.
In summary, I am not really sure what I expect as a response. As my earlier comment stated, I have kind of lost all enthusiasm for this project. I certainly don’t mean to be a nasty internet troll nor to make your day any worse. The whole thing is just kind of sad to me as a missed opportunity.
Forgive me for being so blunt, but is this really the person who is genuinely asking how you can communicate with the community better?
Fair call out. I tend to lurk here and work on Github. But it’s good reminder for me to be more vocal on the forums as well.
And, please, don’t take this the wrong way, but in a sense I don’t care about what you say in your response about the timeline. The actions or inactions of the project have, and will continue to, speak far louder than any of the words I or you type on this thread.
When I say you should hold me accountable, that’s exactly what I mean. This was a bad miss for the project, and I’m frustrated about it. I can’t fix what was done in the past. But I can own the mistakes, learn from them and improve going forward. But the most important thing I can do it put all my effort into delivering these way overdue distributions, so that’s what I’m going to do. I hope to rebuild your trust, but I also understand why you’re feeling burned. Either way, I appreciate your taking the time to comment.
Thanks,
/C
2 Likes
I wanted to provide a quick status update on RPM package creation process for OpenSearch and OpenSearch dashboards.
In-progress:
- Peter opened a PR to support building RPM distribution based on different versions and architecture. We will be able to support multiple architectures (ARM and X64) once its merged. I’d love a review on it if you have a moment.
https://github.com/opensearch-project/opensearch-build/pull/1815
- Tianle opened a different PR to add distributions to appropriate folder supporting additional distribution types (TAR, RPM etc…). Again, if you have time to review, I’d appreciate it.
https://github.com/opensearch-project/opensearch-build/pull/1807
Current Issues:
- Add type support in OpenSearch - We are actively working on adding a TYPE parameter in OpenSearch core. This is specifically useful when we want to install additional plugins, more changes, extra tweaks before packaging them into final release artifacts, without the need to untar/unzip/extract rpm before repackaging them back. ETA to merge is Monday.
https://github.com/opensearch-project/OpenSearch/issues/2571
Next up
- We are working on a PR to integrate RPM build workflow to OpenSearch build system
Once we merge this PR, we will be able to successfully build, assemble and publish both X64 and ARM 64 RPM artifacts to our artifacts storage bucket.
- Create a new Yum repo to support RPM installation
- Add Yum repo creation process to OpenSearch distribution workflow
- Sign the metadata file for Yum repo and RPM package (edited)
- We still need to make additional changes to support RPM for OpenSearch Dashboards (once the directory type issue is resolved)
If you’re interested in helping, let me know.
Thanks,
/C
2 Likes
Find below the current status update on RPM package creation process for OpenSearch and OpenSearch dashboards.
Recent updates:
• Peter has merged the below PR to support multiple architectures (ARM and X64).
https://github.com/opensearch-project/opensearch-build/pull/1815
• The previous blocker with TYPE has been resolved by the below PR
https://github.com/opensearch-project/OpenSearch/pull/2602
In-progress:
• The below PR to separate the target distribution folder is still under review.
https://github.com/opensearch-project/opensearch-build/pull/1807
The below issues are actively worked upon:
Call the generation code for RPM
https://github.com/opensearch-project/opensearch-build/issues/1545
RPM Package validation
https://github.com/opensearch-project/opensearch-build/issues/1555
Next up
• Build, assemble and publish both X64 and ARM 64 RPM artifacts to artifacts storage bucket.
• Create a new Yum repo to support RPM installation
• Add Yum repo creation process to OpenSearch distribution workflow
• Sign the metadata file for Yum repo and RPM package
2 Likes
We have made tremendous headway on reviewing the below PR’s and we will be merging the below PR’s shortly.
https://github.com/opensearch-project/opensearch-build/issues/1545
https://github.com/opensearch-project/opensearch-build/pull/1807
We are still actively working on adding automated validation and planning to complete it by end of this week
Automated validation:
https://github.com/opensearch-project/opensearch-build/issues/1555
We will start working on the below tasks next week.
Next up
• Build, assemble and publish both X64 and ARM 64 RPM artifacts to artifacts storage bucket.
• Create a new Yum repo to support RPM installation
• Add Yum repo creation process to OpenSearch distribution workflow
• Sign the metadata file for Yum repo and RPM package
4 Likes