I’d like to just add that the concept of uninstalling plugins rather than providing a plugin-free artifact is not forwards compatible unless we add an uninstall-all-plugins command. Otherwise, for each new OpenSearch version we would have to account for the additional new plugins to uninstall in our scripts.
1 Like
Just throwing thoughts. Maybe the approach of disabling plugins vs uninstalling may work. So if there is no need for specific plugin, we can add a configuration option to disable the plugin. Like for example there was with x-pack related plugins.
1 Like
Super interesting thoughts and valid perspectives here.
Just a quick note that ES already ships w/ a loaded plugins directory (e.g., analysis-nori, analysis-stempel, repository-azure, store-smb). So I’m curious what percentage of folks are actually using all of those plugins, what percentage of plugins are being used, if there are folks that just ignore them or uninstall them, how you view them (required core plugins even though they’re actually optional), etc… The monolithic approach doesn’t scale well, the separate repository approach suffers from “bundle, install, remove” concerns, offering multiple separate archive downloads can be very confusing. So I don’t think there’s a silver bullet here and I think any step away from a monolithic bundle is a step in the right direction; so much so that I would really like to see the entire codebase move to a VERY microservice like approach - all the way down to extracting the aggregation framework (and their ValueSource support classes) out of server into a separate module.
1 Like
We use a few of the plugins from the plugins directory in ES (such as the repository-s3 plugin). The others don’t need to be uninstalled because they’re not installed yet (they are just idle in the plugins directory).
+1 for this
1 Like
Otherwise, for each new OpenSearch version we would have to account for the additional new plugins to uninstall in our scripts.
Exactly. And yes, it is important here to note the difference between “install” and “remove”. This is what we’re discussing here and the number one difference between plugins and modules. Modules are required (installed), plugins are not. Here we’re only talking about plugins (not modules) and how folks would like to handle them. OpenSearch has inherited a monolithic repository approach where all predecessor OSS plugin source lives in the plugins directory and distributed as part of all “flavors”. For ES, X-Pack lived in a separate monolithic directory and was included in all distribution flavors except oss. Now OpenSerach has the added ODFE plugins, which follow the “microservice” approach living in separate repositories. The idea for OpenSearch was to only provide a single “tgz/zip” artifact on the download page that includes ALL plugins in the plugin directory, but those plugins are still optional and only installed explicitly. In that regard, is there a reason to even provide a -minimal artifact? Just delete the source if you don’t want it on the filesystem? This minimizes the clutter (and possible confusion) on the download page, provides all users/customers with everything with the ability to pick and choose what they want to install instead of having to go to another location and download multiple artifacts.
2 Likes
There is a separate user/customer case: the third party plugin developer. DistributionDownloader is used here; buried under the covers of the opensearchplugin gradle extension. Today, the third party plugin developer extends opensearchplugin which has the burden of downloading the “tgz/zip” artifact (based on desired “flavor”). In this use case, the artifacts mattered. The suggested change here is to publish the released jars on maven central, and nightly build jars (unstable) in a separate opensearch provided snapshot repository. This way third-party plugin developers can write the plugin and only include the desired dependencies in the defined dependencies block of build.gradle. In this use case, there would also be no need to provide a “minimal” download on the opensearch website.
2 Likes
I agree 100% as this maintains the current behavior + as you said, minimizes confusion when faced with several artifacts to use.
Here’s a link to the yaml file with the elasticsearch / Kibana build details. We also have some documentation on how the ELK stack is used in Antrea. Keep in mind that this is just one example of our many products that incorporate these technologies in various ways.
There are a lot of great points in this discussion! Reading through it sounds like there is a need for downloadable artifacts without the OpenSearch plugins (alerting, AD, security, etc…) installed. I see at least two options we can proceed with. 1/ have tar.gz/docker images with all of the OpenSearch plugins downloaded to a directory in the plugin folder, but not have any installed or 2/ have tar.gz/docker images without the OpenSearch plugins packaged at all.
The pro to approach 1 would be that if someone wants to use a subset of the OpenSearch plugins, they would be available to install them without any additional downloads. The con is that the package size would be larger than a package without any of the plugins included.
The pro for approach 2 would be a smaller package with the con being that if someone wanted to use a subset of the OpenSearch plugins they would need to be built from source to use until built versions of the plugins are hosted.
Right now I’m leaning towards approach two to keep the minimal artifacts smaller. Does anyone have a strong preference to 1 or 2? If so, please elaborate 
I also personally like the term “lite” for the name of these artifacts e.g. opensearch-lite-1.0.0. I like lite since less software would be running on these artifacts.
@dblock opened an issue in OpenSearch and an issue in OpenSearch Dashboards for minimal artifacts to track.
2 Likes
No strong preference. Leaning towards #2 as well.
thanks for writing this out, @elifish! i agree with your reasoning and could live with either option but would lean towards #2.
one reason against #1 (and for #2) comes to mind: if you’re not using a plugin but it’s still in your docker image (or other artifact) and you’re running security scans on this you run the risk of the scan finding something and causing you additional effort even though it’s not something affecting you (you’ll have to update the image to use a package with the fixed plugin - if it’s even already fixed at that time).
Looks like we’re leaning towards #2, I think the only right choice for the name is lean 
More seriously my first choice would be -min vs. -lite, because the latter would imply some kind of packaging with other artifacts. But maybe that’s a good thing long term when we want to introduce new required modules 
1 Like
I also like -min
I like -min or -barebones. IMO -lite implies something slightly different - fewer features, or a lighter resource footprint, that aren’t exactly what’s different about this distribution.
I would like to suggest some more options:
-core, -base, and for those who like to speak in flavors -vanilla
so the artifact is out there, thanks to all who were involved!
Are we planning to have a base image for this version as well?
(I couldn’t find the issue for that if it already exists in some way or form)
A docker image for the -min distribution? It’s not planned.
Do you have a super compelling use-case?
Replacing the existing lines in our scripts to go from pulling the elasticsearch-oss image, to pulling the opensearch-min image.
As we stated in multiple threads, the plugins are not required by many users of the project, hence the -min artifact
However, if we are to expect users to replace their usage of elasticsearch-oss with opensearch v1.0.0 they may be surprised that nothing works since the security plugin will be a major blocker. Sure, security is really important, however, anyone using this artifact can be notified that it is not using that plugin by default.
Furthermore, just found this old PR being ignored for a while - Add opensearch-min Dockerfile and config by camerski · Pull Request #81 · opensearch-project/opensearch-build · GitHub
Any updates on this? This would save the community some extra steps to using the project if they require the -min artifact.
@amitai Sorry for the delay in responding - I thought I had already replied. I had a draft just sitting in the thread that wasn’t posted.
-min is really designed for development and embedding. Publishing a docker image for -min means folks could just find it on dockerhub and will likely use it for production directly. Or worse, a blog post/course will instruct people to do a docker pull opensearch-min - there is very little opportunity to instruct the user that they are entering the danger zone. Additionally, users could easily assume they have some sort of security since it’s documented that the project has security
.
The website has the interstitial warning page before downloading to warn about the lack of security as a way to prevent mistakes like above.