Join us on March 14th for our Community Meeting

kris · March 14, 2022, 7:16pm

Link to our agenda & chat log: OpenSearch Community Meeting - 2022-0314

github.com/opensearch-project/OpenSearch

OpenSearch Extensibility

opened 11:16PM - 11 Mar 22 UTC

discuss RFC

## Introduction OpenSearch is committed to being a vibrant and welcoming comm…unity-developed product. Community development, at its best, lets people with diverse interests have a direct hand in guiding and building products they will use; this results in products that meet their needs better than anything else. Additionally, community development allows the project to scale, as the community is able to find and build new areas of development that they are passionate about beyond what a single person or company could support. This acts as a virtuous cycle where new users and contributors add new features, which in turn draw more users and contributors. To drive this flywheel, we propose that extensions become the default way to implement new and extend existing features in OpenSearch. To raise the bar for extensibility, we will provide the community with a well-supported OpenSearch catalog for extensions. Our vision is to build the equivalent of “Visual Studio Code” and the “AppStore” for OpenSearch. In the same way that these tools acted as a force multiplier for the number of problems that an iPhone can solve, we want to build an extension ecosystem that enables the community to solve more with OpenSearch. No single organization has the ability to prioritize every problem, so enabling developers to easily build extensions for OpenSearch will allow the project to address a broader range of end user problems. In the future for OpenSearch, we want to see thousands of new features being quickly and painlessly built by developers. And we want those features to be easily discoverable by the community, who will be able to easily install them with confidence that they’ll be able to use them securely with no impact to their cluster. ## What's next? To reach our goal of making OpenSearch extensible, there are three major areas that we’ll need to make changes in: * API/Versioning * Independence/Sandboxing * Discoverability/Dependency Management ### API/Versioning **Problem**: Plugins are rigid in terms of compatibility and have to be built with a specific x.y.z version of OpenSearch during compile time. This tight coupling reduces the velocity of software development lifecycle for OpenSearch and plugins because it requires all plug-ins to release at the same time when a version number is raised. An additional side effect is that plugins cannot be installed/uninstalled/upgraded/configured without restarting the cluster. The underlying problem is lack of versioning support for [extension points](https://opensearch.org/blog/technical-post/2021/12/plugins-intro/) on which plugins are extended. These extension points are part of core modules of OpenSearch (like settings etc) which do not support versioning. Plugins rely on these extension points to get notified on the changes in the system. **Working Backwards**: Who are the actors in the community: a. Extension developer b. Extension user c. Clients developer What would the customer like to see/use: a. Not worry about updating Extension for every patch version of OpenSearch. b. Extension is not broken when OpenSearch minor version is upgraded. c. Install/Update/Remove an extension without restarting OpenSearch. **How we’d like to solve it:** [OpenSearch#2283](https://github.com/opensearch-project/OpenSearch/issues/2283) 1. Add versioning support for all modules which support extension points (e.g [Settings](https://github.com/opensearch-project/OpenSearch/blob/1.2/server/src/main/java/org/opensearch/common/settings/Settings.java) etc) . 2. Revamp all extension points to add support for versioning and backwards compatibility. 3. Build and publish SPI’s a.k.a new extension framework which supports versioned APIs for developers. 4. Add support for extensions to introduce [API specifications](https://github.com/opensearch-project/OpenSearch/issues/1590) for clients. (Potentially via a new extension point) 5. Add dynamic support for all modules which support extension points. i.e add support to register/remove extension handlers dynamically. 6. Add supporting tool to define and consume extension metadata which includes versioning. ### Independence/Sandboxing **Problem**: Because plugins currently run in the same process (and JVM), plugins have unrestricted access to various resources across the cluster. Plugins can therefore fatally affect the cluster, impacting core functionalities like indexing and searching. to the point that the cluster becomes unavailability **Working Backwards**: Who are the actors in the community: a. Extension developer b. Extension user What would the customer like to see/use: a. Not worry about cluster going down due to an extension misbehaving. b. Run a 3rd party extension and not worry about a 3rd party accessing data and configurations on the cluster to which it should not have permissions. c. Ability to support granular access control of cluster resources for an extension, e.g. CPU, Memory, etc. d. Ability to write an extension in any language of choice. ![152451097-b8814ebb-8bbf-49a5-bfc5-d011274f10d9](https://user-images.githubusercontent.com/16944164/157987957-7acfb880-674a-4200-8243-55c6adffe476.png) **How we’d like to solve it**: [OpenSearch#1422](https://github.com/opensearch-project/OpenSearch/issues/1422) 1. Running extensions within the same process/JVM of OpenSearch limits the ability to secure the cluster. Also it doesn’t scale when we’d like to run many extensions within the same node. We believe adding support to run extensions outside of OpenSearch process solves these problems if we can define a common communication protocol and make extensions independent. It enables all extensions to talk via a common interface and not fatally effect the core of OpenSearch. 2. Build and publish extension SDKs which will translate messages between OpenSearch and an extension. These SDKs should be distributed in multiple languages while keeping the same communication protocol. 3. Add granular security support for cluster resources (in OpenSearch) and node resources (potentially via extension SDK). ### [Discoverability/Dependency Management](https://github.com/opensearch-project/OpenSearch/issues/5572) **Problem**: Plugins are not discoverable from the distribution. There is no way for a customer to know what plugins exist in the community and how to install them. Also customers have to understand the versioning compatibilities of OpenSearch and other plugins. **Working Backwards**: Who are the actors in the community: a. Extension developer b. Extension user What would the customer like to see/use: a. Discover all OpenSearch extensions in one place. b. Not worry about extension and its dependencies, but just install and ready to go. **How we’d like to solve it:** 1. Solving the problem of versioning, it results into another problem which is dependency management. We should build a package manager which understands the extension manifest and manage all its dependencies (including dependencies on other plugins). The extension manifest would contain version, dependencies, security policies etc. 2. Build and publish extension manifest which would be the first step towards the catalog. ### FAQ 1. How is the latency going to be for extensions? Our goal is to get benchmark numbers to understand how much performance impact we’ll see and are tracking via [issue](https://github.com/opensearch-project/OpenSearch/issues/2231) and [issue](https://github.com/opensearch-project/opensearch-sdk-java/issues/24#issuecomment-1309547639). 2. Can we use extensions for plugins like Index Management and K-NN? Our goal is to support performance intense workloads via extensions. Depending on the benchmark results we will explore different solutions to make the communication light weight (like protobuf etc). 3. Will existing plugins break when we launch extensions? No, existing plugin architecture will be supported and will be just another form of extensions (running within the process of OpenSearch). 4. Is performance of OpenSearch be impacted? We do not know yet, but we are actively working towards to get data. (FAQ 1). Our hunch says it will be, since the communication is synchronous today. 5. How does extensions impact Dashboards? Dashboards (and its plugins) doesn’t rely on OpenSearch plugins architecture. They communicate via REST APIs. But dashboards has similar architectural problems which have to be solved through. 6. How will I be impacted if I use a small or single node cluster? The goal of extensibility is to make as easy as possible to develop, build and use them. We will strive to make it simple for clusters of all sizes. 7. With Anomaly Detection plugin as an extension, are we extending any functionality of AD? With AD plugin as an extension, we are working on building entire AD plugin as an extension. AD extension is just mimicking existing AD plugin. 8. Is OpenSearch moving to convert all plugins to extensions? Our goal is to near future move all existing plugins as extensions. Our vision is that all new plugins are build as extensions. 9. Today AD extension which is being built runs on a separate node. How would scaling factor in? Extensions will support all modes, in-proc, separate JVM, separate process, remote. We will let customers determine and choose how they would want to run an extension based on their use-case and needs. 10. Have you considered using gRPC instead of REST for inter service communications? We have and will further dive into it. For this first phase we decided to go with REST since the existing clients is REST based. We will definitely look into gRPC. ### How can I contribute? We would love to have your contributions to make OpenSearch extensible. Within the 3 focus areas, we just started scratching the surface with sandboxing but there is a lot more work to make this happen. Feel free to pick up any of these issues and let’s make it better, together!

https://twitter.com/OpenSearchProj
https://www.linkedin.com/company/opensearch-project

Topic		Replies	Views
Join us on March 1st for our Community Meeting Community community-meeting	3	511	March 3, 2022
OpenSearch Community Meeting - 2024-1001 Community community-meeting	2	65	October 3, 2024
OpenSearch Community Meeting - 2023-0117 Community community-meeting	7	920	January 17, 2023
OpenSearch Community Meeting - 2023-0425 Community community-meeting	7	602	May 1, 2023
OpenSearch Community Meeting - 2024-0312 Community community-meeting	1	211	March 12, 2024

Join us on March 14th for our Community Meeting

Related topics