Join us on April 26th for our Community Meeting

kris · April 22, 2022, 3:34pm

Agenda:

OpenSearch Dashboards 2022 Initiatives - Sean N
ml-commons - integration with PPL in OpenSearch 1.3 - Sean Z & Yaliang W

kris · April 29, 2022, 3:26pm

Thank you to all in attendance. Link to agenda & chat log.

github.com/opensearch-project/OpenSearch-Dashboards

OpenSearch Dashboards 2022 Initiatives

opened 08:05PM - 30 Mar 22 UTC

seanneumann

enhancement discuss

Hello there! We’re the team at Amazon that maintains the [OpenSearch Dashboar…ds repo](https://github.com/opensearch-project/OpenSearch-Dashboards). As part of planning for 2022, we wanted to share with you how we’re thinking about the upcoming year. We'd love to work with you to build out these ideas. This isn’t designed to be a comprehensive list of everything getting built in the next year. OpenSearch is a shared community project, so we're excited to see the ideas and projects that come from the community. These are just the areas where this group of people plans to put its time and attention given what we know now. So if there’s something missing that you want to see, that’s great! Let everyone know by writing issues/feature proposals/code for the thing you want to see. ## 1. Introduction OpenSearch Dashboards (aka Dashboards) is an open-source (Apache 2.0 licensed), Node.js/React/TypeScript based, analytics and search application for the [OpenSearch engine](https://github.com/opensearch-project/OpenSearch/issues/2095). Dashboards is the primary exploration and visualization tool for data in OpenSearch. Dashboards provides a platform for developers to build, test, and deploy extensible experiences, such as fine-grain access control, alerting, and index state management. As a community, we work with the aim of being the most open, secure, and versatile visualization application for analytics data. The growing expertise of contributors is rooted in application and service based architecture, working with diverse data sources, and rich user experiences. ## 2. Tenets 1. We strive for the highest level of community trust by continuously making Dashboards more **secure** and **reliable.** 2. We are committed to being an **open, collaborative, and community-driven** project. We encourage high quality community contributions and educate developers on best practices. 3. We provide a UX that is **intuitive** with a minimal barrier to entry to configure, operate, and monitor while still providing features for advanced users. 4. We proactively identify the needs of our users, and **innovate** on their behalf and with the community. 5. We provide **platform tools** to build enterprise-grade features. 6. We build on **automation** rather than manual effort for sustainability and scalability. 7. We help users gain **insights** rapidly and accurately, with minimal information overload. 8. We will make the latest version of Dashboards **compatible** with as many previous versions of OpenSearch as possible as it needed by the community. 9. We strive to **decouple** Dashboards from the OpenSearch engine to reduce impact to our mutual health and resources. 10. We aim to make Dashboards as **lightweight** as possible, while also ensuring the right functionality exists out of the box. ## 3. Key Initiatives These are the initiatives we’ve identified for 2022. As issues are opened for each, we'll link them here. ### (1) Dashboards Anywhere Dashboards is tightly-coupled with the OpenSearch engine: it depends on its indexes for persistence, and runs on and shares resources with OpenSearch data nodes (i.e. when OpenSearch is out of service, so is Dashboards). Dashboards can only visualize data from a single OpenSearch cluster (i.e. it has no capability to connect to other data sources or mix data sources into a single view). Customers continue to ask for a “single pane of glass“ across Dashboards use cases. Not only do they highlight their inability to triangulate issues as a major frustration, but they continue to highlight the need to use multiple tools to manage single use cases. Finally, the community has also highlighted that some patterns only emerge when data is combined and cross-examined with other datasets. To grow beyond these constraints, we will launch a more modular Dashboards that can run independent of OpenSearch resources to support a more stable and extensible user experience. This initiative will position Dashboards to run anywhere, whether it be on-prem, in a managed cloud service (e.g. AWS, Azure, etc.), a serverless environment, or even on a user’s Desktop. Critical pre-requisites to this initiative include out-of-the-box authentication, metadata storage, and the ability to connect to multiple data endpoints (e.g. OpenSearch, Elasticsearch, Prometheus, etc.). #### (1.1) Authentication Out-of-the-Box To run independently (i.e. not on an OpenSearch data node), Dashboards will require out-of-the-box security solutions. Currently Dashboards has no security features out-of-the-box that make it easy to connect to disparate data sources. We’d like to make interfaces to become first class citizens and various options of authentication to become extensions. We propose to move security APIs into the Dashboards core, enable security by default, and release various modes of authentication as extensions (e.g. basic auth, OIDC, SAML, etc). - [[feature-proposal] Dashboards security out of box #1215](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1215) #### (1.2) Metadata Storage Adapter Currently, Dashboards stores its metadata configuration inside an OpenSearch index. This coupled design approach works well for OpenSearch customers to get up and running quickly, but it introduces challenges while operating at scale and providing high availability for Dashboards. Metadata availability depends on OpenSearch cluster availability and other cluster parameters such as cluster health, state, and version. To mitigate this problem and unblock future extensibility of Dashboards, we are decoupling metadata storage from OpenSearch. This will allow us to configure metadata extensions to use additional databases such as MySQL, Postgres, DynamoDB, Serverless (S3+Athena). - [Proposal: [Discuss] [Core] Storage adaptor for Dashboards #413](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/413) - [[Epic] Dashboards Metadata Storage Decoupling #1441](https://github.com/opensearch-project/opensearch-dashboards/issues/1441) #### (1.3) Datasource Adapter Dashboards is designed to work with OpenSearch as the only data source, the versions must match, and it can’t communicate outside the cluster. If an administrator needs to view data from different sources, they need to use their respective monitoring software. For example, if the administrator needs to view both OpenSearch logging data and AWS CloudWatch metrics, they need to use both Dashboards and the AWS Console. We will create the capability for Dashboards to securely query data from multiple data sources that support OpenSearch DSL and APIs. Our first priorities will be the ability to query OpenSearch 3.x, 2.x, 1.x. We will build the adapter to be extensible so users who need to connect to other disparate data sources can leverage our platform. Once Dashboards can securely connect to these endpoints, we’ll provide users the ability to create a dashboard that composite the multiple data sources into one view, moving us closer to a “Single Pane of Glass'” future. - [[RFC] Enable OpenSearch Dashboards to support multiple OpenSearch clusters #1388](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1388) #### (1.4) Dashboards Desktop Download Dashboards to your computer, configure your connections to your data sources, and enjoy all the same benefits of a cloud visualizations app experience locally. - [[PROPOSAL] Dashboards Desktop MVP #3](https://github.com/opensearch-project/dashboards-desktop/issues/3) ### (2) Dashboards Visualizations and Experiences The Dashboards team now includes core Visualizations and Experiences. We are working closely with both product management and design to explore where we can make the most meaningful impact to usability, accessibility, and internationalization/localization. Today, many of the experiences, especially when plugins are involved, are disjointed and inconsistent. Ultimately, we want to reduce the number of steps it takes for customers to complete their tasks and delight them along the way. #### (2.1) Easy Start and Configuration The current out-of-box-experience (OOBE) was created for commercial up-selling and not designed for ease of getting started or making configurations. There is no integrated experience for users to enable and configure plugins. In order to install a new plugin you have to dive into the filesystem to make configurations. We will provide the ability to surface all configurations in a centralized UI with the community in mind. Both application and plugin capabilities can be easily enabled and customized. Additionally, we'll provide an OOBE to get users up and running quickly, which includes comprehensive tutorials with templates for both developers and non-developers. #### (2.2) Drag and Drop Today, visualization and exploration in Dashboard is harder than it needs to be. Users need to preselect both the chart type and index pattern they will use to create a visualization. Additionally, users may need to navigate between multiple screens (e.g. discovery → visualize → discover → visualize, etc.) to find the right feature, the right chart, and the right index to include in their dashboard. Not only is this more work then it needs to be, it prevents the serendipitous discovery that more accessible exploration allows. Our proposal is to allow users of Dashboards to create data visualizations and gather insights without preselecting the visualization output and with the flexibility to change visualization types and index patterns on the fly. Users would be able to switch visualization types, index patterns, and data fields quickly and easily in a simple Drag & Drop experience. More specifically, we are proposing a number of key features for OpenSearch Dashboard Drag & Drop: (1) Users can create visualizations by dragging a data feature or field and dropping them on the visualization canvas. (2) Users can change the data features and the visualization within the Drag & Drop CX. Eventually this will also include the index pattern. (3) Users will be able to see suggested visualization based on the selected data features. The Drag & Drop experience we're proposing would be created as a first party plugin (i.e. within the Dashboards repo and part of the default Dashboard experience) and accessible via the “New Visualization” option within the Visualize page. - [Drag & Drop Visualization CX #883](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/883) #### (2.3) Accessibility Accessibility means that everyone who uses a product can receive the same benefit, regardless of any condition or disability they may have. For OpenSearch Dashboards, accessibility will focus on identifying barriers that exist in the product and working to remove them. OpenSearch Dashboards will seek to be accessible to, and delightful for, all of our customers, specifically including the ~15% of the world population that has a significant disability. This includes accessible color palettes, compatibility with screen readers, meaningful keyboard navigation, and adherence to visual design guidance. #### (2.4) Internationalization OpenSearch Dashboards is already being used globally, in many countries and languages. Our north star is a localized experience in the customer’s language of preference with a design following the language conventions. #### (2.5) EUI & Charts Elastic UI (EUI) and Elastic Charts are the UI and Visualization frameworks that comprise the bread-and-butter of the core Dashboards experience. While there was room for improvement, they have clear contribution guidelines, component development guides, component design docs, and style guides. Since the latest versions of these libraries are no longer Apache 2.0 licensed, we have forked their respective repositories and started folding them into the Dashboards repository. As a stepping stone, we will upgrade Dashboards to use the latest Apache 2.0 versions of EUI and Charts to address known security issues and stabilize any breaking changes. The decisions on rename/branding these libraries is TBD. - [[RFC] Charts & EUI: Forking vs Merging-in vs Folding-in #695](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/695). #### (2.6) Dynamic Loading of Extensions Currently plugins are tightly coupled with Dashboards making it complex to independently ship enhancements. Running plugin as a separate process will uncover advantages like dynamic loading of plugins. We should adopt a "VSCode Extensions" model. Details on this are still being researched, but we know this will be critical. #### (2.7) Asynchronous Search Searching large volumes of data can take a long time, especially if you’re searching across warm nodes or multiple remote clusters. Asynchronous search in OpenSearch lets you send search requests that run in the background. You can monitor the progress of these searches and get back partial results as they become available. After the search finishes, you can save the results to examine at a later time. - [Support for integration with asynchronous search #1009](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1009) ## 4. Next Steps There’s a lot on this list of goals and realistically they may take multiple years to fully achieve them. But we look forward to making big leaps ahead on all of them in 2022 with you. Over the year, we will also continue to add links to this post for projects, so please watch the post if one of them interests you.

github.com

opensearch-project/ml-commons/blob/main/docs/how-to-add-new-function.md

# How to add new function

This doc explains how to add new function to `ml-commons` with two examples.

## Example 1 - Sample ML algorithm with train/predict APIs
This sample algorithm will train a dummy model first. Then user can call predict API to calculate total sum of a data frame or selected data from indices.

### Step 1: name the function
Add new function name in `common/src/main/java/org/opensearch/ml/common/FunctionName.java`
```
public enum FunctionName {
    ...
    //Add new sample algorithm name
    SAMPLE_ALGO,
    ...
}
```

### Step 2: add input class 
Create new class `common/src/main/java/org/opensearch/ml/common/input/parameter/sample/SampleAlgoParams.java` by implementing `MLAlgoParams` interface.

This file has been truncated. show original

Support ML node

github.com/opensearch-project/OpenSearch

Support ML node role

opened 08:30PM - 12 Apr 22 UTC

ylwu-amzn

enhancement discuss feature distributed framework

OpenSearch doesn't support ML node role currently, check [DiscoveryNodeRole.java…](https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/cluster/node/DiscoveryNodeRole.java), no ML role defined. ## Why asking for dedicated ML node We have released a new ML plugin `ml-commons` in [1.3](https://github.com/opensearch-project/ml-commons/releases/tag/1.3.0.0) and we are planning to adding more models. ML model generally consuming more resources, especially for training process. We are going to support bigger ML models which might require more resources and special hardware like GPU. As OpenSearch doesn’t support ML node, we dispatch ML task to data node only. That means if user want to train some big model, they need to scale up all data nodes which seems costly and not reasonable. If we can support dedicated ML node, user don’t need to scale up their data node at all, just need to configure a new ML node(with different settings, more powerful instance type) and add it to cluster. And we can separate resource usage better by running ML task on dedicated node which can reduce impact to other tasks like search/ingestion. ## What changes we need to make ### OpenSearch * Add new node type ml in [DiscoveryNodeRole.java](https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/cluster/node/DiscoveryNodeRole.java) * Support configuring ML node role in opensearch.yml * Security: should be ok as ML node communicate with other nodes via transport action. ### ML plugin * If ML node exists, dispatch ML task to ML node; otherwise, dispatch to data node.

Support custom model

github.com/opensearch-project/ml-commons

[FEATURE] Proposal - Deep Learning Model Uploading and Inference

opened 02:27AM - 26 Apr 22 UTC

jngz-es

feature

## What / Why ### What are we building? The ML-Commons plugin allows user …to upload deep learning models and use them to make inference. ### Why are we building it? In many domains like NLP, computer vision etc. the deep learning algorithms overwhelm the traditional machine learning algorithms. For OpenSearch customers who are planning to use machine leaning algorithms or using some traditional machine learning algorithms for their bussiness, deep learning is definitely a good way to improve their systems. ### How do we know? Since recent 8.0 release, Elastic supports uploading/inference pytorch model for NLP. https://www.elastic.co/guide/en/machine-learning/8.1/ml-nlp-overview.html ### What matters most to OpenSearch? Bring deep learning models into OpenSearch. Deep learning is increasingly affecting our lives. More and more customers want to use deep learning technology to improve their systems. This feature brings an opportunity for customers to apply deep learning models in OpenSearch for their business. ### What does the customer experience look like? 1. Train the model through a deep learning framework, like Pytorch. 2. Save the model to TorchScript. 3. Upload the model through ML-Commons plugin API. 4. Run the predict API provided by ML-Commons plugin to get the result. ### High-level user stories: - As a forecasting user, I want to apply deep learning algorithm like deepAR so that I can improve the forecasting accuracy for my business. - As a NLP user, I want to apply deep learning algorithms so that I can leverage the state-of-the-art NLP technologies to improve my systems. ### What are the risks and assumptions? #### The format of model There are pretty many deep learning frameworks to train models, like PyTroch, TensorFlow, MXNet etc. They are not compatible for the model’s format. We could start with TorchScript and ONNX as PyTorch is a very popular deep learning framework and ONNX is a standard format across various deep learning frameworks. #### The computation cost We have the limitation on the size of model. Supporting distributed inference or not need to be considered. #### Dedicated ML node with GPU If no dedicated node, we should have more restrict limitation on resources for the models. #### ARM support Support like AWS Graviton.