kris
July 20, 2022, 8:49pm
1
OpenSearch Community Meeting - 2022-0816
Event page Meetup page
Meeting Link
Meeting ID: 816 5511 8321
Would you like to present? Tag @kris @dtaivpp @nateynate and we’ll work to get you added to the agenda!
Feel free to comment on this agenda before the meeting if you want to add an item or have a question.
After the meeting, we will post the chat log and any meeting notes. We welcome you to keep the conversation going here on the forum.
========By joining the OpenSearch Community Meeting, you grant OpenSearch, and our affiliates the right to record, film, photograph, and capture your voice and image during the OpenSearch Community Meeting (the “Recordings”). You grant to us an irrevocable, nonexclusive, perpetual, worldwide, royalty-free right and license to use, reproduce, modify, distribute, and translate, for any purpose, all or any part of the Recordings and Your Materials. For example, we may distribute Recordings or snippets of Recordings via our social media outlets.
kris
August 16, 2022, 11:05pm
3
Chat log:
14:55:29 From Kris Freedain to Waiting Room Participants:Graylog Go GitHub - opensearch-project/opensearch-clients: For all things about OpenSearch clients. Carrying forward the OpenSearch client libraries as a community · OpenSearch Moving from open source Elasticsearch to OpenSearch · OpenSearch OSS 7.10.2 → should be good through all of 1.xElasticsearch to OpenSearch Migration: Prerequisites Compatibility - OpenSearch documentation @kris https://github.com/opensearch-project/OpenSearch-Dashboards/issues/896 https://playground.opensearch.org/ ← try it
kris
August 16, 2022, 11:09pm
4
Slides:
OpenSearch is a community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.
Graylog is a leading centralized log management solution for capturing, storing, and enabling real-time analysis of terabytes of machine data.
Est. reading time: 10 minutes
https://opensearch.org/blog/community/2022/05/opensearchcon/
opened 06:10PM - 10 Aug 22 UTC
enhancement
This is a meta issue tracking automation of release process for all clients
-… [x] `opensearch-go` - https://github.com/opensearch-project/opensearch-go/issues/144
- [x] `opensearch-rs` - https://github.com/opensearch-project/opensearch-rs/issues/63
- [x] `opensearch-py` - https://github.com/opensearch-project/opensearch-py/issues/185
- [x] `opensearch-dsl-py` - https://github.com/opensearch-project/opensearch-dsl-py/issues/67
- [ ] `opensearch-php` - https://github.com/opensearch-project/opensearch-php/issues/77
- [x] `opensearch-ruby` - https://github.com/opensearch-project/opensearch-ruby/issues/96
- [x] `opensearch-js` - https://github.com/opensearch-project/opensearch-js/issues/274
- [x] `opensearch-net` - https://github.com/opensearch-project/opensearch-net/issues/79
- [x] `opensearch-java` - https://github.com/opensearch-project/opensearch-java/issues/349
opened 11:51AM - 27 Apr 22 UTC
meta
**What kind of business use case are you trying to solve? What are your requirem… ents?**
There are [9 language clients for OpenSearch in this organization](https://github.com/opensearch-project/opensearch-clients/blob/main/clients/.meta). Each client is significantly different from another in their implementation (e.g. different pagination support or transport layer), but implement the exact same API (e.g. creating an index). That API matrix is huge, including multiple versions of OpenSearch and plugins that bring their own API.
**What is the problem? What is preventing you from meeting the requirements?**
The maintenance of clients is a large burden. Every time a new API is added or changed, every client needs to be updated, along with tests.
**What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?**
Automate the update process by auto-generating non-interesting parts of the client.
- [ ] https://github.com/opensearch-project/OpenSearch/issues/3090.
- [ ] Generate non-interesting parts of each client based on the API specification with templates
- [ ] Implement additional layers in the client such as pagination or Sigv4 signing
- [ ] Add support for consuming multiple versions of the API spec in clients
- [ ] Automate API updates via GHA, such as every time the specification changes, a PR is raised for each client
As an example, https://github.com/slack-ruby/slack-ruby-client is built this way. The API specification is scraped from Slack in https://github.com/slack-ruby/slack-api-ref and the client uses templates for about 75% of its code.
opened 07:57PM - 18 Jul 22 UTC
enhancement
## What kind of business use case are you trying to solve? What are your require… ments?
As of the time of writing this we have at least some integration testing across all clients (except .NET), but it's unclear how much surface those tests cover. They should be hitting 100% of APIs supported.
**What is the problem? What is preventing you from meeting the requirements?**
Lack of code coverage.
**What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?**
Setup code coverage for all clients. Once reporting from CI/CD, analyze the test gap and fill it.
opened 06:50PM - 12 Jul 22 UTC
enhancement
discuss
**Is your feature request related to a problem?**
The clients need to be smarte… r espl when the system is overloaded and traffic pattern needs to be adjusted to minimize downtimes giving sufficient time for the system to recover. The clients need to ensure that they maximum server uptime without sacrificing too much on performance
**What solution would you like?**
1. Handling too many client connections
2. Preventing retry storm
3. Managing/safe-guarding critical system resources
4. Adaptive timeouts and backoffs
5. Penalizing bad connections/erroneous targets
6. Load balancing across targets
7. Automatic pagination on larger response
8. Asynchronous searches on slower requests
9. Query prioritisation
10. Client-side metrics on data sources like AWS Cloudwatch
**What alternatives have you considered?**
A clear and concise description of any alternative solutions or features you've considered.
**Do you have any additional context?**
Add any other context or screenshots about the feature request here.
opened 04:32PM - 21 Apr 22 UTC
enhancement
discuss
**What kind of business use case are you trying to solve? What are your requirem… ents?**
Coming from https://github.com/opensearch-project/opensearch-clients/issues/12, clients are being upgraded to support OpenSearch 2.0 while breaking compatibility with 1.0. For example, opensearch-java 1.0 works against OpenSearch 1.x, opensearch-java 2.0 works against OpenSearch 2.x, but there's no version of client that works against both OpenSearch 1.x and 2.x. Thus, in order to upgrade from OpenSearch 1.x to 2.x one needs to either run a cluster in mixed mode for a long time, or take downtime.
With rolling upgrades:
1. Upgrade some nodes to 2.0, cluster is now in mixed state.
2. Upgrade all clients to 2.0 and direct trafic from these clients to 2.0 nodes.
3. Finish upgrading the cluster to 2.0.
**What is the problem? What is preventing you from meeting the requirements?**
Users have to upgrade both clients and server at the same time and/or run in mixed cluster mode for a very long time. This is a lot of work, hard to coordinate, risky, etc. Most users will choose not to upgrade across major versions.
**What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?**
Ensure that there's at least 1 version of a client that works against two major versions. For example, opensearch-java 2.0 would work against 1.x and 2.0 and deprecate all 1.x methods, and remove them in 3.0.
**Describe alternatives you've considered**
[REST API versioning](https://github.com/opensearch-project/OpenSearch/issues/3035) so users can specify their desired API version to ensure client compatibility is independent of server upgrades.
**Other**
This is tangentially related to https://github.com/opensearch-project/OpenSearch/issues/3023.
opened 03:29AM - 31 Mar 22 UTC
closed 09:53PM - 19 Sep 22 UTC
enhancement
Below are some of the things that can be done to maintain the general health of … the clients.
- [x] Add a `DEVELOPER_GUIDE` to make it easy for the community to get started with the client. Refer [DEVELOPER_GUIDE.md](https://github.com/opensearch-project/opensearch-java/blob/main/DEVELOPER_GUIDE.md)
- [x] Add a `TESTING_GUIDE` for client development.
- [x] Follow branching standards described in https://github.com/opensearch-project/.github/blob/main/RELEASING.md#branching.
- [x] Add auto backport functionality to maintain development across multiple branches easily. Refer [BACKPORT.md] https://github.com/opensearch-project/opensearch-clients/issues/14(https://github.com/opensearch-project/opensearch-plugins/blob/main/BACKPORT.md)
- [x] Add a compatibility matrix for the client. Refer [COMPATIBILITY.md](https://github.com/opensearch-project/opensearch-java/blob/main/COMPATIBILITY.md) - #8
- [x] Support tests running across compatible versions of OpenSearch for the current client version. - #8
- [x] Add ability to test against developing unreleased OpenSearch to make required changes to the client. This will help in developing and maintaining clients according to changes to OpenSearch.
- [x] Add broken link checker. Refer https://github.com/opensearch-project/opensearch-plugins/blob/main/.github/workflows/links.yml
- [x] Add CVE scanning for the client.
- [x] Add Dependabot auto upgrade to keep the dependencies up-to-date.
- [x] Add standard issue/feature templates from .github.
opened 12:37AM - 30 Oct 21 UTC
closed 02:47AM - 15 Jan 22 UTC
enhancement
discuss
ux / ui
vis builder
## Background
Today, visualization and exploration in OpenSearch Dashboard is… harder than it needs to be. Users need to preselect both the chart type and index pattern they will use to create a visualization. Additionally, users may need to navigate between multiple screens (e.g. discovery → visualize → discover → visualize, etc.) to find the right feature, chart, index to include in their dashboard. Not only is these more work then it should be, it prevents the serendipitous discovery that easy exploration allows. This document outlines the high level design recommendation for a visualization experience that makes exploring data and creating visualizations much easier.
## Tenets
* Minimal learning curve
* Easy of use
* No disruption to existing user workflow
* Get feedback early
* Progress not perfection
## Out of Scope
* This design only covers the initial version of the tool and should not be seen as final
* This is only a high level design and doesn’t include component-level design
## Recommended Design
The recommendation is to create the Drag and Drop experience as a first party plugin within the Opensearch Dashboards repository (i.e. explain to layman). It will act as an additional way to create visualizations alongside the exiting tools within the current visualizations plugin. The tool will be incremental to the visualization tools available to the user in OpenSearch Dashboards today. Due to the large nature of the design, I will divide it into 4 major sections:
* The user flow
* Tool features
* Architecture
* Aggregation logic
While we considered many options for each of our choices, we ultimately recommended this approach so that we could have minimal disruption to existing users workflow while also providing a seamless experience to the users who wish to create visualizations easily. It also lets us iterate and evolve over time.
### The user flow
There are 3 main ways the user can reach the new tool.
1. Through the “New Visualization” option within the Visualize page.
2. Via the edit option for a visualization created by this tool in the dashboard
3. From the saved object viewer
For each of these flows the user is navigated to a full page view of the drag and drop tool that lets the user create or edit the visualization. After creating the visualization, the user can either choose to save the visualization or add the visualization directly to the preferred dashboard (existing or new).
**Alternatives**
1. **Upgrade the existing visualize plugin to support the drag and drop:**
This approach is to update the exiting visualize plugin to be more easy to use and support some of the features this design proposes.
1. **Pros:**
1. No need to maintain two plugins
2. **Cons**:
1. Limits the scope of the features we can introduce. Some features such as the need to select the the visualization type before hand or the need to know the type of aggregation first go against the features introduced by this design. Changing it to align with the design breaks the experience for users that rely on the current experience.
### Tool features
This tool features:
1. **Top Nav:** To search for data within an index pattern. This is identical to the top nav element within the Discover plugin and allows the user to search and filter the index pattern data for available fields and its data. It also allows the user to save the visualization as a saved object or save and add it directly to a dashboard.
2. **Data** **Panel**: A panel for accessing the Index pattern and its available fields. The data panel will feature an index pattern selector, a search bar to filter all the fields available within the index pattern for the given Top Nav filters, and the index pattern fields broadly subdivided (e.g. categorical fields, numeric fields and meta fields). The main features of this panel are:
1. **Index pattern selector**: To select the index pattern
2. **Field filter**: Filter the list of available fields. Helpful if the list of available fields is very large.
3. **Field summary:** Each field, when hovered over or clicked, will show us a summary of the data in the field, similar to the discovery experience.
4. **Add to canvas:** There will also be a button within each field to add the index pattern to the canvas.
3. **Chart type selector**: To allow the user to switch chart types on the fly, we will have a chart type selector. The initial implementation will prioritize bar (vertical and horizontal + stacked), line, area, and pie as well as metric and data table.
4. **Visualization Config Panel**: The panel is where the fields to visualize are added. It also has options to configure the type of aggregation for a field and serves as drop targets for the draggable fields.
5. **Visualization Style Editor**: A panel to edit the overall styles of the visualization such as the title, axis labels, legend, etc.
6. **Visualization Canvas**: The canvas serves as an area to see the final visualization. It also doubles as a drop target for the different draggable fields if a suitable aggregation is possible.
7. **Drag and drop**: All fields available in the Data Pane can be dragged and dropped into either the Visualization Canvas or the Visualization config panel. Doing so triggers an appropriate aggregation call described in the Aggregation section.
### Architecture
**Workflow**
The app uses the fields selected or dropped into the chart context by the user to determine the best aggregation logic (outlined in a later section) to generate the appropriate DSL Query(s) and fetch the data to render the visualization.
**Saved Object & Embeddable**
The tool will introduce a new saved object type and embeddable to render its visualizations. This means that existing visualization will not be compatible with the new tool and separate factory. This is done to prevent breaking existing visualizations while allowing new visualizations to be created easily without much overhead.
Saved object structure
```
const wizardApp: SavedObjectsType = {
name: WIZARD_SAVED_OBJECT,
hidden: false,
// ....
migrations: {},
mappings: {
properties: {
title: {
type: 'text',
},
description: {
type: 'text',
},
// Contains the state of the visualizaion
state: {
type: 'text',
index: false,
},
},
},
};
```
The one big drawback to this approach is not having the ability to edit existing visualizations using the new experience. Since the existing experience has a lot of visualization types that it already supports and a lot of customizations that we will not support for the initial launch and in future, reusing the existing schema will be rather tedious.
If the ability to edit existing visualizations is important, we can optionally provide an api/interface to selectively convert or migrate exiting and compatible visualization saved objects to the new saved object type. This will duplicate the existing visualization to the new format so that if there are any issues with the migration, the existing visualization remains unaffected. This approach also allows us to selectively enable migration for the visualizations that can be safely supported.
**Alternatives**:
**Reuse existing visualization schemas:** This approach lets us keep the existing schema and all new visualizations created using the new experience uses the existing `visualization` saved object type and embeddable. Supported visualization types will open the new experience when edited while the old experience is used for unsupported visualizations. This allows existing visualizations to use the new experience easily and allows wider adoption with no migration required, but not all options supported by the existing experience will be supported in the new experience. This may break workflow for current users. Additionally, this would break existing visualizations when edited and some existing visualizations may never be supported by the new experience (e.g. Vega). Lastly, inheriting a schema that is not optimized for the current experience may limit future expansion.
**Charting Library**
The app will use the EUI Charts library to create the default supported visualizations. This to to maintain visual parity with existing visualization while changing how they are generated. In future we expect all visualization to be created using this app.
While the default visualization types use EUI, there is no restrictions on what charting library can be used since each visualization type will be responsible for rendering itself and defining the fields and config types it supports it need to render correctly. In future this method can be used by other plugins to register their own visualization to exist alongside the default visualization types.
**Drag and Drop**
For the initial release there will only be 2 primary drop targets:
1. Visualization Config Panel
2. Visualization Canvas
The field cards can be dragged from the Data panel into either drop target and the page will automatically provide context to the user based on where the user is dragging the field.
Based on the dropped region an appropriate aggregation strategy will be used to fetch the data and render a visualization is possible. When a particular field cannot be dropped into a particular region, the drop cue does not display.
### Aggregation Logic
One of the key problems we will need to address with the tool is how different interactions render a corresponding visualization that is both useful and makes sense to the user. Each existing visualization tool out there today does this in their own way and have their own strengths and drawbacks. To simplify this problem for our initial release and to get better user feedback on the approach we need to take here, my recommended is to allow the user to only be able to drag and drop a field into the canvas for the first field they wish to visualize. The Visualization generated will be based on a simple lookup below.
|Field Type |Visualization Canvas |
|--- |--- |
|Numeric |median |
|Date |count of records (histogram) |
|String and others |count for top values |
The Visualization Config panel will update with the appropriate fields and aggregation functions and the Visualization canvas will no longer act as a drop target. All subsequent fields need to be added directly to the Visualization Config Panel which assumes an appropriate aggregation function for the Chart axis and field type (similar to the existing visualize tool today). The User will also still be able to configure the visualization by Dragging and dropping fields directly onto the Visualization Config Panel. This approach makes the initial discovery of the data easy while keeping the reason for why a particular visualization was rendered intuitive.
**Alternatives**
1. **The smart alternative:** Similar to Tableau and Lens, we can allow the user to drop in any valid combination of fields into any valid drop target to generate an appropriate visualization. The biggest challenge here is to breakdown the problem such that it makes intuitive sense to the user why a particular visualization was rendered given the fields they dropped in. Given the number of drop targets, field types and aggregation functions Opensearch supports, the number of permutations quickly explode. Hence this approach is not recommended for the first iteration
2. **The super simple approach**: In this approach, we do not allow the user to drop anything into the canvas directly, but instead only allow them the pick the field they want to visualize directly in the Visualization config editor. This removes any ambiguity about what field and aggregation is used to render the visualization. It is not ideal since the user will now have to rely on the field summary in the data panel to explore the data, but it is still better than the experience today since the rest of the tool still makes intuitive sense.
The recommended solution is a meets both these approaches in the middle.
## Future scope
1. Replace the most existing visualization tools and make this the default
2. Allow integration for other query languages
3. Migrate existing visualizations to the new tool
4. Ability to drag and drop any field into the canvas to create visualizations
5. Realtime smart suggestions for visualizations
## Open Questions
1. Should we prioritize converting visualizations to work between the current visualizations experience and the proposed experience?
1. Is the ability edit existing visualizations using the new experience more important?
2. Is the ability to edit a visualization created using the proposed experience with the current experience important?
3. Are they both important?
2. Visualization types to prioritize:
1. Bar
2. Line
3. Metric
4. Data
5. Area
6. Pi
3. How can we provide a preview or beta.
4. Is there anything in the future scope that we should add to the initial release
5. Any core feature we have missed for the initial iteration
6. Should this be a part of [Opensearch Dashboards](https://github.com/opensearch-project/OpenSearch-Dashboards), the [Dashboards-visualizations plugin](https://github.com/opensearch-project/dashboards-visualizations), or be a separate plugin
Agenda:
Juju & charmed operators: Open source in a hybrid & multi-cloud world - Pedro Leao da Cruz, Product Lead, Canonical
Product walk-through of Highlight - Jay Khatri, Founder, Highlight
Event page
Meetup page
Meeting Link
Meeting ID: 856 3904 9172
Passcode: 863809
Would you like to present? Tag @kris @dtaivpp @nateynate and we’ll work to get you added to the agenda!
Feel free to comment on this agenda before the meeting if you want to add an item or have a question.
After the mee…
