OpenSearch Community Meeting - 2022-0927

kris · August 17, 2022, 1:25am

Agenda:

the team at Tracetest present their OpenSearch integration - @revenaugh
feedback on the security issue response process - @davelago
Forecasting Dev Ops Metrics - @xenia

Event page
Meetup page

Meeting Link

Meeting ID: 863 9160 0375
Passcode: 330208

Would you like to present? Tag @kris @dtaivpp @nateynate and we’ll work to get you added to the agenda!

Feel free to comment on this agenda before the meeting if you want to add an item or have a question.

After the meeting, we will post the chat log and any meeting notes. We welcome you to keep the conversation going here on the forum.

========
By joining the OpenSearch Community Meeting, you grant OpenSearch, and our affiliates the right to record, film, photograph, and capture your voice and image during the OpenSearch Community Meeting (the “Recordings”). You grant to us an irrevocable, nonexclusive, perpetual, worldwide, royalty-free right and license to use, reproduce, modify, distribute, and translate, for any purpose, all or any part of the Recordings and Your Materials. For example, we may distribute Recordings or snippets of Recordings via our social media outlets.

revenaugh · September 1, 2022, 12:32pm

Hi @kris @dtaivpp (and Nate)! We at Tracetest (http://tracetest.io) would like to present our OpenSearch integration at the community meeting.

kris · September 1, 2022, 3:25pm

welcome to the community @revenaugh! That’s great, I’ve added you to the agenda - looking forward to hearing all about Tracetest

revenaugh · September 26, 2022, 2:02pm

Hi @kris! Want to keep the presentation short but informational. How long should we shoot for the Tracetest demo?

kris · September 26, 2022, 3:39pm

Hello @revenaugh - Each presenter usually has around 10-15 minutes
We can allow questions during or after – then we usually ask presenters to stay around for the rest of the meeting in case people ask questions afterwards

Also, we ask presenters to show up about 15 minutes early so we can test Zoom and make sure you can share your screen, etc. I’ll send you a meeting invite this morning.

kris · September 27, 2022, 5:20pm

Chat log:
08:00:40 From Kris Freedain to Everyone:
Good morning everyone
08:00:48 From Nate B. to Everyone:
Hi Crew!
08:00:58 From ken to Everyone:
Mornin!
08:01:06 From Andriy Redko to Everyone:
Good Morning!
08:01:10 From Kelly Revenaugh to Everyone:
Good morning!!
08:01:20 From Lukáš Vlček (@Aiven.io) to Everyone:
Hello!
08:01:32 From Xenia Tupitsyna to Everyone:
Hi everyone!
08:01:45 From Kris Freedain to Everyone:
We’ll get going in just a couple minutes
08:02:56 From Ryan Paras to Everyone:
welcome to jungle:musical_note:
08:02:58 From Chris to Everyone:
May I know the name of the youtube channel this will be in?
08:02:59 From Kelly Revenaugh to Everyone:
lol
08:03:21 From Nate B. to Everyone:
YouTube channel is “OpenSearch”
08:03:24 From Kris Freedain to Everyone:
https://www.youtube.com/c/OpenSearchProject
08:03:28 From Hannah Uselman to Everyone:
Morning everyone!
08:05:48 From Kelly Revenaugh to Everyone:
Hannah is the most intimidating bouncer!
08:06:32 From Kris Freedain to Everyone:
OpenSearchCon was amazing - it was great to see the community!
08:06:50 From Andriy Redko to Everyone:
OpenSearchCon was an absolute success!
08:06:53 From Hannah Uselman to Everyone:
Someone has to do it💪
08:06:54 From Kris Freedain to Everyone:
OpenSearch 2.3.0 is ready for download! · OpenSearch
08:06:59 From Xenia Tupitsyna to Everyone:
OpenSearchCon was indeed awesome!
08:07:07 From Alejandro Guida to Everyone:
Àre the presentations of OpenSearchCon going to be shared online? Because I could’t attend,
08:07:10 From Kris Freedain to Everyone:
Release notes: opensearch-build/opensearch-release-notes-2.3.0.md at main · opensearch-project/opensearch-build · GitHub
08:07:32 From Kris Freedain to Everyone:
Yes. We do plan to post the recordings from OpenSearchCon
08:08:11 From Kris Freedain to Everyone:
https://www.youtube.com/c/OpenSearchProject
08:08:36 From Kris Freedain to Everyone:
Subscribe and you will be notified as new content is posted
08:09:06 From Kris Freedain to Everyone:
Document Security Issue Response Process by davidlago · Pull Request #90 · opensearch-project/.github · GitHub
08:09:28 From Nate B. to Everyone:
There’s no bar. Don’t worry folks.
08:09:28 From Hannah Uselman to Everyone:
Follow us on Twitter or LinkedIn for the latest update on OpenSearchCon recordings. @OpenSearchProj
08:10:09 From Kris Freedain to Everyone:
Documenting our security issue response process · OpenSearch
08:11:12 From Kris Freedain to Everyone:
Document Security Issue Response Process by davidlago · Pull Request #90 · opensearch-project/.github · GitHub
08:12:13 From Alejandro Guida to Everyone:
Thanks, perfect!
08:13:31 From Kris Freedain to Everyone:
TraceTest!
08:13:32 From Kris Freedain to Everyone:
https://tracetest.io/
08:15:57 From Kelly Revenaugh to Everyone:
Tracetest is an open source tool. The website Kris shared has blog posts and resources as well as our Github repo: GitHub - kubeshop/tracetest: Tracetest - the best way to develop and test your distributed system with OpenTelemetry
08:16:27 From Kelly Revenaugh to Everyone:
And the Tracetest / OpenSearch integration: tracetest/examples/tracetest-opensearch at main · kubeshop/tracetest · GitHub
08:16:48 From Kris Freedain to Everyone:
Excellent - thank you Kelly
08:22:08 From tonys to Everyone:
Can you use it to load test?
08:22:47 From Lukáš Vlček (@Aiven.io) to Everyone:
Tracetest: Pretty nice!
08:23:24 From Kelly Revenaugh to Everyone:
We just updated our installation process - so thank you!
08:23:46 From Tiffany M to Everyone:
does it work with AWS OpenSearch?
08:24:59 From Kelly Revenaugh to Everyone:
@tiffany - it does! We are working on a post to add to the OpenSearch blog - should be up soon once the PR gets merged.
08:25:16 From Arun Lakshmanan to Everyone:
Nice demo! Thanks.
08:26:51 From Kelly Revenaugh to Everyone:
We’re open to feedback too! Happy to take ideas and requests from users. You can find out Discord on the Tracetest website or submit requests via Github.
08:26:59 From tonys to Everyone:
Cool, thank you
08:27:06 From Kelly Revenaugh to Everyone:
Hope to see some of you at Kubecon next month too - and next year at Opensearchcon!
08:27:27 From ken to Everyone:
Nice presentation Matheus!
08:27:33 From Matheus Nogueira to Everyone:
thanks
08:27:41 From Xenia Tupitsyna to Everyone:
Forecasting proposal: [Proposal] Forecasting Dev Ops Metrics and Financial/Inventory Metrics · Issue #408 · opensearch-project/ml-commons · GitHub
08:29:08 From ken to Everyone:
Discord channel for any other Tracetest questions: Discord
08:30:15 From tonys to Everyone:
Not yet
08:30:20 From vi4life to Everyone:
No
08:30:20 From Alejandro Guida to Everyone:
Not yet
08:30:21 From Ryan Paras to Everyone:
forecasting - not eyt
08:30:23 From Ryan Paras to Everyone:
*yet
08:30:25 From Nate B. to Everyone:
I have a feeling I’m about to.
08:30:25 From Matheus Nogueira to Everyone:
no
08:30:26 From Kelly Revenaugh to Everyone:
Not yet either
08:30:31 From Samuel Herman to Everyone:
+1 Not yet, but if it was will be used for operational purposes
08:30:38 From Shiwan Tribbett to Everyone:
no
08:32:55 From Joshua Bright to Everyone:
How much data would you have to work with Samuel?
08:36:11 From tonys to Everyone:
We use 5 second intervals on metricbeat for node and cluster stats.
08:36:50 From Nate B. to Everyone:
I believe it’s ml-commons based forecasting - AI.
08:46:45 From tonys to Everyone:
I like the look of the predictive alerts.
08:53:26 From tonys to Everyone:
We would look at using doc count, load and disk space.
08:54:23 From tonys to Everyone:
We just use alerting for simple thresholds for load and disk mainly.
08:55:09 From Samuel Herman to Everyone:
Great discussion and thank you so much for checking with community feedback! I have to drop for another meeting. See ya soon
08:55:53 From Kris Freedain to Everyone:
Thanks Samuel!
08:58:41 From Ryan Paras to Everyone:
must drop - thank you!
09:06:36 From Andriy Redko to Everyone:
Thank you guys!
09:07:03 From Hannah Uselman to Everyone:
WE WELCOME ALL DAD JOKES
09:07:24 From tonys to Everyone:
Thanks all
09:07:33 From Lukáš Vlček (@Aiven.io) to Everyone:
Thank you.
09:07:39 From Hannah Uselman to Everyone:
Bye folks!

kris · September 27, 2022, 5:26pm

Slides from today:

github.com

opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.3.0.md

# OpenSearch and Dashboards 2.3.0 Release Notes

## Release Highlights
OpenSearch and OpenSearch Dashboards 2.3.0 unlocks new approaches to data replication, storage, and visualization with three experimental features. These features are disabled by default and can be enabled per the [release documentation](https://opensearch.org/docs/latest).
* Segment replication offers users a new data replication strategy. With segment replication, OpenSearch copies Lucene file segments from the primary shard to its replicas, offering performance improvements for high-ingestion workloads.
* Remote-backed storage lets users deploy cloud storage for increased data durability. Users can back up and restore data from their clusters on a per-index basis using cloud-based storage solutions.
* A new drag-and-drop visualization tool lets users generate different types of visualizations more quickly and intuitively. Users can drag and drop data fields to generate line, bar, area, and metric charts.

## Release Details

OpenSearch and OpenSearch Dashboards 2.3.0 includes the following features, enhancements, bug fixes, infrastructure, documentation, maintenance, and refactoring updates.

OpenSearch [Release Notes](https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-2.3.0.md).

OpenSearch Dashboards [Release Notes](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/release-notes/opensearch-dashboards.release-notes-2.3.0.md).

## FEATURES

### OpenSearch SQL
* Add maketime and makedate datetime functions ([#755](https://github.com/opensearch-project/sql/pull/755))

This file has been truncated. show original

github.com/opensearch-project/.github

Document Security Issue Response Process

opensearch-project:main ← davidlago:security-issue-response

opened 02:36PM - 16 Sep 22 UTC

davidlago

+89 -2

Signed-off-by: Dave Lago <davelago@amazon.com> ### Description Besides makin…g the security issue response process transparent, this policy has a significant addition: **the creation of a pre-disclosure list**. Having a pre-disclosure list (a group of individuals and companies that will be informed of a security issue before it becomes public) is a risk balancing act. If a security issue is pre-disclosed to too broad of an audience, we risk it becoming public ahead of time. On the other hand, we might be missing out on collaborations that would expedite the creation and testing of a fix by not having a mechanism for involving some individuals and companies early, in addition to effectively creating zero-days for all the issues we disclose, leaving everyone scrambling to apply the fixes. This is just the first version of this process, and we most likely won't get it exactly right the first time. But with everyone’s help, we will keep iterating on it and advancing the security of our open-source project. ### Issues Resolved N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

github.com/opensearch-project/ml-commons

[Proposal] Forecasting Dev Ops Metrics and Financial/Inventory Metrics

opened 12:06AM - 23 Aug 22 UTC

brijos

enhancement Priority-High

### **What are you proposing?** * The current log analytics solution is great… at showing how things are performing in the moment. When looking at log metrics, users also want to understand where things are trending so that they can plan accordingly. Forecasting will show the user where their metrics are headed so that they can prepare accordingly (e.g. disk space, server instance upgrades, or even license usage if tracked). - Show a future view of time series metrics displayed on Dashboards which will feed into Observability and Security Analytics - Forecast when something may trigger an alert in the future to help users stay on top of their areas of responsibility - Create a workflow which allows someone to upload their dataset and validate that it is correct ahead of performing the forecasting job - Include “what if” scenario support which allow the user to work through scenarios like what if traffic increased 4x what would happen to my systems - Have popular forecasting dashboard templates to make it easy to setup - Easily share a forecasting dashboards with colleagues and the community - Tee up a scheduled report of a specific dashboard to be reviewed quarterly ### **What problems are you trying to solve?** There are two personas that we propose to solve for. The first priority is the developer operations persona who wants to view dashboards and understand where metrics are headed and the second priority is the business analyst persona. A key differentiator between the personas would be data size. The developer operations persona will likely only have 90-120 days worth of data available to them given that saving application logs is costly. The developer operations persona wants to make sure that the applications, service tiers, or services are up and fully functional. They live more in the moment and would like to be proactively notified if their area of responsibility is going to have problems. The second persona is the business analyst persona who wants to look at data over a longer period of time (120+ days). The data the business analyst is looking at has seasonality trends in it. The business analyst wants to understand how to project out financial metrics such as revenue or inventory levels. ### **User Stories** **Developer Operations Persona (90 - 120 days of data)** - As someone who is responsible for critical services, I want to look at my dashboard visualizations and forecast where metrics are moving, so I can understand if I can take proactive actions and change the outcome. - Display an augmented view of forecasting on a given visualization w ranges of confidence (e.g. as time or variability in the data increases confidence decreases) - As someone who wants to forecast way into the future, I want to be told that my attempts to forecast out that far are not realistic, so I can set the right expectations with myself. - As someone who is responsible for critical services, I want to setup select alerts to notify me if there are anticipated issues with the service, so I can proactively respond to problems instead of reacting - Given that someone has turned on forecasting for their monitor-alert, when an alert is triggered, it needs to sent in a way which differentiates it from other alerts (different urgency) - As someone who wants to manage monitors/alerts from an API, I want the ability to turn on Forecasting, so I don’t have to do things interactively - As someone who is responsible for critical services, I want to setup “what if scenarios”, so I can plan ahead and know if web traffic increases by 2x that my services are going to fall over. - As someone who prefers one forecasting algorithm over another, I want to choose which one to use as a default and through an advanced menu **Business Analyst Persona (120 days + of data)** - As someone who has a large dataset that they want to evaluate, I want to connect that dataset to Forecasting, so I can run a job on the dataset - What kind of datasets are going to be supported and what kind of help with be present in OpenSearch (differentiating between immediately uploaded datasets v. pipeline datasets) - As someone who has connected a large dataset, I want to visualize the dataset so that I can validate that the data I anticipate is in there, is in fact there ahead of forecasting - As someone who has validated the data is ready for forecasting, I want to choose and be informed of which forecasting algorithm to use, so I can forecast with confidence - As someone who has initiated the forecast, I want to view my results on a visualization which can be a part of a Dashboard. - As someone who wants to forecast way into the future, I want to be told that my attempts to forecast out that far are not realistic, so I can set the right expectations with myself. - As someone has evaluated the data, I want to share the results in CSV, so that I can manipulate them offline - As someone who has evaluated the data, I want to setup “what if scenarios”, so I can plan ahead and know if web traffic increases by 2x that my services are going to fall over. **Reporting** - Send out quarterly report to a distribution list to show the forecast of what is happening within the user’s realm of responsibility (e.g. services owned, service tiers owned, etc.) **Administration** - As someone who is concerned with performance, I want to make sure my forecasting workloads don’t interfere with critical OpenSearch ingestion/indexing. - As someone who is concerned with performance, I want to be able to control who has access to the forecasting feature in the system, so I can avoid business critical functions of the system tipping over. ### **Outstanding Questions** - Is the prioritization of personas correct? Any additional personas that we need to be mindful of? - What persona would be valuable from an API first perspective? - Would the community find it valuable to have a dedicated saved Forecasting view/state on their dashboards to return to in the future? - Forecasting is an overloaded term. Is there better naming that we can use to differentiate between the anticipatory results displayed to the developer operations specialist compared with the business analyst? - Forecasting can be helpful in ML Commons, Anomaly Detection, Alerting, Dashboards, and Performance Analyzer. Is there an opinion on how to implement Forecasting which would allow each plugin/extension the ability to take advantage of the feature? - What Forecasting algorithms would the community find interesting? - Would the community find forecasted alerting mechanisms helpful? - Would 'what if scenarios' would be interesting to the community? - Would the community value a SQL integration with Forecasting?

Topic		Replies	Views
OpenSearch Community Meeting - 2022-1025 Community community-meeting	1	897	October 25, 2022
OpenSearch Community Meeting - 2022-0802 Community community-meeting	6	1158	August 5, 2022
OpenSearch Community Meeting - 2023-0523 Community community-meeting	10	792	May 30, 2023
OpenSearch Community Meeting - 2022-0816 Community community-meeting , cve	5	960	August 24, 2022
OpenSearch Community Meeting - 2023-0117 Community community-meeting	7	915	January 17, 2023

Related topics