OpenSearch Community Meeting - 2023-0117

kris · December 20, 2022, 7:28pm

OpenSearch Community Meeting - 2023-0117 - Hope we see you all there!

Agenda:

Demo of migration tool (so far) - Chris Helma, Sr Engineer - OpenSearch (migration team)
Identity feature - intro, discussion, & feedback request - @peternied SDE - OpenSearch security
Sample plugin - test API to describe the available analyzers, tokenizers, and filters - @lukas-vlcek
Code Contributor Coins - reminder to reach out if you didn’t receive an email from us in Dec 2022.

Event page
Meetup page

Meeting Link

Meeting ID: 863 4082 8076
Passcode: 952752

Would you like to present? Tag @kris @dtaivpp @nateynate and we’ll work to get you added to the agenda!

Feel free to comment on this agenda before the meeting if you want to add an item or have a question.

After the meeting, we will post the chat log and any meeting notes. We welcome you to keep the conversation going here on the forum.

========
By joining the OpenSearch Community Meeting, you grant OpenSearch, and our affiliates the right to record, film, photograph, and capture your voice and image during the OpenSearch Community Meeting (the “Recordings”). You grant to us an irrevocable, nonexclusive, perpetual, worldwide, royalty-free right and license to use, reproduce, modify, distribute, and translate, for any purpose, all or any part of the Recordings and Your Materials. For example, we may distribute Recordings or snippets of Recordings via our social media outlets.

peternied · January 5, 2023, 10:12pm

@kris @dtaivpp @nateynate Is there space for ~15 minutes to introduce and discuss the Identity feature during the meeting on the 17th? We will have a blog post, GitHub meta issue and project plan along side to share as well as get feedback on, would love to spend a couple minutes introducing the high level vision and then take questions.

dtaivpp · January 5, 2023, 10:14pm

Sure is @peternied ! Looking forward to having you

kris · January 5, 2023, 11:09pm

Absolutely! Will add you to the agenda - thanks!

dtaivpp · January 7, 2023, 6:06pm

@kris lets also add this one by Lukáš. Should be relatively short. It’s a sample plugin that is a test for an API to describe the available analyzers, tokenizers, and filters.

github.com/opensearch-project/OpenSearch

Get list of all available analyzers. Request for a new API?

opened 05:06PM - 07 Dec 22 UTC

lukas-vlcek

enhancement Search

**Is your feature request related to a problem? Please describe.** I am missi…ng an option to get list of all available analyzers. There is already analyze API documentation and it mentions "built-in" analyzers. But for normal user there is no way how to learn what are all the options. And even for people who are familiar with the code the list is a subject to updates/changes. One option would be to document the list in the documentation page. But I think I would prefer if the cluster itself can give a list of its analyzers (and also tokenizers, charfilters, normalizers). - There are built-in analyzers at the cluster level (in fact the list is kept at the node level, see below) - There are also ad-hoc/custom analyzers at individual index level (but visibility of these should be subject of RBA rules?) **Describe the solution you'd like** As far as I understand list of all built-in analyzers is materialized once `AnalysisModule.setupAnalyzers(plugins)` is called. I think it would be useful to extend one of the `_nodes/` APIs and give it option to return list of all built-in analyzers (and its componenets: tokenizers, ... etc). (It needs to be API at the "nodes"-level because AnalysisRegistry is kept per node and I think a list of built-in analyzers can differ on each node depending on installed plugins.) As for the list of analyzers defined at the index level I am not sure at this point. Maybe later... **Describe alternatives you've considered** Alternative is to go to the documentation (which does not have this list) or go to the code (which is not an option for many people). **Additional context** n/a

kris · January 9, 2023, 7:43pm

Excellent - thanks @dtaivpp! @lukas-vlcek - we’ll be looking forward to it!

kris · January 17, 2023, 6:23pm

Thank you to all who attended!

Chat log:

07:49:58 From David Tippett to Everyone:
- Chris - GitHub - opensearch-project/opensearch-migrations: All things migrations and upgrades for OpenSearch
- Peter - [META] Identity in OpenSearch · Issue #5834 · opensearch-project/OpenSearch · GitHub
- Lukáš - Get list of all available analyzers. Request for a new API? · Issue #5481 · opensearch-project/OpenSearch · GitHub
07:52:03 From Peter Nied to Everyone:
David could you replace my link with [Security] OpenSearch should support authentication and authorization natively · Issue #4514 · opensearch-project/OpenSearch · GitHub
07:52:06 From Chris Helma to Everyone:
Tracking Issue - [META] Upgrade Testing Framework Issue List · Issue #10 · opensearch-project/opensearch-migrations · GitHub

Tool README - https://github.com/opensearch-project/opensearch-migrations/blob/main/upgrades/README.md

07:56:38 From Kris Freedain to Waiting Room Participants:
Hello Everyone - we’ll get going in just a few minutes
07:56:57 From Nate Boot to Everyone:
Ohboyohboyohboy!
08:01:26 From Romain to Everyone:
o/
08:01:39 From Peter Nied to Everyone:
Cleaned the house, nearly spring cleaning!
08:01:55 From George Bridgeman to Everyone:
Laser tag for my kid’s birthday. I had more fun than the children!
08:02:12 From Kris Freedain to Everyone:
Worked in the garden a little
08:02:37 From Nate Boot to Everyone:
Introduce yourself!
08:02:46 From Nate Boot to Everyone:

08:03:42 From David Tippett to Everyone:
[RFC] Search pipelines · Issue #80 · opensearch-project/search-processor · GitHub
08:04:14 From David Tippett to Everyone:
[RFC] Search pipelines · Issue #80 · opensearch-project/search-processor · GitHub
08:04:18 From Peter Nied to Everyone:
Or why you aren’t using the SQL Plugin!
08:04:34 From David Tippett to Everyone:
SQL A community call: share your mappings
08:04:58 From David Tippett to Everyone:
Finding the Right Things with OpenSearch, Tue, Jan 24, 2023, 7:00 PM | Meetup
08:05:17 From Daniel Doubrovkine to Everyone:
+ we’ve published some info on security reviews that we conduct before releasing software: .github/RELEASING.md at main · opensearch-project/.github · GitHub
08:05:58 From Nate Boot to Everyone:
Welcome Chris!
08:06:29 From David Tippett to Everyone:
Reacted to “+ we’ve published so…” with
08:06:31 From David Tippett to Everyone:
Reacted to “+ we’ve published so…” with
08:08:09 From Fernando Rejon Barrera to Everyone:
maybe it’s just me but I can’t see any slides (if there are any)
08:08:21 From David Tippett to Everyone:
Here is the tracking issue for OpenSearch Migrations - [META] Upgrade Testing Framework Issue List · Issue #10 · opensearch-project/opensearch-migrations · GitHub
08:09:06 From Fernando Rejon Barrera to Everyone:
never mind, I can see the screen now :+1
08:09:35 From Kris Freedain to Everyone:
Replying to “never mind, I can se…”

Chris was just doing an intro before - hadn’t shared his screen yet.

08:12:44 From Kris Freedain to Everyone:
Reacted to “+ we’ve published so…” with
08:13:48 From Amitai Stern to Everyone:
Wow!
08:14:50 From David Tippett to Everyone:
Here is the readme for the tool: opensearch-migrations/README.md at main · opensearch-project/opensearch-migrations · GitHub
08:15:00 From Kris Freedain to Everyone:
Reacted to “Here is the readme f…” with
08:18:04 From Mark Cohen to Everyone:
Reacted to “o (Osman) · GitHub…” with
08:18:28 From David Tippett to Everyone:
Here is the tracking issue for OpenSearch Migrations - [META] Upgrade Testing Framework Issue List · Issue #10 · opensearch-project/opensearch-migrations · GitHub
08:22:38 From Charlotte to Everyone:
Oh just rub it in for those of us drowning in rain, Peter
08:22:49 From David Tippett to Everyone:
Reacted to “Oh just rub it in fo…” with
08:22:54 From Nate Boot to Everyone:
I bet those blue clouds are really confusing.
08:23:03 From Kris Freedain to Everyone:
[Security] OpenSearch should support authentication and authorization natively · Issue #4514 · opensearch-project/OpenSearch · GitHub
08:23:30 From Kris Freedain to Everyone:
Peter is looking for feedback on that one
08:28:39 From Kris Freedain to Everyone:
Good blog post on plugins: Introduction to OpenSearch Plugins · OpenSearch
08:34:02 From Sam Herman to Everyone:
Thanks for the clarification DB!
08:34:25 From Daniel Doubrovkine to Everyone:
JSM = [RFC] Replace Java Security Manager (JSM) · Issue #1687 · opensearch-project/OpenSearch · GitHub
08:34:44 From David Tippett to Everyone:
hahaha
08:34:45 From Daniel Doubrovkine to Everyone:
Office office office office …
08:36:24 From Sander to Everyone:
We have a layer in front of the security plugin as well, to do ip filtering. We know which backend roles come from which ip’s, but opensearch doesn’t provide this so we put a Yarp (.net) proxy in front of opensearch. Would be nice if we had this out of the box and could remove the reverse proxy
08:42:42 From Stephen Crawford to Everyone:
Replying to “We have a layer in f…”

Hi Sander, it sounds like you are looking for a way to be able to natively determine the IP origin for different backend roles? Is that right? If you have not made an issue on the security repo about it then I would encourage that here: https://github.com/opensearch-project/security/issues

08:42:46 From David Tippett to Everyone:
[Security] OpenSearch should support authentication and authorization natively · Issue #4514 · opensearch-project/OpenSearch · GitHub
08:42:47 From Dave Lago (OpenSearch) to Everyone:
Thanks Peter!!
08:43:06 From Nate Boot to Everyone:
Thanks Peter. Enjoy your sunshine.
08:44:37 From Sander to Everyone:
Replying to “We have a layer in…”

Thanks, created it in the past https://github.com/opensearch-project/security/issues/987 Just remembered because of your hint

08:44:53 From Stephen Crawford to Everyone:
Replying to “We have a layer in f…”

Great, I will take a look at that for you.

08:45:43 From Sander to Everyone:
Replying to “We have a layer in…”

Thanks Stephen!

08:46:33 From Sander to Everyone:
Replying to “We have a layer in…”

If you need more details, no problem, can provide them in the issue or with a call.

08:47:50 From Stephen Crawford to Everyone:
Replying to “We have a layer in f…”

Sounds great. I will take a look at what is on the issue already and follow up on the issue with any immediate questions. I can also join a call at your convenience if there ends up being a lot to discuss for the issue. I will keep you updated as things progress 🙂

08:49:19 From Sander to Everyone:
Replying to “We have a layer in…”

Cool thanks, really appreciate the interactivity that you guys provide.

08:52:45 From Stephen Crawford to Everyone:
Reacted to “Cool thanks, really …” with
08:54:44 From David Tippett to Everyone:
Here is where you can find the meta issue for the Analyzers API.
Get list of all available analyzers. Request for a new API? · Issue #5481 · opensearch-project/OpenSearch · GitHub
08:57:50 From Fernando Rejon Barrera to Everyone:
Is it possible to organize/host local community events? If so, who should I contact regarding this? We would like to organize something in Amsterdam.
08:58:31 From Chris Helma to Everyone:
Oh, physical coins. I was thinking crypto!
08:58:32 From Sander to Everyone:
`great to see that the community is growing, keep it up
08:58:52 From Sander to Everyone:
@fernando, would be interested as well, working at the University of Wageningen
08:59:38 From Fernando Rejon Barrera to Everyone:
thanks

09:00:05 From Nate Boot to Everyone:

09:00:12 From Peter Nied to Everyone:
Great meeting, thanks for the vibrant discussion!

09:02:18 From Nate Boot to Everyone:
I know sometimes people have asked “What is the actual tokenizer string you type in for your index mapping to use it?” - I: can see how a bit of extra reflection mightg make this information self-discoverable.
09:03:59 From Kris Freedain to Everyone:
Reminder - the Security team holds their triage meetings publicly = next one is happening today: Development Backlog & Triage Meeting - Security - 2023-01-17 · OpenSearch
09:04:20 From Stephen Crawford to Everyone:
Reacted to “Reminder - the Secur…” with
09:04:47 From Sam Herman to Everyone:
Thank you all, have to drop.
09:04:52 From Nate Boot to Everyone:
Thanks for coming Sam!
09:05:01 From David Tippett to Everyone:
Thanks Sam!
09:07:03 From Fernando Rejon Barrera to Everyone:
thanks all
09:07:22 From Lukáš Vlček (@Aiven.io) to Everyone:
LOL, thanks!
09:07:25 From eliatra Excelerate Technology Ltd. to Everyone:
Thanks!
09:07:28 From Arnav Das to Everyone:
Thanks

kris · January 17, 2023, 6:25pm

github.com/opensearch-project/opensearch-migrations

[META] Upgrade Testing Framework Issue List

opened 08:00PM - 21 Nov 22 UTC

chelma

Meta issue to track the work associated with the Upgrade Testing Framework. F…ramework Workstream - [x] https://github.com/opensearch-project/opensearch-migrations/issues/1 - [x] https://github.com/opensearch-project/opensearch-migrations/issues/4 - [x] https://github.com/opensearch-project/opensearch-migrations/issues/6 - [x] https://github.com/opensearch-project/opensearch-migrations/issues/8 - [x] https://github.com/opensearch-project/opensearch-migrations/issues/7 - [x] https://github.com/opensearch-project/opensearch-migrations/issues/50 - [ ] https://github.com/opensearch-project/opensearch-migrations/issues/51 - [ ] https://github.com/opensearch-project/opensearch-migrations/issues/65 Expectations Workstream - [ ] https://github.com/opensearch-project/opensearch-migrations/issues/2 - [ ] https://github.com/opensearch-project/opensearch-migrations/issues/3 - [ ] https://github.com/opensearch-project/opensearch-migrations/issues/5 Miscellaneous - [x] https://github.com/opensearch-project/opensearch-migrations/issues/14 - [x] https://github.com/opensearch-project/opensearch-migrations/issues/48 - [x] https://github.com/opensearch-project/opensearch-migrations/issues/49

github.com

opensearch-project/opensearch-migrations/blob/main/upgrades/README.md

# Upgrade Testing Framework

This package is built on the `cluster_migration_core` library and provides runnable steps to test ElasticSearch and OpenSearch upgrades.  It's purpose is to facilitate the creation of a centralized store of the Elasticsearch/OpenSearch community's understanding of the "happy-path" and "sad-path" behavior expected when performing a cluster version upgrade by providing a mechanism to assert that the knowledge store is accurate.  As an example, it should generally be true that the number of documents stored in a cluster should remain the same before and after a software upgrade; this framework makes it easy to assert that is the case for a specific cluster, starting/ending versions, and upgrade type.  It tests that expectation by spinning up clusters locally in Docker and performing an actual upgrade of the software while interrogating the source/target clusters along the way.

Having a way to test these expectations about upgrade behavior means we have confidence in their accuracy.  That confidence opens up future possibilities, such as being able to provide a report on the issues we expect a user to encounter when perform an cluster software upgrade between two version, WITHOUT them having to actually perform that upgrade themselves.  As more expectations are added to our central store, and more tests added to the UTF, the community will gain greater and greater confidence in the cost and risks of proposed software upgrades and prevent many users from inpendendently/blindly encountering the same problem.  Once the issue is added to the central store and a test for it added to the UTF, its status should be promoted from "unknown unknown" to "known".

## How This Tool Is Structured

The UTF is based around the following concepts:
* A sequence of "Steps" that are executed in a set order as part of a "Workflow"
* A "Runner" that takes a Workflow and executes the steps in it sequentially, while managing things like error handling, logging, and state
* A shared "State" that exists in-memory of the process and enables the results of one step to be re-used by a later step
* A "Test Config" JSON file that encapsulates the specific upgrade to be tested (e.g. snapshot/restore from ES 7.10.2 to OS 1.3.6).  This package will contain a collection of pre-canned Test Configs (`./test_configs/`) that should represent the limits of the UTF's knowledge about what is true about how Upgrades work.  In other words - if there isn't a Test Config file in the included collection that covers the specific cluster setup and upgrade type you're interested in, the UTF is not testing that setup/upgrade type.
* A set of "Test Actions" that are performed at various points in a Workflow via an existing library called the [Robot Framework](https://robotframework.org/?tab=2#getting-started).  These currently at `./upgrade_testing_framework/robot_test_defs/`
* A set of "Expectations" that represent things we expect to be true about the upgrade specified in the Test Config file (number of documents should remain the same before/after).  Each Expectation has an Expectation ID that is used to track which Test Actions are associated with a given Expectation, and determine if the Expectation was true or not.  These Expectation IDs are associated with each Test Action as a tag that the UTF search for and enable selective invocation.
* A "Knowledge Base" that represents all the Expectations we're currently tracking and (hopefully) testing, currently located at `../knowledge_base/`.

## Running the Upgrade Testing Framework

To run the UTF, perform the following steps.

This file has been truncated. show original

github.com/opensearch-project/OpenSearch

[Security] OpenSearch should support authentication and authorization natively

opened 07:25PM - 14 Sep 22 UTC

peternied

enhancement discuss security

**Is your feature request related to a problem? Please describe.** Plugins have… some permissions boundaries established by the java security manager, but those only enforce a subset of actions that a plugin can do. Public interfaces within the OpenSearch core systems are accessible by plugins without any access control systems. Knowing which systems have security considerations or not is non-trivial during plugin development, and represents a maintenance problems, if new granularity is added over time and plugins don't update to support these gaps in functionality are security bugs. **Describe the solution you'd like** Support authentication and authorization within OpenSearch access control systems. Use that system to apply universally on all OpenSearch components, plugins, and the new up and coming extensions. **Describe alternatives you've considered** The Security plugin is available, and it offers solutions for several features that align with this overall goal. Individual components might be worthwhile adopting, as the security system was built independently of OpenSearch the enforcement processes and ability to restrict access are optional by nature. **Additional context** I think some of the to move towards this ultimate state looks like, - [ ] OpenSearch should be able to track a Subject through its system - [ ] OpenSearch should disable or enable support for authentication and authorization via config - [ ] OpenSearch internal process have different subjects for different actions - [ ] OpenSearch should have a user registry describable via config - [ ] OpenSearch users should be able to authenticate with 'Basic' HTTP Authentication - [ ] OpenSearch requests should know the Subject of the caller - [ ] OpenSearch should have a way to describe the permissions available to #### Terminology _From definition following NIST glossary. https://csrc.nist.gov/glossary_ - Subject - An individual, process, or device that causes information to flow among objects or change to the system state. - Authentication- Verifying the identity of a user, process, or device, often as a prerequisite to allowing access to resources in a system. - Authorization - The process of verifying that a requested action or service is approved for a specific entity.

github.com/opensearch-project/OpenSearch

Get list of all available analyzers. Request for a new API?

opened 05:06PM - 07 Dec 22 UTC

lukas-vlcek

enhancement Search

**Is your feature request related to a problem? Please describe.** I am missi…ng an option to get list of all available analyzers. There is already analyze API documentation and it mentions "built-in" analyzers. But for normal user there is no way how to learn what are all the options. And even for people who are familiar with the code the list is a subject to updates/changes. One option would be to document the list in the documentation page. But I think I would prefer if the cluster itself can give a list of its analyzers (and also tokenizers, charfilters, normalizers). - There are built-in analyzers at the cluster level (in fact the list is kept at the node level, see below) - There are also ad-hoc/custom analyzers at individual index level (but visibility of these should be subject of RBA rules?) **Describe the solution you'd like** As far as I understand list of all built-in analyzers is materialized once `AnalysisModule.setupAnalyzers(plugins)` is called. I think it would be useful to extend one of the `_nodes/` APIs and give it option to return list of all built-in analyzers (and its componenets: tokenizers, ... etc). (It needs to be API at the "nodes"-level because AnalysisRegistry is kept per node and I think a list of built-in analyzers can differ on each node depending on installed plugins.) As for the list of analyzers defined at the index level I am not sure at this point. Maybe later... **Describe alternatives you've considered** Alternative is to go to the documentation (which does not have this list) or go to the code (which is not an option for many people). **Additional context** n/a

Topic		Replies	Views
OpenSearch Community Meeting - 2022-1206 Community community-meeting	4	706	December 7, 2022
OpenSearch Community Meeting - 2023-0328 Community community-meeting	5	1107	March 29, 2023
OpenSearch Community Meeting - 2023-0425 Community community-meeting	7	601	May 1, 2023
OpenSearch Community Meeting - 2024-0116 Community community-meeting	7	398	January 23, 2024
OpenSearch Community Meeting - 2023-0523 Community community-meeting	10	792	May 30, 2023

OpenSearch Community Meeting - 2023-0117

Related topics