kris
February 27, 2024, 10:32pm
1
OpenSearch Community Meeting - 2024-0409 - Hope we see you all there!
Agenda:
Ken Hamric with Tracetest
Extending the value of your OpenSearch based OTel Visibility to Testing.
Date: Tue, April 9, 2024Time: 03:00 PM PT (UTC -7)
Event page Meetup page
Meeting Link
Meeting ID: 894 2535 2972
Would you like to present? Tag @kris @nateynate and we’ll work to get you added to the agenda!
Feel free to comment on this agenda before the meeting if you want to add an item or have a question.
After the meeting, we will post the chat log and any meeting notes. We welcome you to keep the conversation going here on the forum.
========By joining the OpenSearch Community Meeting, you grant OpenSearch, and our affiliates the right to record, film, photograph, and capture your voice and image during the OpenSearch Community Meeting (the “Recordings”). You grant to us an irrevocable, nonexclusive, perpetual, worldwide, royalty-free right and license to use, reproduce, modify, distribute, and translate, for any purpose, all or any part of the Recordings and Your Materials. For example, we may distribute Recordings or snippets of Recordings via our social media outlets.
frank
April 9, 2024, 7:22pm
2
@kris @nateynate
Hey folks, I have a late addition to the agenda if possible:
I’d like to discuss a proposal for standardizing cross-module/plugin interactions which sprung out of an issue I am facing deploying a custom version of OpenSearch.
kris
April 25, 2024, 3:41pm
3
Chat log:https://tickets.plainschwarz.com/opensearchconeu/ OpenSearchCon 2024: Europe Session Lineup · OpenSearch Login to Meetup | Meetup OpenSearch 2.13 is ready for download! · OpenSearch https://pokeapi.co/ Just for posterity. https://tracetest.io/case-studies/opentelemetry-uses-tracetest-for-trace-based-integration-testing Issue search results · GitHub
opened 03:42AM - 14 Feb 24 UTC
enhancement
wontfix
meta
Flow Framework allows sequencing of API calls, but these are presently limited t… o a fixed set of implemented steps. We can empower customers to perform automation and configuration using any OpenSearch or plugin API, or even calling external REST APIs.
### Is your feature request related to a problem?
A workflow structure nearly identical to this plugin was proposed in this RFC for the Observability plugin: https://github.com/opensearch-project/observability/issues/1805
> This API is designed to orchestrate multiple steps, each corresponding to a separate API call to a core API within OpenSearch. The primary goal of the Workflow API is to facilitate complex data preparation processes involving enrichment, aggregation, reindexing, and other transformations.
>
> By enabling users to define multi-step workflows, this feature aims to simplify the preparation of raw data for visualization, thereby providing deeper insights into system status and health.
An example workflow step for an API call is included in the linked RFC:
```
"steps": [
{
"name": "add_timestamp_field_to_otel_spans_index",
"method": "PUT",
"endpoint": "/${index_name}",
"body": { ... }
},
....
]
```
### What solution would you like?
Add Workflow Steps which enable calling REST APIs.
The code already exists in our integration test classes, which could be refactored into appropriate steps in the main source tree:
https://github.com/opensearch-project/flow-framework/blob/13f672e1f210473802b292204bdd558963c9b871/src/test/java/org/opensearch/flowframework/TestHelpers.java#L81-L90
This is called with parameters like this:
https://github.com/opensearch-project/flow-framework/blob/13f672e1f210473802b292204bdd558963c9b871/src/test/java/org/opensearch/flowframework/FlowFrameworkRestTestCase.java#L135-L143
I think the best approach here is to write new steps:
- [ ] An `InitHttpClientStep` which creates or configures an appropriate `client()` to make the call. This would be set up very similar to how the `ToolStep` is created, using the workflow parameters to configure a `Client` object. https://github.com/opensearch-project/flow-framework/pull/530
- [ ] A `RestApiStep` which would reference the client in its previous node inputs, and take other params for the actual REST call.
One other consideration is the status API. We keep track of provisioned resources using step name, step id, resource type, and resource id. The linked RFC suggests intermediate status output with "complete" where we'd really want to identify what was done (hard to do generically so maybe we just put "complete") and for in-process steps, running tasks. We could consider that updated provisioning detail in a separate feature request.
We also need supporting code:
- [x] Substitutions as indicated in comments below https://github.com/opensearch-project/flow-framework/pull/525
- [ ] The ability to wait for a particular state
### What alternatives have you considered?
We could write workflow steps for every OpenSearch API (or at least the most common ones) and use the associated Transport Actions.
### Do you have any additional context?
While making REST calls is less efficient than the transport calls, they enable calls outside of OpenSearch as well.
REST client calls may also be needed for future migration to serverless.
While relatively easy to do this with an HTTP Client, supporting HTTPS and preserving appropriate headers may introduce some security complexities.
01:17:29 Nathan Boot: So nice - you don’t have to look up the results of ‘tasks’ to get your model ID’sConfigurable automation for OpenSearch ML use cases · OpenSearch
