Extending OpenSearch Machine Learning with custom algorithms

Plugins Machine Learning
1

Hi all,

Is there a way to extend ml-common with our own custom algorithm? Let us say I want to implement entity extraction or other business specific logic that I want to run on ml nodes.

Thanks

2

We plan to support more model types, feel free to vote for your model type on [RFC] Support more local model types · Issue #1164 · opensearch-project/ml-commons · GitHub

Prefer to learn more details about your use case. For your own custom algorithm, is that algorithm some PyTorch model or just some general code like Python script?

3

Here I am not talking about existing pretrained machine learning model. I am referring to business specific logic that I want to develop and incorporate inside the existing OpenSearch echosystem but have it run specifically inside the ml nodes. If I develop a plugin then most likely it will run on the manager node, ingest or data nodes. But is there a way to enable my custom logic to run inside ml nodes?

4

Got it, actually I’m preparing an RFC now for making the current ML framework more extensible. It takes some time. Will share the RFC link here when it’s ready. It will be possible to support your use case in that RFC.

BTW, can you share more details like what will the input/output look like? Will it be a sync or async process ?

5

I am more towards making it kind of open and not limited to input/output boundaries. In other words, enable developers to create REST end points so that they can accept the I/O format of their choice. At the same time they should be able to internally query and consume data indexed in OpenSearch.

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.