OpenSearch Plugins or high-level REST client?

OpenDistro
1

Hello all,

We are working on POC to migrate from SOLR to OpenSearch.

Currently in SOLR, as an example we have a SearchRequestHandler java class which implements SolrRequestHandler.
At a high level, this class performs:

  • Json to pojo → request validation → query building → call search api…
    This is packaged and embedded in the SOLR server. In other words, there is no separate java client application.

When asked why there is no separate java client, I was told that it’s due to performance reason.

As I understand, there are 2 ways to achieve above in OpenSearch.

  1. A separate application using OpenSearch high-level REST client
  2. Plugins (implement org.opensearch.plugins.ActionPlugin)

Trying to understand what are the pros and cons if we go for Plugins.

  1. Can we use Spring framework in Plugins?
  2. Easy to develop and maintain when upgrade version?
  3. Is load balancing to Node going to be an issue?
  4. Is there really performance benefit if we opt for Plugins instead of separate client application.
  5. Initially we will be using OpenSearch on premise and in future on AWS for some clients. Any implications if we choose plugins?
  6. Any other issues/recommendations?

I am new to OpenSearch so apologies if any question is silly. Thanks!

1 Like
2

Let me follow up to understand the problem better: You’re wanting to implement a custom plugin (ActionPlugin) or use a process that uses REST client?

3

We would like to know if implementing custom plug is a good idea for the example mentioned or having separate process which uses REST client is a better one.
Out of 6 questions, please provide your opinion on as many as you can.
Thanks

4

Hey @Summer ,

Let me try to address some of your concerns. As far as I understand, you want to build proxy / adapter / gateway between your services / applications and Opensearch, not extend Opensearch functionality anyhow. If that is the case, I would recommend to start with a separate application using Opensearch high-level REST client:

  • easy time implement
  • easy to (re)deploy (bugfix / patch)
  • it very likely will change often
  • it very likely will scale differently
  • lower risks to destabilise the cluster
  1. Can we use Spring framework in Plugins?

Better not, reasoning:

  • Spring is just general purpose application development framework, the plugins have much smaller scope and well defined lifecycle
  • the plugins run under SecurityManager (minimal permissions)
  1. Easy to develop and maintain when upgrade version?

It highly depends on the complexity of the plugin, in general you would need to re-release the plugin to match Opensearch releases, see please [1] for the context. Also, if you need to change the plugin very often, you would need to redeploy it.

  1. Is load balancing to Node going to be an issue?

It depends how you implement the plugin since you have access to Opensearch internals. In general, the search request lands on coordinating node which distribute the work to data nodes that have relevant indices.

  1. Is there really performance benefit if we opt for Plugins instead of separate client application.

It looks like you are building the simple adapter / gateway for your applications, not extending the Opensearch functionality whatsoever. Sitting on the Opensearch side as a plugin would save you a network hop for the price of increased complexity.

  1. Initially we will be using OpenSearch on premise and in future on AWS for some clients. Any implications if we choose plugins?

Yes, I believe you won’t be able to deploy arbitrary plugins (fe, yours) for managed offerings.

  1. Any other issues/recommendations?

There are many examples to learn from [2] (if you think plugin is still the best choice).
Hope it helps.

[1] [opensearch-plugin] Cannot install old patch version of plugins on newer opensearch builds · Issue #1707 · opensearch-project/OpenSearch · GitHub
[2] https://github.com/opensearch-project/OpenSearch/tree/main/plugins

3 Likes