We have an experimental release of Search Pipelines in OpenSearch 2.8 that starts at query submission and ends at result fetching. In between, we see any number of possibilities like query rewriting, analyzers, reranking, etc. that can be integrated as processors in the pipeline. We want to hear more from the community as we start to evolve the code in the repo where this issue lives into the pipeline:
[RFC] Search pipelines · Issue #80 · opensearch-project/search-processor · GitHub.
If you want to go directly to the code the search pipeline plugin is here
OpenSearch/server/src/main/java/org/opensearch/search/pipeline at main · opensearch-project/OpenSearch · GitHub and the core processors are here
OpenSearch/modules/search-pipeline-common at main · opensearch-project/OpenSearch · GitHub
Looking forward to good discussion and more possibilities with this feature.
February 7, 2023, 9:28pm
Looks like we’ve received some input - hopefully the community continues to add useful feedback.
@markcohen - how long will the RFC be open?
I’d like to keep this open until mid-March. We are planning to build and release the initial set of features in 2.7.0 in mid-April. Also, the project is beginning to be tracked here:
[META] Search Pipelines · Issue #6278 · opensearch-project/OpenSearch · GitHub for additional input/insight for the community.
@kris, the first PR was just merged into OpenSearch/main and we closed the RFC issue. If it makes sense to close out this RFC now, please go ahead.
April 24, 2023, 11:14pm
@markcohen - I’ll close this thread
May 30, 2023, 2:51pm
@markcohen - opened it back up at your request
@kris. For folks interested, currently the best place to see most work happening in Search Pipelines is here: Search Applications Vertical · GitHub.
[RFC]: Search Phase Injector Processor · Issue #152 · opensearch-project/neural-search · GitHub is a great example of another phase being included in search pipelines.
There’s more we can do here and we want input/questions/PRs/issues for Search Pipelines. Documentation is coming soon.
The base plugin is a good place to start:
OpenSearch/SearchPipelinePlugin.java at 69f4ac189c83270265a0c178a642e8d0de3edd90 · opensearch-project/OpenSearch · GitHub
We’re targeting to close this forum post on the original search pipelines implementation by 2023-07-15.