Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.19.2
Describe the issue :_explain API in OpenSearch to improve the relevancy of my queries. I’m currently using a hybrid query that combines lexical and neural search, along with radial search using min_score.
However, when I check the explanation for the neural part, it always just says “within top k” in the description. I also tried isolating the neural part of the hybrid query to inspect it separately, but the explanation returns “no explanation.”
Is this a limitation of the _explain API—that it currently doesn’t provide detailed information for the neural component?
Configuration :
Relevant Logs or Screenshots :
pablo
June 30, 2025, 9:24am
2
@OSvince Have you tested 3.0.0 or 3.1.0? There were improvements around the _explain API.
# OpenSearch and OpenSearch Dashboards 3.0.0 Release Notes
## Release Highlights
OpenSearch 3.0 delivers significant upgrades for performance, data management, security, vector database functionality, and more to help you build and deploy powerful, flexible solutions for search, analytics, observability, and other use cases.
### New and Updated Features
* Among the significant performance improvements included in OpenSearch 3.0 is an update to range queries. Applying smarter strategies for numeric and date fields, OpenSearch can now answer range filters with far fewer I/O operations to deliver 25% faster performance in Big5 benchmarks.
* New optimization features for high-cardinality queries introduce execution hints for cardinality aggregation, enabling users to better balance precision and performance. This enhancement achieves a 75% reduction in p90 latency in benchmark testing compared to the previous release.
* Concurrent segment search is now enabled by default for k-NN, delivering up to 2.5x faster query performance. Additionally, improvements to the floor segment size setting help improve tail latencies by up to 20%.
* Date histogram aggregations now benefit from enhanced filter rewrite optimization that supports sub-aggregations, offering significant performance gains for real-world use cases requiring multi-level aggregations.
* Derived source for k-NN vectors is production ready in this release, optimizing vector search performance with up to 30x improvement in cold start query latencies. This feature can also reduce storage requirements by 3x across the Faiss, Lucene, and NMSLIB libraries.
* Semantic sentence highlighting introduces context-aware highlighting to identify and highlight relevant sentences based on meaning, not just keyword matches, working seamlessly with traditional search as well as neural and hybrid search. This feature includes a pre-trained model for basic semantic highlighting use cases.
* Concurrent Segment Search is now enabled by default for k-NN, delivering up to 2.5x faster query performance. Additionally, improvements to the floor segment size setting help improve tail latencies by up to 20%.
* The new explain parameter for Faiss engine queries provides detailed insights into k-NN query scoring processes. This enhancement helps users understand and optimize their query results by providing a comprehensive view into search result scores.
* This release changes the default BM25 scoring function from LegacyBM25Similarity to BM25Similarity. This provides better compatibility with the latest Apache Lucene optimizations and removes unnecessary legacy code, leading to cleaner, more maintainable implementations while preserving search result quality.
* Piped Processing Language (PPL) receives powerful new capabilities with lookup, join, and subsearch commands, improving log correlation and filtering capabilities. These enhancements, backed by Apache Calcite, enable better query planning and execution for interactive data exploration.
* Query insights see major improvements with a new live queries API for real-time monitoring and a verbose parameter for optimized dashboard performance. Dynamic columns in the query insights dashboard support efficient query analysis.
* The observability experience is enhanced with contextual launch for anomaly detection, allow you to launch an anomaly detector from the main dashboard and automatically populating relevant logs in the Discover view. This streamlined workflow can significantly accelerate the task of investigating anomalies.
show original
main ← neetikasinghal:explain
opened 07:40PM - 17 Jan 25 UTC
### Description
Add support for explain for Exact/ANN/Radial/Disk/Filtering k-n… n search. Score calculation explanation is currently added only for ANN search.
Proposal for explain is given here: https://github.com/opensearch-project/k-NN/issues/875#issuecomment-2611349466
### Related Issues
Resolves #875
### Check List
- [x] New functionality includes testing.
- [ ] New functionality has been documented.
- [ ] API changes companion pull request [created](https://github.com/opensearch-project/opensearch-api-specification/blob/main/DEVELOPER_GUIDE.md).
- [x] Commits are signed per the DCO using `--signoff`.
- [x] Public documentation issue/PR [created](https://github.com/opensearch-project/documentation-website/issues/new/choose).
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/k-NN/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).
1 Like