Sign up to join the meeting at Meetup:
Link to previous meeting’s post: OpenSearch Lucene Study Group Meeting - Monday, April 29th, 2024
Welcome to the OpenSearch Lucene Study Group!
Apache Lucene is the open-sourced search library that powers OpenSearch and many search applications large and small.
We start the meeting with a Lucene learning topic or Q&A session. In the second half of the meeting, we review recent developments in Apache Lucene and discuss their potential impact to OpenSearch, with a particular focus on new and exciting Lucene features that we can (and should) expose through OpenSearch. Since some changes require a deep dive to fully understand, we sometimes ask participants to volunteer for “homework” to dig deeper into changes and report back for the next meeting.
Standing Agenda:
- Welcome / introduction (5 minutes)
- Lucene learning series - someone will either present a Lucene-related talk or we will do Lucene Q&A (20 minutes, recorded)
- Review assigned issues from last time (10 minutes)
- Review new Lucene changes and assign homework (20 minutes)
By joining the OpenSearch Lucene Study Group Meeting, you grant OpenSearch, and our affiliates the right to record, film, photograph, and capture your voice and image during the OpenSearch Community Meeting (the “Recordings”). You grant to us an irrevocable, nonexclusive, perpetual, worldwide, royalty-free right and license to use, reproduce, modify, distribute, and translate, for any purpose, all or any part of the Recordings and Your Materials. For example, we may distribute Recordings or snippets of Recordings via our social media outlets.
Since it’s been a while, we have a longer list of Lucene issues to review this week:
I originally scheduled this for Monday, May 20th, but @lukas-vlcek kindly reminded me that it’s a holiday in much of Europe and in Canada (and maybe elsewhere?)
I’ve rescheduled this week’s meeting to Thursday instead.
1 Like
Summary of this week’s meeting:
We spent some time talking about Add IndexInput#prefetch. by jpountz · Pull Request #13337 · apache/lucene · GitHub and the broader issue in Improve Lucene's I/O concurrency · Issue #13179 · apache/lucene · GitHub. The Lucene issues are looking at eagerly paging parts of files into memory before they’re needed, which really helps when the whole index doesn’t fit in the page cache. The benchmarks in the comments suggest a roughly 50% reduction in latency for those cases. This is exciting for OpenSearch’s remote store feature, which uses a custom Lucene Directory implementation to “page” file chunks from the remote store onto local disk. While prefetching saves microseconds on disk access, it can save milliseconds (or more) on fetching from a remote store.
We also looked at the exciting vector improvements, especially the amazing work done to support SIMD-accelerated dot-product on compressed int4 vectors (Improve int4 compressed comparisons performance by benwtrent · Pull Request #13321 · apache/lucene · GitHub).
Related to SIMD optimizations and their use of Project Panama, we talked a bit about the state of JDK21 for Lucene and OpenSearch, including a feared performance regression identified in JDK 21, lusearch, and Lucene "regression" · Issue #264 · dacapobench/dacapobench · GitHub (that turned out to be a result of bad behavior in the benchmark – recreating IndexReader on every iteration in a loop across many threads, instead of instantiating one IndexReader and letting everyone use it).
We discussed Performance improvements to use RWLock to access LRUQueryCache by boicehuang · Pull Request #13306 · apache/lucene · GitHub and how the Lucene query cache works more broadly.
For next week, we decided to spend time talking some more about OpenSearch’s aggregations and Lucene’s facets. As discussed in [DISCUSS] Identifying Gaps in Lucene’s Faceting · Issue #12553 · apache/lucene · GitHub, there are some great opportunities to cross-pollinate between the projects.
Link to YouTube video of the meeting coming soon…
