Added FlushIndexTask to flush documents at index thread level. - #2349
Added FlushIndexTask to flush documents at index thread level.#2349balmukundblr wants to merge 2 commits into
Conversation
Sorry, something went wrong.
mikemccand
left a comment
There was a problem hiding this comment.
This looks great! This IndexWriter.flushNextBuffer() API is a relatively recent addition to Lucene and did not exist when we were actively iterating on benchmarks module!
I left some small comments, and +1 to push this once we resolve those!
Sorry, something went wrong.
| CreateIndex | ||
| [{ "MAddDocs" AddDoc } : 5000] : 4 | ||
| #[{ "MAddDocs" AddDoc } : 5000] : 4 | ||
| [{ {{"MAddDocs" AddDoc } : 5000} FlushIndex } ] : 20 |
There was a problem hiding this comment.
Maybe 1) remove the old (commented out) line entirely? And 2), 20 threads seems a bit much for a "typical" box now? Maybe go to 8 instead?
Hmm, it would be nice if benchmarks made the local machine's CPU concurrency available (Runtime.getRuntime().availableProcessors()) so we could set this dynamically. But that is for another day!
Sorry, something went wrong.
There was a problem hiding this comment.
- Sure Mike, we will remove the old (commented out) line entirely.
- We can change #parallel indexing thread to 8, to work with a typical box.
Sorry, something went wrong.
|
Thanks @balmukundblr this looks great! Could you please open a new PR on the new Lucene GitHub repo? https://github.com/apache/lucene Thanks! |
Sorry, something went wrong.
As you suggested, i've raised a PR(apache/lucene#116) |
Sorry, something went wrong.
Description
Longer completion time for Close Index call.
Once AddDoc task completes, Benchmark algo calls ForceMerge/CloseIndex task, which eventually allows all pending flushes to be completed. Since flushes during CloseIndex call are sequential, it takes longer time to complete and delays the overall Index completion time. While indexing 1 million documents with reuters21578 (plain text Document derived from reuters21578 corpus), we observed CloseIndex call takes around 35% of total time.
Solution
Developed a new FlushIndexTask, which uses flushNextBuffer() Lucene API, to flush document at Index thread level, while not impacting any other Index threads. Adding this task in the algo file, immediately after AddDoc task, would ensure flushing all docs before calling ForceMerge/CloseIndex task.
With this solution in place, CloseIndex task time was reduced significantly and it also improved total time for Indexing.
Tests
Since, we are using existing Lucene API - flushNextBuffer(), hence it already has test cases.
-Passed existing tests
Checklist
Please review the following and check all that apply:
[ ] I have created a Jira issue and added the issue ID to my pull request title.