feat: split Studio search into separate course and library indexes [backport to verawood] - #39122
blarghmatey wants to merge 5 commits into
Conversation
Sorry, something went wrong.
Library authoring waits synchronously on Meilisearch, and indexing cost grows with index size. On large instances the shared studio_content index is almost entirely course blocks, so creating a library component can time out (openedx#38993). Course content is indexed asynchronously and doesn't need to share an index with libraries. Course blocks stay in studio_content so existing course documents don't need reindexing. Libraries V2 blocks, containers and collections move to a new studio_library_content index. Writes are routed by document type or key context, rebuild locks and _new temp indexes are per index, and reconcile creates/configures both. The studio search endpoint now returns course_index_name and library_index_name with one tenant token covering both indexes. index_name is kept for one release, pointing at the course index. Existing installs run `reindex_studio --libraries-only` once after migrate. It rebuilds the library index and deletes `type != "course_block"` documents from the course index without reindexing courses. Full rebuilds run the same cleanup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J43t1WXNsmVV5mbdzxv6iT
Each Studio search surface already queries course content or library content, never both, so the frontend picks one index per surface rather than issuing a multi-index search. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J43t1WXNsmVV5mbdzxv6iT
The library pass logged and skipped MeilisearchError from add_documents, so a failed write still swapped the partial library index in and then deleted the library documents from the course index. rebuild_library_index also reported success, so its Celery autoretry never fired. index_course already lets these errors propagate; the library pass now does the same. Per-document build errors are still logged and skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRNUbu4khegCEFfy4j9a5R
IncrementalIndexCompleted rows only record a context key. A library row left by an interrupted run from before the index split means the library was written to the course index, but an incremental rebuild would trust it and skip the library for the new library index, then delete its old copies from the course index. Tutor's init job runs exactly that incremental reindex_studio after migrate. Clearing library rows once at migration time makes those libraries reindex; library passes are cheap, so no per-index checkpoint column is needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRNUbu4khegCEFfy4j9a5R
Per review: reconcile_indexes() reconciles the course and library indexes, and reconcile_index(index_name) reconciles one. reconcile_index() shipped in Verawood taking no index name, so callers of that signature must switch to reconcile_indexes(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRNUbu4khegCEFfy4j9a5R
|
Thanks for the pull request, @blarghmatey! This repository is currently maintained by @openedx/wg-maintenance-openedx-platform-oncall. Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information? If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:
Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
Sorry, something went wrong.
Description
Backport of #39103 to release/verawood. It splits the Studio Meilisearch index into studio_content (course blocks) and studio_library_content (Libraries V2 content), so library authoring no longer waits on indexing into the large course index (#38993). See #39103 for the full description, upgrade steps and decision record.
The only difference from #39103 is that the new rebuild_library_index task has @set_code_owner_attribute, like every other task in Verawood's tasks.py (master has since removed that decorator in #39047). The review follow-ups from #39103 are cherry-picked unchanged: Meilisearch write failures abort the library rebuild, data migration search.0003 clears pre-split library checkpoints, and reconcile_index() is renamed reconcile_indexes(). That rename changes a function that shipped in Verawood, which is why this backport includes it: the API stays the same between Verawood and master.
This should merge after #39103.
Testing instructions
Same as #39103. pytest openedx/core/djangoapps/content/search passes on this branch: 128 tests.
🤖 Generated with Claude Code
https://claude.ai/code/session_01JRNUbu4khegCEFfy4j9a5R