Lazily register most builtin magics to improve startup time - #15376
Closed
Carreau wants to merge 3 commits into
Closed
Lazily register most builtin magics to improve startup time#15376Carreau wants to merge 3 commits into
Carreau wants to merge 3 commits into
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
init_magics used to import and instantiate every builtin Magics class
(execution, namespace, osm, history, display, packaging, pylab, script,
code, logging) at shell startup. Most of these are now registered via
MagicsManager.lazy_magics, the mechanism already used for user/extension
lazy magics, so the underlying module is only imported the first time one
of its magics is actually invoked.
To make this correct rather than just fast:
- MagicAlias, %alias_magic and the automagic prefilter checker now resolve
targets through the lazy-loading path instead of a plain magics lookup,
so e.g. `%hist` (alias for `history`) or typing `pwd` without `%` still
works before the underlying magic has ever been loaded.
- The debugger's %pinfo/%pdef/%pdoc/%pfile/%psource/%pinfo2 helpers use the
lazy-loading lookup too.
- register_lazy() gained an optional magic_kind ("line"/"cell"/"line_cell")
so tab completion can list not-yet-loaded magics (and complete `%%bash`
style sub-commands) without importing them.
- Fixed test_script_config leaking a ScriptMagics.script_magics config
override into later tests now that ScriptMagics is only instantiated on
first use.
A handful of magics classes are still registered eagerly since they are
needed unconditionally during startup or are already cheap to import:
AutoMagics, BasicMagics, ConfigMagics, ExtensionMagics, AsyncMagics.
- Apply darker/ruff formatting to init_magics' lazy magic table (CI formatting check). - enable_matplotlib() reached into magics_manager.registry['ExecutionMagics'] directly, which is only populated once ExecutionMagics has actually been loaded. Force a lazy lookup of 'run' first so the registry entry exists before it's used to set default_runner.
_find_with_lazy_load is typed, unlike the plain find_magic lookup this replaced, so the "type: ignore[no-untyped-call]" is now flagged as unused by mypy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements lazy registration of most builtin IPython magics to reduce startup time. Previously, all builtin magic modules were imported and instantiated eagerly during shell initialization. Now, only essential magics are registered upfront, while the rest are loaded on-demand when first used.
Key Changes
Lazy magic registration in init_magics(): Moved most builtin magic classes (CodeMagics, DisplayMagics, ExecutionMagics, HistoryMagics, LoggingMagics, NamespaceMagics, OSMagics, PackagingMagics, PylabMagics, ScriptMagics) to lazy registration via MagicsManager.register_lazy(). Only AutoMagics, BasicMagics, ConfigMagics, ExtensionMagics, and AsyncMagics are registered eagerly.
Enhanced MagicsManager.register_lazy(): Added optional magic_kind parameter to track whether a lazy magic is "line", "cell", or "line_cell" without importing the module. Added new lazy_magic_kinds() method to query magic kinds for tab completion.
New lazy_magics_kind attribute: Stores the kind information for lazy magics, enabling tab completion to list not-yet-loaded magics with their correct type.
Added load_ipython_extension() functions: Each builtin magic module now includes a load_ipython_extension() function to support lazy loading as an extension.
Updated magic lookup: Modified _find_with_lazy_load() calls in debugger, completer, and basic magics to support lazy loading of magics on first use.
Enhanced tab completion: Updated magic_matcher() in completer to include lazy magic names without forcing their import, and updated _extract_code() to check both loaded and lazy magics.
Automagic support: Updated prefilter to check lazy_magics registry when determining if automagic should be triggered.
Test fixes: Updated test_script_config() to properly restore previous configuration state.
Implementation Details
https://claude.ai/code/session_01TzmUcFAdqoDwkWaKJLzJKm