gh-154904: Speed up import of shutil by probing compression extensions by maxday · Pull Request #154908 · python/cpython · GitHub

Speed up import shutil by probing compression extensions instead of importing the wrappers

shutil imports bz2, lzma and compression.zstd at module scope only to set the _*_SUPPORTED booleans, then discards them, also dragging incompression._common[._streams] and compression.zstd._zstdfile. Every program importing shutil pays for it even if it never touches an archive. They are pure Python wrappers whose only dependency that can be missing is the extension they wrap, so probe that directly:

-    import bz2
-    del bz2
+    import _bz2
+    del _bz2

Same for lzma→_lzma, compression.zstd→_zstd. zlib is left alone: it is the extension. tarfile/zipfile still import the wrappers when an archive is created or extracted.

Net import cost (median of 40 fresh interpreters, minus -c pass, one core):

	before	after	delta
import shutil	12.75 ms	10.25 ms	−2.50 ms (−19.6%)
import urllib.request	36.64 ms	34.30 ms	−2.34 ms (−6.4%)

urllib.request benefits via tempfile→shutil. Motivation is short-lived processes on constrained hardware (CLI tools, serverless cold starts).

No API change: flags, both registries and get_{archive,unpack}_formats() are unchanged, including on builds missing a compression library. The probe is a real import, so ImportError semantics are exact, unlike find_spec('bz2'), which resolves the always-present wrapper and reports success where _bz2 failed to compile (the case on my machine). PEP 810 lazy import also won't do: the flags are read at module scope (shutil.py:1145, :1354), so must be final at import time.

Tests

Added to TestArchives: test_compression_supported_flags pins the flags to real import semantics, test_compression_wrappers_not_imported_by_shutil prevents the eager imports returning. Existing test_unpack_archive_* coverage unchanged; patchcheck clean.

$ ./python -m test test_shutil -v -m "test_compression*"   ok, ok
$ ./python -m test test_shutil test_zipfile test_tarfile test_importlib \
    test_site test_venv test_zipimport test_bz2 test_lzma test_zstd \
    test_gzip test_urllib2 test_logging -j6
  SUCCESS — 14 OK, run=3,959 skipped=314 (test_bz2 skips on main too)

negative controls, each test fails without the half it covers:
  find_spec variant:  ..._supported_flags FAIL: True != False
  unpatched main:     ..._wrappers_not_imported FAIL:
                      "['lzma', 'compression', 'compression.zstd']" != '[]'

fixes: #154904

Issue: shutil: avoid importing bz2, lzma and compression.zstd just to probe availability #154904

…ensions shutil imported bz2, lzma and compression.zstd at module scope only to set the _*_SUPPORTED flags, then discarded the bindings. Executing those pure Python wrappers also pulled in compression._common[._streams] and compression.zstd._zstdfile, a cost paid by every program that imports shutil even if it never touches an archive. Their only importable dependency that may be missing is the extension module each one wraps, so probe _bz2, _lzma and _zstd directly instead. This keeps the exact ImportError semantics of the previous code -- unlike a find_spec() check, which resolves the always-present wrapper and would report success on builds where the extension fails to load. The wrappers are still imported by tarfile/zipfile when an archive is actually created or extracted. Cuts the net cost of "import shutil" by ~2.5 ms (-19.6%) and of "import urllib.request" by ~2.3 ms (-6.4%).

All commit authors signed the Contributor License Agreement.

Maybe we can use ensure_lazy_imports to make it cleaner:

Suggested change

      
                def test_compression_wrappers_not_imported_by_shutil(self):
          
                    # Importing shutil must not pull in the compression wrappers: they are
          
                    # only needed once an archive is actually created or extracted, and
          
                    # importing them measurably slows down every process that uses shutil.
          
                    wrappers = ('bz2', 'lzma', 'compression', 'compression.zstd')
          
                    script = (
          
                        'import sys, shutil; '
          
                        f'print([m for m in {wrappers!r} if m in sys.modules])'
          
                    )
          
                    # -I so that a sitecustomize/usercustomize importing one of these
          
                    # cannot make the test fail spuriously.
          
                    rc, stdout, stderr = assert_python_ok('-I', '-c', script)
          
                    self.assertEqual(stdout.decode().strip(), '[]', stderr)
          
                def test_compression_wrappers_not_imported_by_shutil(self):
          
                    # gh-154904: Importing shutil must not pull in the compression wrappers: they are
          
                    # only needed once an archive is actually created or extracted, and
          
                    # importing them measurably slows down every process that uses shutil.
          
                    ensure_lazy_imports("shutil", {"bz2", "lzma", "compression", "compression.zstd"})

I'm not test it, and you need to add the import statement at the top of this file:

from test.support.import_helper import ensure_lazy_imports

I'm not quite sure if test_compression_supported_flags is needed here. The wrappers already import the extension modules at the top level, so this test can hardly fail. For example, the zlib subtest just compares the result of import zlib with itself.

Per review feedback on pythonGH-154908: - Rewrite test_compression_wrappers_not_imported_by_shutil on top of test.support.import_helper.ensure_lazy_imports instead of hand-rolling an assert_python_ok subprocess. - Remove test_compression_supported_flags: since each wrapper imports its extension module at the top level, the assertion is near-tautological, and the zlib subtest compared importing zlib with itself.

Thanks for the review @aisk, both applied.

One open question: I left off @support.cpython_only. Nearly all existing ensure_lazy_imports call sites carry it (test_argparse.py is the exception), so happy to add it back if that's the preference :)

I guess it's the behavior of don't add a placeholder object to the module, when lazy importing, is the implement detail of CPython, so added the decorator to let other implementation to skip this.

In this PR, we don't use lazy import, so we should not add it.

Thanks @aisk leaving it as is then! Let me know if you have other comments

cc @emmatyping who may be interested in this.

Another idea, replace the three try/excepts with:

from importlib.util import find_spec

_BZ2_SUPPORTED = find_spec('_bz2') is not None
_LZMA_SUPPORTED = find_spec('_lzma') is not None
_ZSTD_SUPPORTED = find_spec('_zstd') is not None

Gives:

Before: 8 ms	PR: 6 ms	find_spec: 5 ms

Thanks @hugovk! I did try that first, and it's what pushed me to the import probe. As fas as I understandfind_spec answers a different question: it asks whether a module can be located, not whether it can be initialised. For an extension module those come apart, and when they do find_spec reports success where the import fails.

find_spec('_bz2') returns a spec as soon as ExtensionFileLoader finds the .so on the path. Everything that happens after that: dlopen, symbol resolution, PyInit__bz2, module exec, is where a broken extension actually fails, and find_spec never gets there.

Let's say the extension is compiled against a library that isn't present at runtime (ie: missing/mismatched libbz2). The .so is on disk, dlopen fails:

find_spec('_bz2')  -> True
import _bz2        -> ImportError: libbz2.so.1: cannot open shared object file

Let me know if I'm missing something, happy to use find_spec if you think that scenario can not happen but it might feels a bit safer to use import?

Ah yes, it is a change in behaviour. With main/this PR if the .so is corrupt, and we try and use it, we get:

ValueError: unknown archive format 'bztar'

With find_spec:

tarfile.CompressionError: bz2 module is not available

Which is maybe more informative? Anyway, it's probably not worth changing the behaviour for this small extra speedup, especially as shutil.get_archive_formats() then also claims "bztar"is available.

Thanks for confirming!

@hugo @aisk Let me know if you have any remaining feedback! I would love to see that landing in 3.15 if possible :) let me know!

I would love to see that landing in 3.15 if possible

The feature freeze has passed, so it will land only in 3.16.

ok! Thanks for confirming @StanFromIreland :)

…ensions directly (python#154908)

maxday requested a review from giampaolo as a code owner July 30, 2026 02:36

bedevere-app Bot mentioned this pull request Jul 30, 2026

shutil: avoid importing bz2, lzma and compression.zstd just to probe availability #154904

Closed

bedevere-app Bot added the awaiting review label Jul 30, 2026

aisk reviewed Jul 30, 2026

View reviewed changes

aisk approved these changes Aug 3, 2026

View reviewed changes

bedevere-app Bot added awaiting core review and removed awaiting review labels Aug 3, 2026

hugovk approved these changes Aug 10, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Aug 10, 2026

emmatyping approved these changes Aug 11, 2026

View reviewed changes

emmatyping merged commit b98e1a5 into python:main Aug 11, 2026
58 checks passed

bedevere-app Bot removed the awaiting merge label Aug 11, 2026

mbeijen pushed a commit to mbeijen/cpython that referenced this pull request Aug 14, 2026

pythongh-154904 : Speed up import of shutil by probing compression ext…

1ff806f

…ensions directly (python#154908)

DavidCEllis mentioned this pull request Sep 8, 2026

Importing bz2 will succeed under lazy_imports=all even if bz2 is not supported #150167

Open

gh-154904: Speed up import of shutil by probing compression extensions by maxday · Pull Request #154908 · python/cpython · GitHub

Repository navigation

gh-154904: Speed up import of shutil by probing compression extensions - #154908

gh-154904: Speed up import of shutil by probing compression extensions#154908emmatyping merged 2 commits intopython:mainpython/cpython:mainfrom maxday:maxday/shutil-perfmaxday/cpython:maxday/shutil-perfCopy head branch name to clipboard

Conversation

maxday commented Jul 30, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests

Uh oh!

python-cla-bot Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aisk Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aisk commented Jul 30, 2026

Uh oh!

maxday commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aisk commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxday commented Jul 30, 2026

Uh oh!

hugovk commented Jul 30, 2026

Uh oh!

maxday commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hugovk commented Jul 30, 2026

Uh oh!

maxday commented Jul 30, 2026

Uh oh!

maxday commented Aug 3, 2026

Uh oh!

StanFromIreland commented Aug 7, 2026

Uh oh!

maxday commented Aug 7, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gh-154904: Speed up import of shutil by probing compression extensions#154908
emmatyping merged 2 commits into
python:mainfrom
maxday:maxday/shutil-perf

maxday commented Jul 30, 2026 •
edited by bedevere-app Bot
Loading

python-cla-bot Bot commented Jul 30, 2026 •
edited
Loading

aisk Jul 30, 2026 •
edited
Loading

maxday commented Jul 30, 2026 •
edited
Loading

aisk commented Jul 30, 2026 •
edited
Loading

maxday commented Jul 30, 2026 •
edited
Loading