Repository navigation
Implement PEP 784 - Adding Zstandard to the Python standard library #132983
Description
Feature or enhancement
This is a tracking issue for implementing PEP 784. See the PEP text for more details.
Since the diff is significant (~10k lines) I wanted to split up the PRs a bit.
Implementation Plan:
- Add compression module just re-exporting existing compression modules. Move the _compression module.
- Add _zstd native module with Unix build config
- Add Windows build config for _zstd (blocked on adding libzstd to cpython-source-deps) and SBOM config.
- Add zstd Python module with tests
- add NEWS/What's New section
- Add documentation for zstd
- Decide on whether or not to check parameter types for options
- Verify style conformance of C and Python code
- Python
- C
- Refactor _train_dict and _finalize_dict to share common code
- Fix test_zstd failed on ubuntu with free-threading #133885
- Improve error message when options value can't be converted to int
Linked PRs
- gh-132983: Introduce compression package and move _compression module #133018
- gh-132983: Introduce _zstd bindings module #133027
- gh-132983: Install compression package contents #133063
- gh-132983: make compression._common a full-fledged package #133076
- gh-132983: Convert compression._common to a package #133086
- gh-132983: [DRAFT] Test _zstd code with Python test suite #133185
- gh-132983: [Draft] Run refleak tests against _zstd with tests #133282
- gh-132983: Add compression.zstd and Python tests #133365
- GH-132983: Build _zstd on Windows #133366
- GH-132983: Use a standard autoconf version check for zstd #133479
- gh-132983: Add What's New entry for PEP 784 implementation #133495
- GH-132983: Remove zstd version check in the header file #133502
- gh-132983: Remove pyzstd in identifiers #133535
- gh-132983: Style improvements for compression.zstd #133547
- GH-132983: Support finding libzstd without pkg-config #133550
- GH-132983: Restore libzstd fallback detection #133565
- [3.14] gh-132983: Remove pyzstd in identifiers (GH-133535) #133629
- gh-132983: Clean-ups for _zstd #133670
- gh-103092: Support subinterpreters in _zstd #133674
- GH-132983: Remove subclassing support from zstd types #133694
- [3.14] gh-103092: Support subinterpreters in _zstd (GH-133674) #133695
- gh-132983: Fix small issues with zstd support in zipfile #133723
- gh-132983: Don't allow trailer data in ZstdFile #133736
- [3.14] gh-132983: Clean-ups for _zstd (GH-133670) #133756
- [3.14] GH-132983: Restore libzstd fallback detection (GH-133565) #133757
- [3.14] GH-132983: Remove subclassing support from zstd types (GH-133694) #133762
- gh-132983: Simplify _zstd_exec() #133775
- gh-132983: Make zstd types immutable #133784
- GH-132983: remove empty_bytes from _zstd module state #133785
- [3.14] gh-132983: Simplify _zstd_exec() (GH-133775) #133786
- [3.14] GH-132983: remove empty_bytes from _zstd module state (GH-133785) #133788
- GH-132983: PEP 7 and Argument Clinic changes for zstd #133791
- [3.14] GH-132983: PEP 7 and Argument Clinic changes for zstd (GH-133791) #133792
- gh-132983: Reduce the size of _zstdmodule.h #133793
- [3.14] gh-132983: Don't allow trailer data in ZstdFile (GH-133736) #133799
- [3.14] gh-132983: Reduce the size of _zstdmodule.h (GH-133793) #133854
- gh-132983: Remove EndlessZstdDecompressor remains #133856
- [3.14] gh-132983: Make zstd types immutable (GH-133784) #133857
- [3.14] gh-132983: Remove leftovers from EndlessZstdDecompressor (GH-133856) #133859
- gh-132983: Convert zstd __new__ methods to Argument Clinic #133860
- gh-132983: Add documentation for compression.zstd #133911
- [3.14] gh-132983: Convert zstd __new__ methods to Argument Clinic (GH-133860) #133915
- gh-132983: Split _zstd_set_c_parameters #133921
- gh-132983: Convert dict_content to take Py_buffer #133924
- gh-132983: Fix zstd compiler warning about unused function mt_continue_should_break() #133947
- [3.14] gh-132983: Fix compiler warning about unused function mt_continue_should_break() (GH-133947) #133950
- gh-132983: Call Py_XDECREF rather than PyObject_GC_Del in failed __new__ #133962
- [3.14] gh-132983: Fix small issues with zstd support in zipfile (GH-133723) #133974
- [3.14] gh-132983: Style improvements for compression.zstd (GH-133547) #134001
- gh-132983: Add zstd version info to test.pythoninfo #134230
- [3.14] gh-132983: Call Py_XDECREF rather than PyObject_GC_Del in failed __new__ (GH-133962) #134305
- [3.14] gh-132983: Add documentation for compression.zstd (GH-133911) #134425
- gh-132983: Refactor shared code in train_dict and finalize_dict #134432
- [3.14] gh-132983: Refactor shared code in train_dict and finalize_dict (GH-134432) #134442
- gh-132983: Fix refleak in zstd dictionary functions #134459
- [3.14] gh-132983: Fix refleak in zstd dictionary functions (gh-134459) #134463
- gh-132983: Slightly tweak error messages for _zstd compressor/decompressor options dict #134601
- [3.14] gh-132983: Slightly tweak error messages for _zstd compressor/decompressor options dict (GH-134601) #134602
- gh-132983: Make _zstd C code PEP 7 compliant #134605
- [3.14] gh-132983: Make _zstd C code PEP 7 compliant (GH-134605) #134609
- [3.14] gh-132983: Convert dict_content to take Py_buffer in ZstdDict() (GH-133924) #134723
- [3.14] gh-132983: Split _zstd_set_c_parameters (GH-133921) #134838
- gh-132983: Minor fixes and clean up for the _zstd module #134930
- [3.14] gh-132983: Minor fixes and clean up for the _zstd module (GH-134930) #134998
- gh-132983: Add missing references to Zstandard in shutil docstrings #136617
- [3.14] gh-132983: Add missing references to Zstandard in shutil docstrings (GH-136617) #137052
- gh-132983: _zstd: use Py_UNREACHABLE in _zstd_load_impl #137320
- gh-132983: Fix docstrings in ZstdDict #137321
- [3.14] gh-132983: Fix docstrings in ZstdDict (GH-137321) #137343
- [3.14] gh-132983: Use Py_UNREACHABLE in _zstd_load_impl() (GH-137320) #137360
Metadata
Metadata
Assignees
Labels
Projects
- Status
Activity
picnixz commented on Apr 26, 2025
It's something I should have asked on discourse, but is compression the final name of the public module? I see in the PEP that we want to mimic hashlib.algorithms_guaranteed as well and one way to mitigate the name shadowing is to actually use compressionlib as the name (we have hashlib, annotationlib, and we could have compressionlib as well; I also think compressionlib is a less likely name to be chosen but it's not the best one as well). Feel free to ignore this question as the PEP has been accepted.
emmatyping commented on Apr 26, 2025
Yes, the decision was to use compression. A search of Github code showed that it was used very infrequently. Literally one in a million files https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/66
gh-132983: Introduce `compression` package and move `_compression` mo…
gh-132983: Install compression package contents (#133063)
pythongh-132983: Convert compression._common to a package
gh-132983: Convert compression._common to a package (#133086)
gh-132983: Introduce `_zstd` bindings module (GH-133027)
AA-Turner commented on May 4, 2025
I've had a go at adding support for building on Windows in #133366, let me know thoughts.
A
Merge remote-tracking branch 'origin/main' into call-len
ned-deily commented on May 5, 2025
It looks like there are a couple of problems with the configure.ac changes to discover and check libzstd.
172 remaining items
[3.14] gh-132983: Use ``Py_UNREACHABLE`` in ``_zstd_load_impl()`` (GH…
pythongh-132983: Minor fixes and clean up for the _zstd module (pytho…
pythongh-132983: Add missing references to Zstandard in shutil docstr…
pythongh-132983: Fix docstrings in ``ZstdDict`` (python#137321)
pythongh-132983: Use ``Py_UNREACHABLE`` in ``_zstd_load_impl()`` (pyt…
[3.14] pythongh-132983: Add missing references to Zstandard in shutil…
[3.14] pythongh-132983: Fix docstrings in ``ZstdDict`` (pythonGH-137321…
[3.14] pythongh-132983: Use ``Py_UNREACHABLE`` in ``_zstd_load_impl()…
pythongh-132983: Add documentation for compression.zstd (pythonGH-133911
pythongh-132983: Refactor shared code in train_dict and finalize_dict (…
pythongh-132983: Fix refleak in zstd dictionary functions (pythongh-1…
pythongh-132983: Slightly tweak error messages for _zstd compressor/d…
pythongh-132983: Make _zstd C code PEP 7 compliant (pythonGH-134605)
pythongh-132983: Convert dict_content to take Py_buffer in ``ZstdDict…
pythongh-132983: Split ``_zstd_set_c_parameters`` (python#133921)