gh-91349: Replace zlib with zlib-ng in Windows build by zooba · Pull Request #131438 · python/cpython · GitHub

Issue: Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349

I'll admit I just guessed this, in order to unblock my testing (couldn't build at all with an invalid SBOM). If it's not right, let me know

This is likely the CPE that would get used, although we can't be for certain until a CVE exists with it... the only one I could find with some searching is cpe:...:zlib-ng:minizip-ng which is for a different component but at least the organization is a match.

Figured we'd want some way to detect this other than looking at the text in ZLIB_VERSION

makes sense. annoying to have optional attributes that one would use hasattr or getattr on instead of blindly accessing, but realistically nobody should care as this is more internal informational so this is fine.

Yeah, the main place it'll be used is in pythoninfo, which already handles absent attributes.

We have more offensive optional attributes in our extension modules 😆

I've reviewed the three header files added to PC and they seem generic. It's a shame they aren't part of the sources, but figured it's easier/safer to just copy them into our source tree so that the cpython-source-deps repo is a straight import.

🤖 New build scheduled with the buildbot fleet by @zooba for commit 08eecb1 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F131438%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

The two failing buildbots appear unrelated, and everything else passed. All I changed in the last commit was docs, so I'm not going to rerun the buildbots unless someone thinks it's necessary or worthwhile.

SBOM changes LGTM!

This is likely the CPE that would get used, although we can't be for certain until a CVE exists with it... the only one I could find with some searching is cpe:...:zlib-ng:minizip-ng which is for a different component but at least the organization is a match.

I'm kicking off a pyperformance benchmarking run with this -- we have a non-zero usage of zlib in that suite.

The results for this on pyperformance are basically within the noise / no change. Not all that surprising, given that there are no benchmarks specifically aimed at zlib. https://github.com/faster-cpython/benchmarking-public/tree/main/results/bm-20250319-3.14.0a6+-548daa7

makes sense. annoying to have optional attributes that one would use hasattr or getattr on instead of blindly accessing, but realistically nobody should care as this is more internal informational so this is fine.

This has broken the tailcall builds on Windows: https://github.com/python/cpython/actions/runs/13967899073/job/39102464260

D:\a\cpython\cpython\externals\\zlib-ng-2.2.4\\functable.c(79,26): error : use of undeclared identifier 'slide_hash_sse2'; did you mean 'slide_hash_c'? [D:\a\cpython\cpython\PCbuild\zlib-ng.vcxproj]
D:\a\cpython\cpython\externals\\zlib-ng-2.2.4\\functable.c(123,26): error : use of undeclared identifier 'slide_hash_avx2'; did you mean 'slide_hash_c'? [D:\a\cpython\cpython\PCbuild\zlib-ng.vcxproj]

Probably needs some fixed preprocessor checks (hopefully only in what's in the PC directory) to better handle the intrinsics being present/absent.

Though I notice that build warns NOTE: Visual Studio not detected. LLVM does not provide a C/C++ standard library and may be unable to locate MSVC headers., which might also be relevant.

I think, this is because here
https://github.com/zlib-ng/zlib-ng/blob/860e4cff7917d93f54f5d7f0bc1d0e8b1a3cb988/fallback_builtins.h#L4-L25
in line 4 __clang__ gets excluded and hence #define HAVE_BUILTIN_CTZ is not set (but in the MSVC case it is).

Then, in
https://github.com/zlib-ng/zlib-ng/blob/860e4cff7917d93f54f5d7f0bc1d0e8b1a3cb988/arch/x86/x86_functions.h#L13-L20
we dont get the prototype for slide_hash_sse2 in line 17, hence the error.

Sorry, I cannot get the permalinks to work here the usual way I am doing it :(

Though I notice that build warns NOTE: Visual Studio not detected. LLVM does not provide a C/C++ standard library and may be unable to locate MSVC headers., which might also be relevant.

FWIW, when building with the bundled clang-cl of VS I do not have these warnings, but get the same error.

Most likely zlib-ng would generate different versions of the header files I checked into the PC directory for Clang. Figuring out what differences belong in there and detecting it dynamically rather than using CMake is the way forward.

This should get a new issue (because clang-cl isn't a supported configuration, so we don't have to revert this change until it's working). Tag me on it and we can continue there.

Does this mean that for all executables X86_AVX512 code will be emitted and used?
See e.g. adler32_avx512.c, where the whole file is guarded with X86_AVX512 and there is code like __m512i in it.

I believe it's detected and called dynamically based on availability, but it's all compiled at compile time.

Yeah, I hope so, too.

Unconditionally letting the compiler generate code up to AVX2 for all *.c files here might result in the binary not running on all CPUs?

The big changes here are: - Switching to zlib-ng on Windows (python/cpython#131438) - Using hmac for hashing functions (python/cpython#130157) --------- Co-authored-by: Geoffrey Thomas <geofft@ldpreload.com>

zooba added 4 commits March 19, 2025 00:56

pythongh-91349 : Replace zlib with zlib-ng in Windows build

de56164

Updated readme.txt

1347c50

Fix includes for PGO build

7fd8f26

Use correct include dirs

f919749

zooba requested a review from gpshead March 19, 2025 01:11

zooba requested review from a team and sethmlarson as code owners March 19, 2025 01:11

bedevere-app Bot added the awaiting core review label Mar 19, 2025

bedevere-app Bot mentioned this pull request Mar 19, 2025

Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349

Closed

Add NEWS

fb6500d

zooba commented Mar 19, 2025

View reviewed changes

zooba added 2 commits March 19, 2025 01:25

Fix debug reference and some warnings

89d44a1

Exclude Intel optimizations better on ARM64

08eecb1

zooba added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 19, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 19, 2025

rhpvorderman reviewed Mar 19, 2025

View reviewed changes

Comment thread PCbuild/pythoncore.vcxproj

Update docs and whatsnew

548daa7

sethmlarson approved these changes Mar 19, 2025

View reviewed changes

Remove unresolved reference from docs

f108044

gpshead approved these changes Mar 19, 2025

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Mar 19, 2025

Include ZLIBNG_VERSION in pythoninfo

085fa10

zooba enabled auto-merge (squash) March 19, 2025 18:38

zooba merged commit 63a638c into python:main Mar 19, 2025

bedevere-app Bot removed the awaiting merge label Mar 19, 2025

zooba deleted the gh-91349 branch March 19, 2025 19:25

chris-eibl reviewed Mar 20, 2025

View reviewed changes

chris-eibl mentioned this pull request Mar 20, 2025

clang-cl on Windows fails to compile after zlib-ng #131521

Closed

chris-eibl reviewed Mar 20, 2025

View reviewed changes

hugovk mentioned this pull request Mar 21, 2025

Consider distributing zlib-ng astral-sh/python-build-standalone#464

Open

chris-eibl mentioned this pull request Mar 22, 2025

GH-131521: fix clangcl build on Windows for zlib-ng #131526

Merged

zanieb mentioned this pull request Apr 9, 2025

Bump to 3.14.0a7 astral-sh/python-build-standalone#580

Merged

andersonhc mentioned this pull request Oct 15, 2025

Add Python 3.14 support py-pdf/fpdf2#1622

Merged

4 tasks

+                    <PrecompiledHeader>NotUsing</PrecompiledHeader>
+                    <AdditionalIncludeDirectories>$(zlibNgDir);$(PySourceDir)PC;$(GeneratedZlibNgDir);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+                    <PreprocessorDefinitions>%(PreprocessorDefinitions);ZLIB_COMPAT;WITH_GZFILEOP;NO_FSEEKO;HAVE_BUILTIN_ASSUME_ALIGNED;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;</PreprocessorDefinitions>
+                    <PreprocessorDefinitions Condition="$(Platform) == 'Win32' or $(Platform) == 'x64'">%(PreprocessorDefinitions);X86_FEATURES;X86_HAVE_XSAVE_INTRIN;X86_SSE2;X86_SSSE3;X86_SSE42;X86_PCLMULQDQ_CRC;X86_AVX2;X86_AVX512;X86_AVX512VNNI;X86_VPCLMULQDQ_CRC</PreprocessorDefinitions>

+                    <PreprocessorDefinitions>%(PreprocessorDefinitions);ZLIB_COMPAT;WITH_GZFILEOP;NO_FSEEKO;HAVE_BUILTIN_ASSUME_ALIGNED;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;</PreprocessorDefinitions>
+                    <PreprocessorDefinitions Condition="$(Platform) == 'Win32' or $(Platform) == 'x64'">%(PreprocessorDefinitions);X86_FEATURES;X86_HAVE_XSAVE_INTRIN;X86_SSE2;X86_SSSE3;X86_SSE42;X86_PCLMULQDQ_CRC;X86_AVX2;X86_AVX512;X86_AVX512VNNI;X86_VPCLMULQDQ_CRC</PreprocessorDefinitions>
+                    <PreprocessorDefinitions Condition="$(Configuration) == 'Debug'">%(PreprocessorDefinitions);ZLIB_DEBUG</PreprocessorDefinitions>
+                    <EnableEnhancedInstructionSet Condition="$(Platform) == 'Win32' or $(Platform) == 'x64'">AdvancedVectorExtensions2</EnableEnhancedInstructionSet>

                       {
                         "referenceCategory": "SECURITY",
-                        "referenceLocator": "cpe:2.3:a:zlib:zlib:1.3.1:*:*:*:*:*:*:*",
+                        "referenceLocator": "cpe:2.3:a:zlib-ng:zlib-ng:2.2.4:*:*:*:*:*:*:*",

                       return -1;
                   }
+              #ifdef ZLIBNG_VERSION
+                  if (PyModule_Add(mod, "ZLIBNG_VERSION",

gh-91349: Replace zlib with zlib-ng in Windows build by zooba · Pull Request #131438 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-91349: Replace zlib with zlib-ng in Windows build - #131438

gh-91349: Replace zlib with zlib-ng in Windows build#131438zooba merged 10 commits intopython:mainpython/cpython:mainfrom zooba:gh-91349zooba/cpython:gh-91349Copy head branch name to clipboard

Conversation

zooba commented Mar 19, 2025 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zooba commented Mar 19, 2025

Uh oh!

bedevere-bot commented Mar 19, 2025

Uh oh!

Uh oh!

zooba commented Mar 19, 2025

Uh oh!

sethmlarson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdboom commented Mar 19, 2025

Uh oh!

mdboom commented Mar 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chris-eibl commented Mar 20, 2025

Uh oh!

zooba commented Mar 20, 2025

Uh oh!

chris-eibl commented Mar 20, 2025

Uh oh!

chris-eibl commented Mar 20, 2025

Uh oh!

zooba commented Mar 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

gh-91349: Replace zlib with zlib-ng in Windows build#131438
zooba merged 10 commits into
python:mainfrom
zooba:gh-91349

zooba commented Mar 19, 2025 •
edited by bedevere-app Bot
Loading