Run a job that the test suite with MSan to the CI by StanFromIreland · Pull Request #158625 · python/cpython · GitHub

We have to disable extension modules that link against system libraries (which are not built with MSan) since memory those libraries initialise is reported as uninitialised. While this does significantly reduce coverage of some modules, it saves a great amount of CI time. I also had to unpoison a few buffers filled by libc that MSan does not intercept.

A little fix is included, allocate_instrumentation_data() now zeroes the tool_versions of a _PyCoMonitoringData, which update_instrumentation_data() previously read uninitialised. This doesn't have an affect in practice, as the garbage data only decides whether to clear some bits that are already zero, so the outcome is the same either way.

Inspired by #158584.

Note, test_bytes will be failing till #158584 lands.

Interesting change.

test_faulthandler seems to log SEGV from faulthandler_raise_sigsegv and log FPE from faulthandler__sigfpe_impl().

TSan is run with TSAN_OPTIONS="handle_segv=0 (...)".

At least, skip_if_sanitizer_signal() of test_faulthandler can be updated to add memory=True:

    return support.skip_if_sanitizer(f"TSAN/UBSan itercepts {signame}",
                                     thread=True, ub=True, memory=True)

Co-authored-by: Victor Stinner <victor.stinner@gmail.com>

I merged my bytes.fromhex() fix #158584 so you update the branch to retrieve the fix.

It passes now! 🎉 And in good time, slightly faster than UBSan.

LGTM.

I still see faulthandler__sigsegv in "Display logs" of the MSan job. I suggest this change:

diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py
index 82b347c8f8c..2c15cd57455 100644
--- a/Lib/test/test_faulthandler.py
+++ b/Lib/test/test_faulthandler.py
@@ -172,6 +172,7 @@ def check_windows_exception(self, code, line_number, name_regex, **kw):
         self.check_error(code, line_number, fatal_error, **kw)
 
     @skip_segfault_on_android
+    @skip_if_sanitizer_signal("SIGSEGV")
     def test_sigsegv(self):
         self.check_fatal_error("""
             import faulthandler

It may interesting document how to disable extensions in https://docs.python.org/dev/using/configure.html#cmdoption-with-memory-sanitizer documentation. Give an example of Modules/Setup.local file (as shown below).

I tested manually the change:

$ cat Modules/Setup.local
*disabled*
_bz2 _ctypes _curses _curses_panel _dbm _decimal _gdbm _hashlib
_lzma _sqlite3 _ssl _tkinter _uuid _zstd readline zlib

$ ./configure --config-cache OPT="-O2 -g" --with-memory-sanitizer --with-assertions CC=clang LD=clang
$ make clean
$ make
$ export MSAN_OPTIONS="log_path=$PWD/san_log allocator_may_return_null=1"

There is another error logged in "Display logs". I suppose that conftest is a C program created by configure. I failed to reproduce the issue locally so far.

==> /home/runner/work/cpython/cpython/san_log.5297 <==
==5297==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55555562618d in main (/home/runner/work/cpython/cpython/conftest+0xd218d) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208)
    #1 0x7ffff7c2a8c0  (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2a8c0) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #2 0x7ffff7c2a9d7 in __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2a9d7) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #3 0x555555587324 in _start (/home/runner/work/cpython/cpython/conftest+0x33324) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208)

  Uninitialized value was stored to memory at
    #0 0x555555626126 in main (/home/runner/work/cpython/cpython/conftest+0xd2126) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208)
    #1 0x7ffff7c2a8c0  (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2a8c0) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #2 0x7ffff7c2a9d7 in __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2a9d7) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #3 0x555555587324 in _start (/home/runner/work/cpython/cpython/conftest+0x33324) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208)

  Uninitialized value was created by an allocation of 'rc' in the stack frame
    #0 0x555555626071 in main (/home/runner/work/cpython/cpython/conftest+0xd2071) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/home/runner/work/cpython/cpython/conftest+0xd218d) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208) in main
Exiting

Maybe Objects/obmalloc.c can be modified to not use debug hooks on memory allocations when Python is built with --with-memory-sanitizer. When I did tests on Valgrind, I set PYTHONMALLOC=malloc environment variable to disable debug hooks and disable pymalloc, to use the generic malloc()/free().

I will do it as a follow-up to this/#157934. I thought about building with debug here and using PYTHONMALLOC=malloc here, but it does not propagate to subprocesses so I worry we would lose some coverage.

There is another error logged in "Display logs". I suppose that conftest is a C program created by configure. I failed to reproduce the issue locally so far.

Ah! I found the last error. It's comes from a ffi_call() in configure. I suggest this fix:

diff --git a/configure.ac b/configure.ac
index e1d55a7a3ce..86e12ef671f 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4458,7 +4458,7 @@ int main(void)
     {
         return 2;
     }
-    ffi_arg rc;
+    ffi_arg rc = 0;
     ffi_call(&cif, FFI_FN(z_is_expected), &rc, values);
     return !rc;
 }

I suppose that MSAN complains because my system libffi library was not built with MSAN.

cc @skirpichev who wrote that configure test if I recall correctly.

Co-authored-by: Victor Stinner <victor.stinner@gmail.com>

Documentation build overview

📚 cpython-previews | 🛠️ Build #34933474 | 📁 Comparing ef1342e against main (9d22a53)

🔍 Preview build

1 file changed
± using/configure.html

LGTM. Thanks for the update.

LGTM!

Thanks @StanFromIreland for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

Sorry, @StanFromIreland and @pablogsal, I could not cleanly backport this to 3.15 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker b93fb19a6e3118857d9a1dc4ffcc179b632a88c1 3.15

GH-158832 is a backport of this pull request to the 3.15 branch.

GH-158834 is a backport of this pull request to the 3.14 branch.

* Run a job that the test suite with MSan to the CI (#158625) * Run the test suite with MSan in CI * Additional fixes * Add `_Py_MSAN_UNPOISON_STRING` * Apply Victor's suggestions Co-authored-by: Victor Stinner <victor.stinner@gmail.com> * Apply Victor's suggestions Co-authored-by: Victor Stinner <victor.stinner@gmail.com> --------- Co-authored-by: Victor Stinner <victor.stinner@gmail.com> (cherry picked from commit b93fb19) * Mark bytes returned by getrandom as initialized for MSan * Handle invalid non-ASCII struct formats in the fuzz harness --------- Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Victor Stinner <victor.stinner@gmail.com>

Run a job that the test suite with MSan to the CI (#158625) * Run the test suite with MSan in CI * Additional fixes * Add `_Py_MSAN_UNPOISON_STRING` * Apply Victor's suggestions * Apply Victor's suggestions --------- (cherry picked from commit b93fb19) Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Victor Stinner <victor.stinner@gmail.com>

Please see #151828 (comment) : test_os.test_posix failed in the MSan job. Also, the MSan job is required. Would it be possible to make it optional until it's proven to be stable?

Also, the MSan job is required. Would it be possible to make it optional until it's proven to be stable?

I created #158865 for that, since I'm unable to reproduce the test_os.test_posix failure on Fedora 44.

I created issue #158868 to track the test_os failure on MSan.

This reverts commit 1bc78de.

StanFromIreland added 2 commits October 2, 2026 20:52

Run the test suite with MSan in CI

2e1fb0e

Additional fixes

c01ed73

StanFromIreland added skip issue skip news infra CI, GitHub Actions, buildbots, Dependabot, etc. labels Oct 2, 2026

StanFromIreland mentioned this pull request Oct 2, 2026

gh-158583: Fix uninitialized memory read in bytes.fromhex() #158584

Merged

StanFromIreland requested a review from kumaraditya303 October 2, 2026 20:52

StanFromIreland marked this pull request as ready for review October 2, 2026 21:11

StanFromIreland requested review from AA-Turner, JacobCoffee, ezio-melotti, gpshead, hugovk, itamaro, pablogsal and webknjaz as code owners October 2, 2026 21:11

bedevere-app Bot added the awaiting core review label Oct 2, 2026

StanFromIreland requested a review from vstinner October 3, 2026 09:26

kumaraditya303 reviewed Oct 3, 2026

View reviewed changes

Comment thread Modules/socketmodule.c Outdated

kumaraditya303 reviewed Oct 3, 2026

View reviewed changes

Comment thread Modules/socketmodule.c Outdated

Add _Py_MSAN_UNPOISON_STRING

5c0f7ae

StanFromIreland requested a review from kumaraditya303 October 3, 2026 14:02

vstinner reviewed Oct 3, 2026

View reviewed changes

Comment thread Include/pyport.h Outdated

Comment thread Modules/socketmodule.c Outdated

Comment thread Python/instrumentation.c

Apply Victor's suggestions

ab2ad7e

Co-authored-by: Victor Stinner <victor.stinner@gmail.com>

StanFromIreland requested a review from vstinner October 3, 2026 16:11

Merge branch 'main' into msan-ci

e51b518

kumaraditya303 approved these changes Oct 4, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Oct 4, 2026

vstinner approved these changes Oct 4, 2026

View reviewed changes

Apply Victor's suggestions

ef1342e

Co-authored-by: Victor Stinner <victor.stinner@gmail.com>

StanFromIreland requested review from corona10, emmatyping and erlend-aasland as code owners October 4, 2026 20:28

vstinner approved these changes Oct 4, 2026

View reviewed changes

pablogsal approved these changes Oct 5, 2026

View reviewed changes

pablogsal merged commit b93fb19 into python:main Oct 5, 2026
82 of 83 checks passed

bedevere-app Bot removed the awaiting merge label Oct 5, 2026

StanFromIreland deleted the msan-ci branch October 5, 2026 06:36

pablogsal added the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Oct 5, 2026

miss-islington-app Bot assigned pablogsal Oct 5, 2026

pablogsal mentioned this pull request Oct 5, 2026

[3.15] Add MSan to CI (GH-158625) #158832

Merged

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Oct 5, 2026

pablogsal mentioned this pull request Oct 5, 2026

[3.14] Add MSan to CI (GH-158625) #158834

Merged

pablogsal added a commit to pablogsal/cpython that referenced this pull request Oct 5, 2026

Revert "[3.15] Add MSan to CI (pythonGH-158625 ) (python#158832 )"

0155229

This reverts commit 1bc78de.

                   # compile-time blowup on some interpreter files.
                   # (https://github.com/llvm/llvm-project/issues/179695)
+                  # MSan uses --with-assertions instead of --with-pydebug because its
+                  # hooks on the Python memory allocators hide uninitialized reads.

Run a job that the test suite with MSan to the CI by StanFromIreland · Pull Request #158625 · python/cpython · GitHub

Repository navigation

Run a job that the test suite with MSan to the CI - #158625

Run a job that the test suite with MSan to the CI#158625pablogsal merged 6 commits intopython:mainpython/cpython:mainfrom StanFromIreland:msan-ciStanFromIreland/cpython:msan-ciCopy head branch name to clipboard

Conversation

StanFromIreland commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StanFromIreland commented Oct 2, 2026

Uh oh!

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vstinner commented Oct 3, 2026

Uh oh!

vstinner commented Oct 4, 2026

Uh oh!

StanFromIreland commented Oct 4, 2026

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

StanFromIreland Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

vstinner commented Oct 4, 2026

Uh oh!

read-the-docs-community Bot commented Oct 4, 2026

Documentation build overview

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app Bot commented Oct 5, 2026

Uh oh!

miss-islington-app Bot commented Oct 5, 2026

Uh oh!

bedevere-app Bot commented Oct 5, 2026

Uh oh!

bedevere-app Bot commented Oct 5, 2026

Uh oh!

vstinner commented Oct 5, 2026

Uh oh!

vstinner commented Oct 5, 2026

Uh oh!

vstinner commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Run a job that the test suite with MSan to the CI#158625
pablogsal merged 6 commits into
python:mainfrom
StanFromIreland:msan-ci

StanFromIreland commented Oct 2, 2026 •
edited
Loading