`test_tracemalloc_track_race` segfault on buildbot · Issue #143143 · python/cpython · GitHub

Crash report

Seen on the ARM Raspbian 3.x buildbot, but I suspect it's not Raspbian specific:

https://buildbot.python.org/#/builders/424/builds/12446/steps/6/logs/stdio

0:02:36 load avg: 3.51 [177/497/1] test_tracemalloc worker non-zero exit code (Exit code -11 (SIGSEGV))
test_late_untrack (test.test_tracemalloc.TestCAPI.test_late_untrack) ... ok
test_stop_track (test.test_tracemalloc.TestCAPI.test_stop_track) ... ok
test_stop_untrack (test.test_tracemalloc.TestCAPI.test_stop_untrack) ... ok
test_tracemalloc_track_race (test.test_tracemalloc.TestCAPI.test_tracemalloc_track_race) ... Fatal Python error: Segmentation fault

Thread 0xf4afe400 [python] (most recent call first):

Thread 0xf52ff400 [python] (most recent call first):
  <freed thread state>

<freed thread state>

Current thread's C stack trace (most recent call first):
  <system returned no stack trace>

Extension modules: math.integer, _testcapi, _testinternalcapi (total: 3)

cc @kumaraditya303

Might be related to:

gh-143057: avoid lock in tracemalloc APIs when it is not enabled #143065

Hello, I wanted to check whether this issue is still available to work on. If no one is currently handling it, I’d like to contribute. Thanks!

I am not able to reproduce this even after running it in a loop for 100k times.

We are seeing this in the Yocto Autobuilder (CI) system intermittently: https://bugzilla.yoctoproject.org/show_bug.cgi?id=16182

It's happening on an arm64-based Debian host, running the tests inside qemuarm64.

EDIT: also, it occurred using Python 3.13.

Also occurred again (~8 days ago) on the Raspbian after Victor's latest tracemalloc fix:

https://buildbot.python.org/#/builders/424/builds/12867

Ok, I was able to repro the issue. The problem -- or at least one of the problems -- is that when stopping tracemalloc we set the ctx back to NULL. Another thread can see an inconsistency: it may call tracemalloc_alloc but without the right context.

It's a lot easier to repro on the free threading build, but the reported crashes are all with the GIL enabled.

import _testcapi
for i in range(5000):
    _testcapi.tracemalloc_track_race()

You can also increase the chance in a GIL-enabled build, by changing PyMem_RawMalloc to:

void *
PyMem_RawMalloc(size_t size)
{
    void *ctx = _PyMem_Raw.ctx;
    sched_yield();
    void *(*malloc_func)(void *, size_t) = _PyMem_Raw.malloc;
    return malloc_func(ctx, size);
}

If I understood correctly, there is a race condition between:

Thread A calls tracemalloc.stop() which calls PyMem_SetAllocator() to change memory allocators (functions+ctx pointer).
Thread B calls PyTraceMalloc_Track() which calls PyGILState_Ensure() which calls PyMem_RawCalloc().

There is basically no synchronization between thread A and thread B, and so thread B can get an inconsistent state when calling a memory allocator: ctx doesn't match the allocator function.

PyMem_SetAllocator() uses ALLOCATORS_MUTEX, but allocator functions such as PyMem_RawMalloc() don't use this lock.

The test spawns 50 native threads which only gets a temporary thread state when calling PyTraceMalloc_Track() and PyTraceMalloc_Untrack(). The thread C function:

static void
tracemalloc_track_race_thread(void *data)
{
    PyTraceMalloc_Track(123, 10, 1);
    PyTraceMalloc_Untrack(123, 10);

    PyThread_type_lock lock = (PyThread_type_lock)data;
    PyThread_release_lock(lock);
}

python3: skiptest tracemalloc_track_race

`test_tracemalloc_track_race` segfault on buildbot · Issue #143143 · python/cpython · GitHub

Repository navigation

test_tracemalloc_track_race segfault on buildbot #143143

Crash report

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

`test_tracemalloc_track_race` segfault on buildbot · Issue #143143 · python/cpython · GitHub

Repository navigation

test_tracemalloc_track_race segfault on buildbot #143143

Description

Crash report

Activity

colesbury commented on Dec 24, 2025

colesbury commented on Dec 24, 2025

VanshAgarwal24036 commented on Dec 25, 2025

kumaraditya303 commented on Feb 9, 2026

threexc commented on Feb 26, 2026

colesbury commented on Feb 26, 2026

colesbury commented on Feb 26, 2026

vstinner commented on Mar 18, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions