Repository navigation
Free-threaded CPython: concurrent new_threadstate() / PyGILState_Ensure() from N≥~8 OS threads deadlocks in _PyEval_StopTheWorld #157666
Description
Bug report
Bug description:
In Python/pystate.c, new_threadstate() → _Py_qsbr_reserve() → (when QSBR freelist is empty) _PyEval_StopTheWorld(interp) (see qsbr.c: _Py_qsbr_reserve, and 3.14.0's pystate.c STW path). When ≥~8 OS threads enter PyGILState_Ensure simultaneously from fresh threads, the first one to hit an empty QSBR freelist requests an STW; the other ≥7 threads are concurrently in (or about to enter) tstate_wait_attach/tstate_try_attach/new_threadstate. They cannot proceed because the STW is not yet complete, and the STW requester cannot complete because the attach path is blocked — a circular stall (futex_do_wait, ~0 CPU).
Distinct from #151518 (and not fixed by the first pull request there).
Short reproducer:
/* N pthreads, each: PyGILState_Ensure -> loop of 5,000x PyEval_SaveThread/RestoreThread -> PyGILState_Release.
Trigger: >=8 threads (deterministic at 8, 12, 16, 20 on every version tested);
6 and below: passes.
cc -O2 -pthread -o repro repro_ft.c \
-I/tmp/<prefix>/include/python3.1Xt \
/tmp/<prefix>/lib/libpython3.1Xt.a -lpthread -lm -lutil -ldl
timeout 45 ./repro 8 # -> "launched worker 8" then HANG (exit 124)
*/
#include <Python.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
static volatile long long counter = 0;
static volatile int g_stop = 0;
static int g_iters = 5000;
static int g_mode = 1; /* 1=churn, 0=once */
static void *worker(void *arg) {
PyGILState_STATE st = PyGILState_Ensure();
long long n = 0;
for (long long i = 0; i < g_iters && !g_stop; i++) {
if (g_mode == 0) { /* once: already ensured once, loop is Save/Restore only */
}
PyThreadState *ts = PyEval_SaveThread();
for (volatile int k = 0; k < 100; k++) {}
PyEval_RestoreThread(ts);
n++;
}
PyGILState_Release(st);
__sync_fetch_and_add(&counter, n);
return NULL;
}
int main(int argc, char **argv) {
int n = argc > 1 ? atoi(argv[1]) : 8;
g_mode = argc > 2 && !strcmp(argv[2], "once") ? 0 : 1;
g_iters = argc > 3 ? (int)atol(argv[3]) : 5000;
Py_Initialize();
fprintf(stderr, "Py_VERSION=%s\n", Py_GetVersion());
pthread_t *t = malloc(sizeof(pthread_t) * n);
for (int i = 0; i < n; i++) { pthread_create(&t[i], NULL, worker, NULL); fprintf(stderr, "launched worker %d\n", i); }
for (int i = 0; i < n; i++) pthread_join(t[i], NULL);
fprintf(stderr, "ALL WORKERS COMPLETED (PASS) counter=%lld\n", counter);
Py_Finalize();
return 0;
}Environment / repro notes
- Tested on Ryzen 7700X (8 physical cores, 16 logical) (my B200 is undergoing repairs, honestly)
- Ubuntu 25.04
- Linux kernel 7.2.2
- GNU C (cc) 14.2.0
- GIL-disabled builds only (Py_GIL_DISABLED=1); GIL-enabled builds do not exercise the Py_GILState_* STW path in this code.
- LD_PRELOAD none; no ASAN/UBSAN at build time.
Results
| Threads | 3.14.0 (installed, FT) | 3.15.0rc2 (unpatched, FT) | 3.15.0rc2 + PR #152826 (FT) |
|---|---|---|---|
| 4 | PASS | PASS | PASS |
| 6 | PASS | PASS | PASS |
| 8 | HANG (exit 124) | HANG (exit 124) | HANG (exit 124) |
| 12 | HANG (exit 124) | HANG (exit 124) | HANG (exit 124) |
| 20 | HANG (exit 124) | — | — |
Checkbox (per CPython "Reporting bugs" convention)
- I can reproduce this on the current stable (3.14.0) and current development/RC (3.15.0rc2) free-threading builds.
- I built from source (3.15.0rc2) and I can provide the exact --disable-gil build recipe.
- I can reproduce this on the Windows or macOS free-threading builds — not tested (Linux x86_64 only so far).
- I can reproduce it on Python < 3.14 — not tested (free-threading is 3.13+/3.14+/3.15+; I expect the same stall on 3.13 free-threading but have not measured it).
CPython versions tested on:
3.15
Operating systems tested on:
Linux
Activity
CypherGrue commented on Sep 17, 2026
My machine further advises options for maintainer discussion:
ngoldbaum commented on Sep 17, 2026
Is this a theoretical problem found by analyzing the implementation or a real-world problem for a downstream project or work using Python? If so, some details about that would help prioritize whether this is worth focusing on.
CypherGrue commented on Sep 17, 2026
I hit the problem without looking for it. The application is cryptographic research. I am trying to parallelize testing and verification using Cython translation layer for speedups. My task thread code is pretty simple:
This is practical for me as this limits me to 5 workers + 1 dispatch thread, even though the machine has 8 cores, however it is not any sort of production system. If I get a hint as to which solution might be more acceptable, I can probably come up a PR.
colesbury commented on Sep 17, 2026
colesbury commented on Sep 18, 2026
There is a bug in the reproducer:
pthread_join is blocking. You need Py_BEGIN_ALLOW_THREADS around it, e.g.:
Py_BEGIN_ALLOW_THREADS for (int i = 0; i < n; i++) { pthread_join(t[i], NULL); } Py_END_ALLOW_THREADScolesbury commented on Sep 18, 2026
I haven't found a deadlock bug in the qsbr attach path. I don't think the summary is right:
via Codex:
The description gets the second dependency backwards: stop-the-world does not need blocked threads to finish attaching. It needs them to remain unattached.
Specifically:
So “attachment waits for STW” is true, but “STW waits for attachment” is not. The description hasn’t established a circular dependency.
The actual cycle in their reproducer is:
STW waits for the attached main thread to stop → main waits in pthread_join() for workers → workers wait for STW.
ngoldbaum commented on Sep 18, 2026
FWIW this is a very common gotcha. I think pretty much everyone learning how to use the C API with threads will eventually hit issues related to forgetting to detach before a possibly-blocking call. This is also an issue Rust doesn't help with either because sound code can deadlock.
I have vague plans to write AI skills that emphasize these gotchas but finding time for docs work is always tricky.
CypherGrue commented on Sep 18, 2026
Gotcha for me was just how perfectly well my buggy code works up to 6 threads! Hopefully the next bot will find this issue as documentation. Thank you for your time.