Bug report

How I got here

I'm benchmarking scikit-learn on several machine, one of them is a 2 x 86-core Xeon 6787P (344 hardware threads).

On this machine, scikit-learn random forests fitted with 344 threads (what you get with n_jobs=-1) were often slower on free-threaded 3.14 than with the GIL (median 0.45x over 31 representative benchmark cases).

With gdb, I first found most threads parked on the LOAD_GLOBAL specialization lock (see my comment on gh-152075).

Then I saw the same collapse on other locks hit by the forests: sys.modules lookups of function-level imports, and the lru_cache behind inspect.getattr_static.

I noticed the slowdown always appeared suddenly past a few tens of threads, so I started to investigate that.

Bug description:

On the free-threaded build, a lock that many threads take at the same time doesn't slow down gradually with the number of threads: past some point, it collapses. On the 2 x 86-core Xeon 6787P, a cache hit on an lru_cache'd function takes 0.2 µs with 1 thread, ~6.6 ms with 43 threads and ~58 ms with 344 threads, on main and on 3.14.

I think the cause is the fair handoff in PyMutex (Python/lock.c). When the unlocking thread wakes a waiter that has waited more than TIME_TO_BE_FAIR_NS (1 ms), it hands the lock over directly, and the lock stays idle until that thread is scheduled (~100 µs here). Once the waits exceed 1 ms, every unlock is a handoff, so the waits get even longer and the lock never gets back to the fast path. Rebuilding with TIME_TO_BE_FAIR_NS set to 100 ms, and nothing else changed, removes most of it.

Reproducer (standard library only):

import functools
import sys
import threading
import time

N_CALLS = 1_000


@functools.lru_cache
def cached(x):
    return x


def time_per_call(n_threads):
    """Average time (µs) of one call of an lru_cache'd function (cache hit) when
    n_threads threads call it at the same time."""
    cached(0)
    barrier = threading.Barrier(n_threads + 1)
    durations = []

    def work():
        barrier.wait()
        tic = time.perf_counter()
        for _ in range(N_CALLS):
            cached(0)
        durations.append(time.perf_counter() - tic)

    threads = [threading.Thread(target=work) for _ in range(n_threads)]
    for t in threads:
        t.start()
    barrier.wait()
    for t in threads:
        t.join()
    return 1e6 * sum(durations) / len(durations) / N_CALLS


print(sys.version, "| GIL enabled:", sys._is_gil_enabled())
thread_counts = [int(n) for n in sys.argv[1:]] or [1, 2, 4, 8, 14]
times = [min(time_per_call(n) for _ in range(3)) for n in thread_counts]
print("threads                " + "".join(f"{n:>9d}" for n in thread_counts))
print("time per call (µs)     " + "".join(f"{t:9.1f}" for t in times))

Time per call (µs), built from source with --disable-gil (same flags for all builds), 2 x 86-core Xeon 6787P:

16 threads is around the threshold on this machine: it collapsed in the 3.14.7 run but not in the main one.

On a 14-core laptop (3.14.6t) the threshold is barely reached: 0.1 µs with 1 thread, 15 µs with 14 threads.

This applies to any lock taken by many threads at the same time.

1 ms seems to be reached easily with a few tens of threads on a 2-socket machine. Would it make sense to tune it for many-core machines, for example with a threshold that grows with the number of waiters?

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

Linux