Summary

Fix a deadlock in ProcessPoolExecutor when using max_tasks_per_child.
The executor stopped scheduling queued tasks after a worker process exited
upon reaching its task limit.

from concurrent.futures import ProcessPoolExecutor

if __name__ == "__main__":
    with ProcessPoolExecutor(1, max_tasks_per_child=2) as exe:
        futs = [exe.submit(print, i) for i in range(10)]

Prints 0 and 1, then hangs forever.

The bug

The idle worker semaphore counts task completions, not idle workers: a
token is released on every non-final task completion, but only submit()
consumes tokens. When a backlog is queued, workers take their next task
directly from the call queue, so a token can outlive the idle period it
was released for -- and outlive the worker itself once the worker exits
at its max_tasks_per_child limit. The worker-replacement path then
acquired such a stale token, concluded an idle worker existed, and never
spawned a replacement. With no workers left, the queued tasks deadlocked.

Present since max_tasks_per_child was introduced in Python 3.11
(GH-27373). Affects 3.11 through 3.15.0beta3.

The fix

Worker replacement after a process exit no longer consults the semaphore:
a new _replace_dead_worker() spawns a replacement whenever the pool is
below max_workers, using len(self._processes) as the source of truth.
The submit() path is unchanged, preserving on-demand spawning and idle
worker reuse (bpo-39207). This is the semantics suggested by @pitrou in
the GH-115642 review; the approach was proposed and production-tested by
@tabrezm.

Replacement is skipped when the executor is shutting down with no
pending work, so a worker reaching its task limit during shutdown does
not spawn a process that shutdown immediately reaps. The previously
disabled gh-90622 assertion in _adjust_process_count() is re-enabled:
with worker replacement split out, that method is reachable only from
submit() with a non-fork start method, so the assertion is valid now.

The semaphore still drifts above the true idle count while a backlog
exists. After this change the drift is harmless: it can only briefly
suppress pool growth below max_workers (each suppressed spawn drains
one token), never lose a worker. Whether the semaphore should be
redesigned or removed is left as a follow-up (see GH-115642 discussion).

The documentation warning added in GH-140897 is replaced with a
versionchanged:: next entry, so each branch's docs will name the
release that fixed it; backports should likewise replace the notes
added by GH-143302 / GH-143303.

Related Issues

• Closes ProcessPoolExecutor hangs when 1<max_tasks_per_child<num_submitted//max_workers #115634
• Alternative to PR gh-115634: Force the process pool to adjust when a process worker exits #115642
• May also fix ProcessPoolExecutor executor manager thread crashes if task fails in some circumstances #119592 and ProcessPoolExecutor.map may hang forever when the submitted iterable is far greater than max_tasks_per_child #111498 (untested)

Acknowledgments

• Thanks to @tabrezm for the elegant fix idea and production testing
• Thanks to @gaogaotiantian for the original PR gh-115634: Force the process pool to adjust when a process worker exits #115642 and extensive analysis
• Thanks to all who reported and helped diagnose this issue

Claude Sonnet 4.5 helped do the original work on this PR last year. Claude Fable 5 helped tighten it up and validate everything.

• Issue: ProcessPoolExecutor hangs when 1<max_tasks_per_child<num_submitted//max_workers #115634