gh-125451: Fix deadlock in ProcessPoolExecutor shutdown - #125492
Conversation
Sorry, something went wrong.
There was a deadlock when `ProcessPoolExecutor` shuts down at the same time that a queueing thread handles an error when processing a task. Don't use `_shutdown_lock` to protect the `_ThreadWakeup` pipes -- use an internal lock instead. This fixes the ordering deadlock where the `ExecutorManagerThread` holds the `_shutdown_lock` and joins the queueing thread, while the queueing thread is attempting to acquire the `_shutdown_lock` while closing the `_ThreadWakeup`.
|
🤖 New build scheduled with the buildbot fleet by @colesbury for commit 2efa056 🤖 If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
Sorry, something went wrong.
|
🤖 New build scheduled with the buildbot fleet by @colesbury for commit 2a30e4f 🤖 If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
Sorry, something went wrong.
vstinner
left a comment
There was a problem hiding this comment.
Near line 724, please remove the comment:
# _shutdown_lock must be locked to access _ThreadWakeup (...)
Oh, I made commit a4dfe8e in 2020 to fix another race condition:
commit a4dfe8ede5a37576e17035dccfe109ba7752237e
Author: Victor Stinner <vstinner@python.org>
Date: Wed Apr 29 03:32:06 2020 +0200
bpo-39995: Fix concurrent.futures _ThreadWakeup (GH-19760)
Fix a race condition in concurrent.futures._ThreadWakeup: access to
_ThreadWakeup is now protected with the shutdown lock.
Sorry, something went wrong.
The failure of these 2 workers is known: #125535 |
Sorry, something went wrong.
|
!buildbot AMD64 Fedora Stable Refleaks |
Sorry, something went wrong.
|
🤖 New build scheduled with the buildbot fleet by @colesbury for commit 84db201 🤖 The command will test the builders whose names match following regular expression: AMD64 Fedora Stable Refleaks The builders matched are:
|
Sorry, something went wrong.
|
Thanks @colesbury for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13. |
Sorry, something went wrong.
…GH-125492) There was a deadlock when `ProcessPoolExecutor` shuts down at the same time that a queueing thread handles an error processing a task. Don't use `_shutdown_lock` to protect the `_ThreadWakeup` pipes -- use an internal lock instead. This fixes the ordering deadlock where the `ExecutorManagerThread` holds the `_shutdown_lock` and joins the queueing thread, while the queueing thread is attempting to acquire the `_shutdown_lock` while closing the `_ThreadWakeup`. (cherry picked from commit 760872e) Co-authored-by: Sam Gross <colesbury@gmail.com>
|
Sorry, @colesbury, I could not cleanly backport this to 3.12 due to a conflict. cherry_picker 760872efecb95017db8e38a8eda614bf23d2a22c 3.12 |
Sorry, something went wrong.
|
GH-125598 is a backport of this pull request to the 3.13 branch. |
Sorry, something went wrong.
…ythonGH-125492) There was a deadlock when `ProcessPoolExecutor` shuts down at the same time that a queueing thread handles an error processing a task. Don't use `_shutdown_lock` to protect the `_ThreadWakeup` pipes -- use an internal lock instead. This fixes the ordering deadlock where the `ExecutorManagerThread` holds the `_shutdown_lock` and joins the queueing thread, while the queueing thread is attempting to acquire the `_shutdown_lock` while closing the `_ThreadWakeup`. (cherry picked from commit 760872e) Co-authored-by: Sam Gross <colesbury@gmail.com>
|
GH-125599 is a backport of this pull request to the 3.12 branch. |
Sorry, something went wrong.
…5492) (GH-125598) There was a deadlock when `ProcessPoolExecutor` shuts down at the same time that a queueing thread handles an error processing a task. Don't use `_shutdown_lock` to protect the `_ThreadWakeup` pipes -- use an internal lock instead. This fixes the ordering deadlock where the `ExecutorManagerThread` holds the `_shutdown_lock` and joins the queueing thread, while the queueing thread is attempting to acquire the `_shutdown_lock` while closing the `_ThreadWakeup`. (cherry picked from commit 760872e) Co-authored-by: Sam Gross <colesbury@gmail.com>
…5492) (#125599) There was a deadlock when `ProcessPoolExecutor` shuts down at the same time that a queueing thread handles an error processing a task. Don't use `_shutdown_lock` to protect the `_ThreadWakeup` pipes -- use an internal lock instead. This fixes the ordering deadlock where the `ExecutorManagerThread` holds the `_shutdown_lock` and joins the queueing thread, while the queueing thread is attempting to acquire the `_shutdown_lock` while closing the `_ThreadWakeup`. (cherry picked from commit 760872e)
…#125492) There was a deadlock when `ProcessPoolExecutor` shuts down at the same time that a queueing thread handles an error processing a task. Don't use `_shutdown_lock` to protect the `_ThreadWakeup` pipes -- use an internal lock instead. This fixes the ordering deadlock where the `ExecutorManagerThread` holds the `_shutdown_lock` and joins the queueing thread, while the queueing thread is attempting to acquire the `_shutdown_lock` while closing the `_ThreadWakeup`.
There was a deadlock when ProcessPoolExecutor shuts down at the same time that a queueing thread handles an error when processing a task. Don't use _shutdown_lock to protect the _ThreadWakeup pipes -- use an internal lock instead. This fixes the ordering deadlock where the ExecutorManagerThread holds the _shutdown_lock and joins the queueing thread, while the queueing thread is attempting to acquire the _shutdown_lock while closing the _ThreadWakeup.