| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
…n a max_tasks_per_child worker exits Worker replacement went through the executor object: the manager thread read executor attributes that shutdown(wait=False) clears concurrently, and could not replace workers at all once the executor was garbage collected. A worker exiting at its max_tasks_per_child limit in those states left the remaining submitted work permanently unexecuted and hung interpreter exit; the racing case could crash the manager thread. Replace workers from the executor manager thread using its own state plus configuration read through the live executor weakref, which shutdown() never clears: - After shutdown(wait=False) with the executor still referenced, a replacement is spawned and the remaining work is executed as documented. - Once the executor has been garbage collected (pythongh-152967), or a replacement worker cannot be started and no workers remain, the remaining futures now fail with BrokenProcessPool instead of never resolving. - A new _force_shutting_down flag stops both spawn paths from starting workers that would escape terminate_workers()/kill_workers(). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
| self._join_executor_internals(broken=True) | ||
|
|
||
| def terminate_broken(self, cause): | ||
| def terminate_broken(self, cause, bpe_message=None): |
There was a problem hiding this comment.
While this might look like a public API change in the diff... it's on the _ExecutorManagerThread internal use only class. Fine to backport.
Sorry, something went wrong.
|
🤖 New build scheduled with the buildbot fleet by @gpshead for commit fd234c9 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F152978%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
Sorry, something went wrong.
|
Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15. |
Sorry, something went wrong.
|
Sorry, @gpshead, I could not cleanly backport this to 3.14 due to a conflict. cherry_picker 0c6422ff6a13ae309493fb7a358cb35d7ea959c8 3.14 |
Sorry, something went wrong.
|
GH-153363 is a backport of this pull request to the 3.15 branch. |
Sorry, something went wrong.
…ted work when a max_tasks_per_child worker exits (GH-152978) (#153363) gh-119592: gh-152967: Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits (GH-152978) gh-119592: Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits Worker replacement went through the executor object: the manager thread read executor attributes that shutdown(wait=False) clears concurrently, and could not replace workers at all once the executor was garbage collected. A worker exiting at its max_tasks_per_child limit in those states left the remaining submitted work permanently unexecuted and hung interpreter exit; the racing case could crash the manager thread. Replace workers from the executor manager thread using its own state plus configuration read through the live executor weakref, which shutdown() never clears: - After shutdown(wait=False) with the executor still referenced, a replacement is spawned and the remaining work is executed as documented. - Once the executor has been garbage collected (gh-152967), or a replacement worker cannot be started and no workers remain, the remaining futures now fail with BrokenProcessPool instead of never resolving. - A new _force_shutting_down flag stops both spawn paths from starting workers that would escape terminate_workers()/kill_workers(). (cherry picked from commit 0c6422f) Reviewed-multiple-times-by: Gregory P. Smith Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
Please don't forget about backports. |
Sorry, something went wrong.
|
GH-154873 is a backport of this pull request to the 3.14 branch. |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits. Worker replacement went through the executor object: the manager thread read executor attributes that shutdown(wait=False) clears concurrently, and could not replace workers at all once the executor was garbage collected. A worker exiting at its max_tasks_per_child limit in those states left the remaining submitted work permanently unexecuted and hung interpreter exit; the racing case could crash the manager thread.
Replace workers from the executor manager thread using its own state plus configuration read through the live executor weakref, which shutdown() never clears:
Drafted and investigated entirely by Claude Fable 5 based on the issues. edit: Looks to be in good shape. Undrafting. (I originally put this up as a draft to better iterate on review to see what shape this should take and how feasible backporting this further as a bugfix could be).