FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-94777: Fix deadlock in ProcessPoolExecutor by lpaulot · Pull Request #94784 · python/cpython · GitHub

  1. python
  2. cpython

Repository navigation

gh-94777: Fix deadlock in ProcessPoolExecutor - #94784

Merged
gpshead merged 7 commits into
python:mainfrom
lpaulot:fix_process_pool_hang
Jul 10, 2023
ConversationCommits (7)ChecksFiles changed
Merged

gh-94777: Fix deadlock in ProcessPoolExecutor#94784
gpshead merged 7 commits into
python:mainfrom
lpaulot:fix_process_pool_hang

Conversation

lpaulot commented Jul 12, 2022 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Contributor

ghost commented Jul 12, 2022 •
edited by ghost
Loading

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

bedevere-bot commented Jul 12, 2022

Copy link
Copy Markdown

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

lpaulot mentioned this pull request Jul 12, 2022
Closed
lpaulot changed the title gh-94777: Drain call_queue to avoid hanging in ProcessPoolExecutor gh-94777: Fix deadlock in ProcessPoolExecutor Jul 13, 2022
itamaro approved these changes Jul 8, 2023
View reviewed changes

itamaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

thanks for the report and the fix!

I was able to repro the issue (on MacOS, 3.13 main branch as of today) and confirmed this patch fixes it.

carljm requested review from gpshead and pitrou July 10, 2023 16:44

carljm commented Jul 10, 2023

Copy link
Copy Markdown
Member

@gpshead @pitrou You are listed in devguide as experts for multiprocessing module; any concerns about merging this fix?

We ran into this deadlock in an internal codebase and @mpage independently investigated and produced this exact same fix, before we found this issue and PR. So I'm inclined to merge, but not a multiprocessing expert.

pitrou reviewed Jul 10, 2023
View reviewed changes
Comment thread Lib/concurrent/futures/process.py
pitrou reviewed Jul 10, 2023
View reviewed changes
Comment thread Lib/test/test_concurrent_futures.py

pitrou commented Jul 10, 2023

Copy link
Copy Markdown
Member

@carljm Thanks for the ping. No objection from me.

gpshead added type-bug An unexpected behavior, bug, or error needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes topic-multiprocessing labels Jul 10, 2023
gpshead enabled auto-merge (squash) July 10, 2023 21:16

gpshead commented Jul 10, 2023

Copy link
Copy Markdown
Member

Oh nice, LGTM. Thanks! I've tweaked the NEWS entry and hit the auto-merge button.

gpshead merged commit 6782fc0 into python:main Jul 10, 2023

miss-islington commented Jul 10, 2023

Copy link
Copy Markdown
Contributor

Thanks @lpaulot for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

miss-islington commented Jul 10, 2023

Copy link
Copy Markdown
Contributor

Sorry @lpaulot and @gpshead, I had trouble checking out the 3.12 backport branch.
Please retry by removing and re-adding the "needs backport to 3.12" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 6782fc050281205734700a1c3e13b123961ed15b 3.12

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 10, 2023
pythongh-94777: Fix deadlock in ProcessPoolExecutor (pythonGH-94784)
fc537a3 
Fixes a hang in multiprocessing process pool executor when a child process crashes and code could otherwise block on writing to the pipe.  See pythonGH-94777 for more details.
(cherry picked from commit 6782fc0)

Co-authored-by: Louis Paulot <55740424+lpaulot@users.noreply.github.com>

bedevere-bot commented Jul 10, 2023

Copy link
Copy Markdown

GH-106607 is a backport of this pull request to the 3.11 branch.

bedevere-bot removed the needs backport to 3.11 only security fixes label Jul 10, 2023
carljm added needs backport to 3.12 only security fixes and removed needs backport to 3.12 only security fixes labels Jul 10, 2023

miss-islington commented Jul 10, 2023

Copy link
Copy Markdown
Contributor

Thanks @lpaulot for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 10, 2023
pythongh-94777: Fix deadlock in ProcessPoolExecutor (pythonGH-94784)
25edb68 
Fixes a hang in multiprocessing process pool executor when a child process crashes and code could otherwise block on writing to the pipe.  See pythonGH-94777 for more details.
(cherry picked from commit 6782fc0)

Co-authored-by: Louis Paulot <55740424+lpaulot@users.noreply.github.com>

bedevere-bot commented Jul 10, 2023

Copy link
Copy Markdown

GH-106609 is a backport of this pull request to the 3.12 branch.

bedevere-bot removed the needs backport to 3.12 only security fixes label Jul 10, 2023
carljm pushed a commit that referenced this pull request Jul 10, 2023
[3.11] gh-94777: Fix deadlock in ProcessPoolExecutor (GH-94784) (#106607
d63953d 
)

gh-94777: Fix deadlock in ProcessPoolExecutor (GH-94784)

Fixes a hang in multiprocessing process pool executor when a child process crashes and code could otherwise block on writing to the pipe.  See GH-94777 for more details.
(cherry picked from commit 6782fc0)

Co-authored-by: Louis Paulot <55740424+lpaulot@users.noreply.github.com>
carljm pushed a commit that referenced this pull request Jul 10, 2023
[3.12] gh-94777: Fix deadlock in ProcessPoolExecutor (GH-94784) (#106609
90ea3be 
)

gh-94777: Fix deadlock in ProcessPoolExecutor (GH-94784)

Fixes a hang in multiprocessing process pool executor when a child process crashes and code could otherwise block on writing to the pipe.  See GH-94777 for more details.
(cherry picked from commit 6782fc0)

Co-authored-by: Louis Paulot <55740424+lpaulot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

mpage mpage left review comments
pitrou pitrou left review comments
gpshead gpshead approved these changes
itamaro itamaro approved these changes

Assignees

gpshead

Labels

topic-multiprocessing type-bug An unexpected behavior, bug, or error

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ProcessPoolExecutor deadlock when a child process crashes while data is being sent in call queue

9 participants

Back | FazBrowse Home | New Git URL