FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

bpo-47029: Fix BrokenPipeError in multiprocessing.Queue at garbage collection and explicit close by geryogam · Pull Request #31913 · python/cpython · GitHub

  1. python
  2. cpython

Repository navigation

bpo-47029: Fix BrokenPipeError in multiprocessing.Queue at garbage collection and explicit close #31913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JelleZijlstra merged 4 commits into from
May 3, 2022
Conversation 13 Commits 4 Checks 0 Files changed
Merged

bpo-47029: Fix BrokenPipeError in multiprocessing.Queue at garbage collection and explicit close #31913

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d227300
Fix a BrokenPipeError when a multiprocessing.Queue is garbage collected
geryogam Mar 15, 2022
9950aaa
Close self._reader from the queue thread instead of the main thread
geryogam Apr 26, 2022
cf7d8ec
Update queues.py
geryogam Apr 26, 2022
d8e8957
Create 2022-04-26-19-01-13.bpo-47029.qkT42X.rst
geryogam Apr 26, 2022
File filter

Filter by extension

Filter by extension .py  (1) .rst  (1) All 2 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
23 changes: 11 additions & 12 deletions Lib/multiprocessing/queues.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -139,13 +139,10 @@ def put_nowait(self, obj):

def close(self):
self._closed = True
try:
self._reader.close()
finally:
close = self._close
if close:
self._close = None
close()
close = self._close
if close:
self._close = None
close()
Comment thread
geryogam marked this conversation as resolved.

def join_thread(self):
debug('Queue.join_thread()')
Expand All @@ -169,8 +166,9 @@ def _start_thread(self):
self._thread = threading.Thread(
target=Queue._feed,
args=(self._buffer, self._notempty, self._send_bytes,
self._wlock, self._writer.close, self._ignore_epipe,
self._on_queue_feeder_error, self._sem),
self._wlock, self._reader.close, self._writer.close,
self._ignore_epipe, self._on_queue_feeder_error,
self._sem),
name='QueueFeederThread'
)
self._thread.daemon = True
Expand Down Expand Up @@ -211,8 +209,8 @@ def _finalize_close(buffer, notempty):
notempty.notify()

@staticmethod
def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe,
onerror, queue_sem):
def _feed(buffer, notempty, send_bytes, writelock, reader_close,
writer_close, ignore_epipe, onerror, queue_sem):
debug('starting thread to feed data to pipe')
nacquire = notempty.acquire
nrelease = notempty.release
Expand All @@ -238,7 +236,8 @@ def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe,
obj = bpopleft()
if obj is sentinel:
debug('feeder thread got sentinel -- exiting')
close()
reader_close()
writer_close()
return

# serialize the data before acquiring the lock
Expand Down
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/Library/2022-04-26-19-01-13.bpo-47029.qkT42X.rst
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Always close the read end of the pipe used by :class:`multiprocessing.Queue`
*after* the last write of buffered data to the write end of the pipe to avoid
:exc:`BrokenPipeError` at garbage collection and at
:meth:`multiprocessing.Queue.close` calls. Patch by Géry Ogam.
Back | FazBrowse Home | New Git URL