Repository navigation
Refactor subprocess.Popen to let a subclass handle IO asynchronously #72474
Description
| BPO | 28287 |
|---|---|
| Nosy | @gpshead, @pitrou, @giampaolo, @1st1, @Martiusweb |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/gpshead'
closed_at = None
created_at = <Date 2016-09-27.14:26:43.785>
labels = ['type-feature', '3.8']
title = 'Refactor subprocess.Popen to let a subclass handle IO asynchronously'
updated_at = <Date 2018-05-15.21:06:57.317>
user = 'https://github.com/Martiusweb'bugs.python.org fields:
activity = <Date 2018-05-15.21:06:57.317>
actor = 'python-dev'
assignee = 'gregory.p.smith'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2016-09-27.14:26:43.785>
creator = 'martius'
dependencies = []
files = ['44844']
hgrepos = []
issue_num = 28287
keywords = ['patch']
message_count = 4.0
messages = ['277517', '298358', '300839', '301024']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'pitrou', 'giampaolo.rodola', 'yselivanov', 'martius']
pr_nums = ['6878']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28287'
versions = ['Python 3.8']
Activity
Martiusweb commented on Sep 27, 2016
Hi,
Currently, subprocess.Popen performs blocking IO in its constructor (at least on Unix): it reads on a pipe in order to detect outcome of the pre-exec and exec phase in the new child. There is no way yet to modify this behavior as this blocking call is part of a long Popen._execute_child() method.
This is a problem in asyncio (asyncio.subprocess_exec and asyncio.subprocess_shell).
I would like to submit a patch which breaks Popen.__init__() and Popen._execute_child() in several methods so it becomes possible to avoid blocking calls (read on pipe and waitpid) by overriding a few private methods without duplicating too much code. The goal is to use it in asyncio, as described in this pull request (which currently monkey-patches Popen):
python/asyncio#428
This patch only targets the unix implementation.
Thanks for your feedback.
giampaolo commented on Jul 14, 2017
Is the main goal to just make Popen.__init__ non-blocking? If not, bpo-1191964 aims at providing non-blocking reads/writes (including on Windows).
Martiusweb commented on Aug 25, 2017
Yes, the goal is to isolate the blocking IO in __init__ into other methods so Popen can be subclassed in asyncio.
The end goal is to ensure that when asyncio calls Popen(), it doesn't block the process. In the context of asyncio, there's no need to make Popen() IOs non-blocking as they will be performed with the asyncio API (rather than the IO methods provided by the Popen object).
pitrou commented on Aug 30, 2017
By the way, we're using GitHub PRs now for development, though you can still submit patch files if you prefer that.