bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread by asvetlov · Pull Request #14344 · python/cpython · GitHub

The resurrection of #13630
Now without dangling threads in tests :)

https://bugs.python.org/issue35621

…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.

I tested it with manual adding 1-second timeout in spawned waiter threads to emulate slow test machine.
After making sure that ./python -Werror -m test -v --fail-env-changed test_asyncio doesn't produce an error I removed the sleep.

The check is replaced by watcher.is_active() call in subprocess transport

LGTM overall. Thanks for pushing this through, Andrew.

Co-Authored-By: Yury Selivanov <yury@magic.io>

Thanks for review!

Thanks @asvetlov for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

GH-14484 is a backport of this pull request to the 3.8 branch.

…ted in non-main thread (pythonGH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

…ted in non-main thread (GH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

…ted in non-main thread (pythonGH-14344)

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

PR #5862 Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344 * https://docs.python.org/3/library/asyncio-policy.html#asyncio.MultiLoopChildWatcher Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com>

PR #5862 Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344 * https://docs.python.org/3/library/asyncio-policy.html#asyncio.MultiLoopChildWatcher Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com> (cherry picked from commit 7080a8b)

With Python 3.7 asynchronous tests randomly fail with "RuntimeError: Event loop is closed" during shutdown. Backtrace of the error includes `SafeChildWatcher` calls. Python 3.8 has replaced `SafeChildWatcher` with a new `ThreadedChildWatcher` by default [1] as a bugfix for "asyncio.create_subprocess_exec() only works with main event loop" bug [2]. Python 3.7 scheduled end of life is 2023-06-27 according to <https://devguide.python.org/versions/>. [1] python/cpython#14344 [2] https://bugs.python.org/issue35621

asvetlov added 7 commits June 4, 2019 02:32

Close pid waiters on watcher.close()

3695efd

Revert "Revert "bpo-35621: Support running subprocesses in asyncio wh…

3559581

…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.

Merge branch 'dangling-threads-in-watcher' into threaded-waiter

b18e6b9

Merge remote-tracking branch 'upstream/master' into threaded-waiter

1bc1d8e

Work on

ef1e21c

Merge branch 'master' into threaded-waiter

c982e4a

Fix dangling thread warnings in tests

6912ec7

asvetlov requested review from 1st1 and gpshead as code owners June 24, 2019 15:30

the-knights-who-say-ni added the CLA signed label Jun 24, 2019

bedevere-bot added the awaiting core review label Jun 24, 2019

asvetlov commented Jun 24, 2019

View reviewed changes

Update documentation

efb19c0

asvetlov added topic-asyncio needs backport to 3.8 labels Jun 24, 2019

Add docstrings

eb07403

1st1 approved these changes Jun 30, 2019

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jun 30, 2019

asvetlov and others added 8 commits June 30, 2019 11:50

Update Doc/library/asyncio-policy.rst

6aa3357

Co-Authored-By: Yury Selivanov <yury@magic.io>

Fix doc text

25d9ac2

Update Doc/library/asyncio-policy.rst

ae3a706

Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Doc/library/asyncio-policy.rst

8baf48c

Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Doc/library/asyncio-policy.rst

6e6a783

Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Lib/asyncio/unix_events.py

1e1281a

Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Lib/asyncio/unix_events.py

936aeb0

Co-Authored-By: Yury Selivanov <yury@magic.io>

Update docs

cc5a752

asvetlov merged commit 0d671c0 into python:master Jun 30, 2019

bedevere-bot removed the awaiting merge label Jun 30, 2019

asvetlov deleted the threaded-waiter branch June 30, 2019 09:55

bedevere-bot removed the needs backport to 3.8 label Jun 30, 2019

miss-islington added a commit that referenced this pull request Jun 30, 2019

bpo-35621 : Support running subprocesses in asyncio when loop is execu…

bf8cb31

…ted in non-main thread (GH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019

bpo-35621: Support running subprocesses in asyncio when loop is execu…

af4d7c9

…ted in non-main thread (pythonGH-14344)

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020

bpo-35621: Support running subprocesses in asyncio when loop is execu…

b340eb2

…ted in non-main thread (pythonGH-14344)

webknjaz mentioned this pull request Jan 25, 2021

pytest-xdist breaks asyncio-based code expecting to be run in the main thread pytest-dev/pytest-xdist#620

Closed

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021

Use ThreadedChildWatcher in tests where available

a7f5a2a

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021

Use ThreadedChildWatcher in tests where available

7b7a324

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz mentioned this pull request Jan 25, 2021

Re-introduce pytest-xdist in supported envs aio-libs/aiohttp#5431

Merged

2 of 5 tasks

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021

Use ThreadedChildWatcher in tests where available

e3df8bc

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Feb 21, 2021

Use ThreadedChildWatcher in tests where available

0df32d5

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Feb 24, 2021

Use ThreadedChildWatcher in tests where available

bea959a

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

link2xt mentioned this pull request May 19, 2023

"Event loop is closed" error in _provide_clean_event_loop() pytest-dev/pytest-asyncio#548

Closed

link2xt mentioned this pull request May 19, 2023

ci: require Python 3.8 for deltachat-rpc-client chatmail/core#4410

Merged

gvanrossum mentioned this pull request Oct 15, 2023

test_asyncio.test_subprocess leaked dangling threads on PPC64LE Fedora Stable 3.x: #110205

Closed

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread by asvetlov · Pull Request #14344 · python/cpython · GitHub

Repository navigation

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread - #14344

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344
asvetlov merged 17 commits into
python:masterfrom
asvetlov:threaded-waiter

asvetlov commented Jun 24, 2019 •
edited by bedevere-bot
Loading

Uh oh!

Uh oh!

asvetlov commented Jun 24, 2019

Uh oh!

asvetlov Jun 24, 2019

Uh oh!

1st1 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asvetlov commented Jun 30, 2019

Uh oh!

miss-islington commented Jun 30, 2019

Uh oh!

bedevere-bot commented Jun 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

                   def add_child_handler(self, pid, callback, *args):
                       assert self._forks, "Must use the context manager"
-                      if self._loop is None:

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread by asvetlov · Pull Request #14344 · python/cpython · GitHub

Repository navigation

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread - #14344

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344asvetlov merged 17 commits intopython:masterpython/cpython:masterfrom asvetlov:threaded-waiterasvetlov/cpython:threaded-waiterCopy head branch name to clipboard

Conversation

asvetlov commented Jun 24, 2019 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asvetlov commented Jun 24, 2019

Uh oh!

asvetlov Jun 24, 2019

Choose a reason for hiding this comment

Uh oh!

1st1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asvetlov commented Jun 30, 2019

Uh oh!

miss-islington commented Jun 30, 2019

Uh oh!

bedevere-bot commented Jun 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344
asvetlov merged 17 commits into
python:masterfrom
asvetlov:threaded-waiter

asvetlov commented Jun 24, 2019 •
edited by bedevere-bot
Loading