asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables · Issue #88342 · python/cpython · GitHub

BPO	44176
Nosy	@asvetlov, @1st1, @alexdelorenzo
PRs	bpo-44176: Allow asyncio.as_completed()'s first parameter to be a generator yielding awaitables #26228
Files	as_completed_gen.py

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-05-19.05:04:33.179>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.7', 'library', 'expert-asyncio']
title = 'asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables'
updated_at = <Date 2021-05-19.05:53:09.349>
user = 'https://github.com/alexdelorenzo'

bugs.python.org fields:

activity = <Date 2021-05-19.05:53:09.349>
actor = 'alexdelorenzo'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'asyncio']
creation = <Date 2021-05-19.05:04:33.179>
creator = 'alexdelorenzo'
dependencies = []
files = ['50052']
hgrepos = []
issue_num = 44176
keywords = []
message_count = 1.0
messages = ['393923']
nosy_count = 3.0
nosy_names = ['asvetlov', 'yselivanov', 'alexdelorenzo']
pr_nums = ['26228']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44176'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

GH-88342: clarify that asyncio.as_completed accepts generators yielding tasks #103626

According to the documentation, asyncio.as_completed() takes a positional argument, aws, as an iterable of awaitables[1]:

    asyncio.as_completed(aws, *, loop=None, timeout=None)
      Run awaitable objects in the aws iterable concurrently.

As seen in the attached as_completed_gen.py file, built-in containers like lists, and iterators over them, are accepted by as_completed() as the first parameter without raising an error.

However, as_completed() raises TypeError if it is called with an iterable of awaitables that is also a generator. There are examples of this behavior in as_completed_gen.py, but here is a short example using a generator expression in the main() coroutine function:

    from asyncio import run, as_completed
    
    async def example(): pass
    
    async def main():
      coros = (example() for _ in range(10))
    
      for coro in as_completed(coros):  # raises TypeError
        await coro
    
    run(main())

Running that example will raise a TypeError with this message:

TypeError: expect an iterable of futures, not generator

If we look at the first line in the body of as_completed(), we can see why this error is thrown for generators that yield awaitables:

    def as_completed(fs, *, loop=None, timeout=None):
      if futures.isfuture(fs) or coroutines.iscoroutine(fs):
        raise TypeError(f"expect an iterable of futures, not {type(fs).__name__}")
      ...

Because generators are coroutines, and the first condition in as_completed() is True, and TypeError gets raised:

    from asyncio import coroutines

# generators and generator expressions are coroutines
assert coroutines.iscoroutine(example() for _ in range(10))

Perhaps as_completed() can use inspect.isawaitable() instead, like so:

    from inspect import isawaitable

    def as_completed(fs, *, loop=None, timeout=None):
      if futures.isfuture(fs) or isawaitable(fs):
        ...

I made a pull request with that change here[2].

[1] https://docs.python.org/3/library/asyncio-task.html#asyncio.as_completed

[2] #26228

3.12 allows generators so this is completed.

#103626 clarifies this in doc.

GH-88342: clarify that `asyncio.as_completed` accepts generators yiel…

Merge branch 'main' into superopt

asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables · Issue #88342 · python/cpython · GitHub

Repository navigation

asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables #88342

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables · Issue #88342 · python/cpython · GitHub

Repository navigation

asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables #88342

Description

Linked PRs

Activity

alexdelorenzo commented on May 19, 2021

kumaraditya303 commented on Apr 19, 2023

kumaraditya303 commented on Apr 19, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions