gh-119154: Simplify consumers by making `asyncio.Queue` an asynchronous iterable by nineteendo · Pull Request #120491 · python/cpython · GitHub

Purpose

Currently consuming items from a queue is very complex. You need to

use an infinite while loop
call queue.task_done() after processing each item
add all worker tasks to a list
join the queue
cancel all worker tasks.
wait until all worker tasks are cancelled

By making asyncio.Queue an asynchronous iterable, this becomes a lot easier, you only need to call queue.shutdown().

Overview of changes

Calling asyncio.Queue.__aiter__() returns an iterator which iterates over the queue of items
A private _AsyncQueueIterator has been added to handle the iteration
The example in the documentation is greatly simplified by the new addition

Example

Without iteration

import asyncio

async def producer(queue):
    for i in range(5):
        print(f'{i} -> queue')
        await queue.put(i)
        await asyncio.sleep(.1)

async def consumer(queue):
    while True:
        i = await queue.get()
        print(f'queue -> {i}')
        await asyncio.sleep(.3)
        queue.task_done()

async def main():
    queue = asyncio.Queue()
    tasks = []
    for _ in range(1):
        task = asyncio.create_task(consumer(queue))
        tasks.append(task)

    async with asyncio.TaskGroup() as tg:
        tg.create_task(producer(queue))

    print('produced everything')
    await queue.join()
    for task in tasks:
        task.cancel()

    await asyncio.gather(*tasks, return_exceptions=True)
    print('consumed everything')

asyncio.run(main())

With iteration

import asyncio

async def producer(queue):
    for i in range(5):
        print(f'{i} -> queue')
        await queue.put(i)
        await asyncio.sleep(.1)

async def consumer(queue):
    async for i in queue:
        print(f'queue -> {i}')
        await asyncio.sleep(.3)

async def main():
    queue = asyncio.Queue()
    async with asyncio.TaskGroup() as tg1:
        tg1.create_task(consumer(queue))
        async with asyncio.TaskGroup() as tg2:
            tg2.create_task(producer(queue))

        queue.shutdown()
        print('produced everything')

    print('consumed everything')

asyncio.run(main())

Output

0 -> queue
queue -> 0
1 -> queue
2 -> queue
queue -> 1
3 -> queue
4 -> queue
produced everything
queue -> 2
queue -> 3
queue -> 4
consumed everything

Issue: Add asyncio.Queue.__aiter__ #119154

📚 Documentation preview 📚: library/asyncio-queue.html

Co-authored-by: Guido van Rossum <guido@python.org>

cc @gvanrossum, @Zac-HD

I hope sumeone will review this. Right now I am not available as a reviewer, sorry.

Two bits of idiom noted below, but overall this looks great to me - thanks, @nineteendo!

Co-authored-by: Zac Hatfield-Dodds <zac.hatfield.dodds@gmail.com>

Co-authored-by: Kumar Aditya <kumaraditya@python.org>

Hi @nineteendo, Thanks for the PR. I left a comment on the doc and will take a closer look at the implementation after I hear back about the expected behavior.

Hi @nineteendo @Zac-HD @kumaraditya303 @itamaro @ambv, @carljm,

I've noticed a recent trend that the ayncio PR titles and body text are often very terse without much context on what the change is and why.

To help reviewers and to give context to future contributors, let's raise the bar a bit on how we document asyncio PRs.

Write clear titles and descriptions for your pull requests so that reviewers can quickly understand what the pull request does. In the pull request body, include:

the purpose of the pull request
an overview of what changed

Two sentences (one with purpose: why; one with change: what) would be very helpful for reviewers for both individual and stacked task PRs. It would likely also help us turn around PRs in a more timely manner. Thanks!

I've adjusted the PR title and body text. I hope it's a bit clearer now. The example is based on the current one in the documentation.

I want to think some more about what would be the best approach to support this for both synchronous and asynchronous queues.

I finished the alternative pull request: #120925. Decide if you prefer it over the current pull request.
I think an explicit method is better, but that's just my opinion.

@nineteendo Please refrain from opening multiple pull requests at one time for the same issue. Instead, iterating on one PR toward a preferred solution. Here is the preferred approach:

Core devs and reviewers expect you to submit what you believe is the best solution as one PR.
Respond to feedback on the open PR and iterate on it.
If needed, add more tests to prove that the approach is sound.

Thanks!

Sorry for the trouble, the original issue technically didn't apply to syncronous queues (as per the title).

Marking this as closed for the reasons listed in #120925

Thanks for the work on this. It was a plausible idea but queues are more versatile than the iterator protocol allows.

Add asyncio.Queue.__aiter__()

e9c1a3e

Co-authored-by: Guido van Rossum <guido@python.org>

nineteendo changed the title Add asyncio.Queue.__aiter__() gh-119154: Add asyncio.Queue.__aiter__() Jun 14, 2024

bedevere-app Bot mentioned this pull request Jun 14, 2024

Add asyncio.Queue.__aiter__ #119154

Closed

nineteendo and others added 10 commits June 14, 2024 11:47

Reduce diff

3cd59ee

Add test

16fc429

Simplify code

f13ddb3

Add to __all__

afd7ad4

📜🤖 Added by blurb_it.

214548c

Simplify example

9824b63

Fix indentation

6358b8e

Improve order

316b800

Test StopAsyncIteration

92f3295

Document AsyncQueueIterator

a18767d

nineteendo marked this pull request as ready for review June 14, 2024 11:32

nineteendo requested a review from rhettinger as a code owner June 14, 2024 11:32

bedevere-app Bot added the awaiting review label Jun 14, 2024

rhettinger removed their request for review June 14, 2024 17:24

Zac-HD reviewed Jun 14, 2024

View reviewed changes

Comment thread Lib/test/test_asyncio/test_queues.py Outdated

Comment thread Doc/library/asyncio-queue.rst Outdated

nineteendo and others added 6 commits June 15, 2024 06:36

Apply suggestions from code review

320691c

Co-authored-by: Zac Hatfield-Dodds <zac.hatfield.dodds@gmail.com>

Unpack for loop

f43bdb1

Insert newline

4ca4a93

Simplify example

fffacb2

Indent correctly

92d7b16

Document __aiter__()

ce23fce

nineteendo requested a review from Zac-HD June 15, 2024 05:30

nineteendo added 2 commits June 15, 2024 07:34

Fix indent attempt 1

640d2a8

Make AsyncQueueIterator private

6c9aa6d

nineteendo requested a review from 1st1 as a code owner June 18, 2024 18:07

Make queue a private attribute

d71eb91

Co-authored-by: Kumar Aditya <kumaraditya@python.org>

willingc reviewed Jun 19, 2024

View reviewed changes

Comment thread Doc/library/asyncio-queue.rst Outdated

nineteendo added 2 commits June 20, 2024 08:03

Clarify behaviour

23e37f4

Fix grammar

3bb4e65

nineteendo changed the title gh-119154: Add asyncio.Queue.__aiter__() gh-119154: Simplify consumers by making asyncio.Queue an asynchronous iterable Jun 20, 2024

nineteendo commented Jun 20, 2024

View reviewed changes

Comment thread Doc/whatsnew/3.14.rst Outdated

nineteendo marked this pull request as draft June 23, 2024 17:20

bedevere-app Bot removed the awaiting review label Jun 23, 2024

nineteendo marked this pull request as ready for review June 23, 2024 20:34

bedevere-app Bot added the awaiting review label Jun 23, 2024

Zac-HD mentioned this pull request Jun 23, 2024

gh-120924: Simplify consumers by adding asyncio.Queue.iter() and asyncio.Queue.iter_nowait() #120927

Closed

willingc mentioned this pull request Jun 24, 2024

gh-119154: Simplify consumers by adding *.Queue.iter() and *.Queue.iter_nowait() #120925

Closed

nineteendo added 2 commits June 24, 2024 09:31

Update 2024-06-14-10-30-43.gh-issue-119154.1MHWA-.rst

9eff835

Remove parentheses

8580e98

picnixz reviewed Jun 24, 2024

View reviewed changes

Comment thread Doc/whatsnew/3.14.rst Outdated

Update Doc/whatsnew/3.14.rst

670611f

nineteendo changed the title gh-119154: Simplify consumers by making asyncio.Queue an asynchronous iterable gh-120947: Simplify consumers by making asyncio.Queue an asynchronous iterable Jun 24, 2024

bedevere-app Bot mentioned this pull request Jun 24, 2024

Simplify consumers with iteration over *.Queue #120947

Closed

nineteendo changed the title gh-120947: Simplify consumers by making asyncio.Queue an asynchronous iterable gh-119154: Simplify consumers by making asyncio.Queue an asynchronous iterable Jun 24, 2024

nineteendo marked this pull request as draft June 24, 2024 13:44

bedevere-app Bot removed the awaiting review label Jun 24, 2024

rhettinger closed this Sep 25, 2024

nineteendo deleted the add-asyncio.Queue.__aiter__ branch September 25, 2024 05:39

gh-119154: Simplify consumers by making `asyncio.Queue` an asynchronous iterable by nineteendo · Pull Request #120491 · python/cpython · GitHub

Repository navigation

gh-119154: Simplify consumers by making asyncio.Queue an asynchronous iterable - #120491

gh-119154: Simplify consumers by making asyncio.Queue an asynchronous iterable#120491nineteendo wants to merge 33 commits intopython:mainpython/cpython:mainfrom nineteendo:add-asyncio.Queue.__aiter__nineteendo/cpython:add-asyncio.Queue.__aiter__Copy head branch name to clipboard

Conversation

nineteendo commented Jun 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Overview of changes

Example

Without iteration

With iteration

Output

Uh oh!

nineteendo commented Jun 14, 2024

Uh oh!

gvanrossum commented Jun 14, 2024

Uh oh!

Zac-HD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

willingc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

willingc commented Jun 20, 2024

Uh oh!

nineteendo commented Jun 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nineteendo commented Jun 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nineteendo commented Jun 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

willingc commented Jun 24, 2024

Uh oh!

nineteendo commented Jun 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

rhettinger commented Sep 25, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

gh-119154: Simplify consumers by making asyncio.Queue an asynchronous iterable#120491
nineteendo wants to merge 33 commits into
python:mainfrom
nineteendo:add-asyncio.Queue.aiter

nineteendo commented Jun 14, 2024 •
edited
Loading

willingc left a comment •
edited
Loading

nineteendo commented Jun 20, 2024 •
edited
Loading

nineteendo commented Jun 23, 2024 •
edited
Loading

nineteendo commented Jun 23, 2024 •
edited
Loading

nineteendo commented Jun 24, 2024 •
edited
Loading