FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

[3.14] GH-135171: Revert async generator expressions behavior by efimov-mikhail · Pull Request #135352 · python/cpython · GitHub

Repository navigation

[3.14] GH-135171: Revert async generator expressions behavior - #135352

Merged
hugovk merged 4 commits into
python:3.14from
efimov-mikhail:issue_135171_async_genexpr
Jun 16, 2025
Merged

hugovk merged 4 commits into
python:3.14from
efimov-mikhail:issue_135171_async_genexpr

Conversation

efimov-mikhail commented Jun 10, 2025 •
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Member

This PR could be a part of #135225.
Changes in behavior of async generator expressions are also reverted.

@serhiy-storchaka @markshannon @Yhg1s

Comment thread Python/codegen.c
else {
/* Sub-iter - calculate on the fly */
VISIT(c, expr, gen->iter);
ADDOP(c, LOC(gen->iter), GET_AITER);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Hm, is this change necessary? I think the test passes without it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I think so. There was a bug in previous commit. I've reverted related logic in codegen.c.

Comment thread Lib/test/test_asyncgen.py Outdated
efimov-mikhail marked this pull request as ready for review June 10, 2025 15:29
efimov-mikhail marked this pull request as draft June 10, 2025 15:32
efimov-mikhail marked this pull request as ready for review June 10, 2025 16:19

Copy link
Copy Markdown
Member Author

I've reverted all generator expressions related changes in codegen.c.
And for now I can see no problems with tests, included those which are added in related PR's.
Could you please review this again?
And backport for 3.13 is also neccessary.

serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM.

Yhg1s commented Jun 10, 2025

Copy link
Copy Markdown
Member

@markshannon Do you want to take a look at this fix? Do you have time to do?

Copy link
Copy Markdown
Member

@efimov-mikhail are you sure that this isn't adding back too many GET_ITER/GET_AITER instructions?
Particularly, for inlined list comprehensions, this might add an extra GET_[A]ITER.

We should have tests for generator expressions and list comprehensions, both sync and async that __iter__ is called only once.
It seems that we only have test for sync gen expressions. The test is called test_genexpr_only_calls_dunder_iter_once, can you extend it to the other cases?

Copy link
Copy Markdown
Member

I've a had a quick look at the generated bytecode and looks like this PR is OK.

@Yhg1s if you need this in ASAP, I think it's good to merge.
We can add those tests to main in another PR and backport them later.

markshannon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I'd like tests to check that __iter__/__aiter__ is only called once in inlined comprehensions, but that can wait for another PR.

Copy link
Copy Markdown
Member

Changes in this PR look similar to the full rollback in #135390, so I think it adds the right number of GET_ITER/GET_AITER at right place. The total effect of #135293 and this PR should restore the old bytecode generation.

Copy link
Copy Markdown
Member Author

Changes in this PR look similar to the full rollback in #135390, so I think it adds the right number of GET_ITER/GET_AITER at right place. The total effect of #135293 and this PR should restore the old bytecode generation.

Yes, changes in these two PRs are very similar. Although, there aren't exactly the same.
In fact, on 3.14 after this PR will be merged we will have issue #125038 fixed w/o additional GET_ITER instruction, but with null pointer checks in FOR_ITER.

But on 3.13 with #135390 being merged there is a different solution (from #125846).

So, I have a question why we don't revert #125846?
Do we want to preserve bytecode changes in 3.13.1 vs 3.13.0?

Copy link
Copy Markdown
Member

Was not it reverted as the result of all these changes?

efimov-mikhail commented Jun 11, 2025 •
edited
Loading

Copy link
Copy Markdown
Member Author

Was not it reverted as the result of all these changes?

It was reverted here.
But it wasn't on 3.13 even after #135390 is being merged.

Copy link
Copy Markdown
Member

Thank you, this is a great observation.

Yhg1s commented Jun 11, 2025

Copy link
Copy Markdown
Member

Do we want to preserve bytecode changes in 3.13.1 vs 3.13.0?

I don't think I was aware of bytecode changes in 31.1. I tested the stdlib's compiled bytecode (including Lib/test) and saw no differences between 13.0, 13.3 and my rollback (and saw many changes between 13.3, 13.4 and 13.4 + the other fix-forward, but I didn't bother trying with this one applied).

In any case it's too late to fix a change in bytecode that slipped into 13.1, if 13.1-13.3 are all the same we should stick with that.

Copy link
Copy Markdown
Member

It is not only change in bytecode generation. It is a behavior change in comparison with 3.12, 3.13.0 and 3.13.4.

efimov-mikhail commented Jun 11, 2025 •
edited
Loading

Copy link
Copy Markdown
Member Author

To be precise, there is a bytecode change:

-> % ./python
Python 3.13.0 (tags/v3.13.0:60403a5409, Jun 11 2025, 19:23:44) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> dis.dis('(x for x in range(10))')
  0           RESUME                   0

  1           LOAD_CONST               0 (<code object <genexpr> at 0x7fa6c309f450, file "<dis>", line 1>)
              MAKE_FUNCTION
              LOAD_NAME                0 (range)
              PUSH_NULL
              LOAD_CONST               1 (10)
              CALL                     1
              GET_ITER
              CALL                     0
              RETURN_VALUE

Disassembly of <code object <genexpr> at 0x7fa6c309f450, file "<dis>", line 1>:
   1           RETURN_GENERATOR
               POP_TOP
       L1:     RESUME                   0
               LOAD_FAST                0 (.0)
       L2:     FOR_ITER                 6 (to L3)
               STORE_FAST_LOAD_FAST    17 (x, x)
               YIELD_VALUE              0
               RESUME                   5
               POP_TOP
               JUMP_BACKWARD            8 (to L2)
       L3:     END_FOR
               POP_TOP
               RETURN_CONST             0 (None)

  --   L4:     CALL_INTRINSIC_1         3 (INTRINSIC_STOPITERATION_ERROR)
               RERAISE                  1
ExceptionTable:
  L1 to L4 -> L4 [0] lasti

and

-> % ./python                  
Python 3.13.1 (tags/v3.13.1:0671451779, Jun 11 2025, 19:25:28) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> dis.dis('(x for x in range(10))')
  0           RESUME                   0

  1           LOAD_CONST               0 (<code object <genexpr> at 0x7f4332397bc0, file "<dis>", line 1>)
              MAKE_FUNCTION
              LOAD_NAME                0 (range)
              PUSH_NULL
              LOAD_CONST               1 (10)
              CALL                     1
              GET_ITER
              CALL                     0
              RETURN_VALUE

Disassembly of <code object <genexpr> at 0x7f4332397bc0, file "<dis>", line 1>:
   1           RETURN_GENERATOR
               POP_TOP
       L1:     RESUME                   0
               LOAD_FAST                0 (.0)
               GET_ITER
       L2:     FOR_ITER                 6 (to L3)
               STORE_FAST_LOAD_FAST    17 (x, x)
               YIELD_VALUE              0
               RESUME                   5
               POP_TOP
               JUMP_BACKWARD            8 (to L2)
       L3:     END_FOR
               POP_TOP
               RETURN_CONST             0 (None)

  --   L4:     CALL_INTRINSIC_1         3 (INTRINSIC_STOPITERATION_ERROR)
               RERAISE                  1
ExceptionTable:
  L1 to L4 -> L4 [0] lasti
>>> 

You can see additional GET_ITER instruction before FOR_ITER.
This change is made in #125846.

IMO, it's better to revert this change and fix regression in #127682.

Copy link
Copy Markdown
Member Author

I've created new PR for 3.13: #135403

Copy link
Copy Markdown
Member Author

I'd like tests to check that __iter__/__aiter__ is only called once in inlined comprehensions, but that can wait for another PR.

There were 3 tests related to this topic, and I've added the fourth.
Is it enough?
Or I miss something?

efimov-mikhail commented Jun 13, 2025 •
edited
Loading

Copy link
Copy Markdown
Member Author

We should remove backport-3.13 label.

And what about changes on main branch?
Maybe some thread on d.p.o. should be started?

serhiy-storchaka removed the needs backport to 3.13 only security fixes label Jun 14, 2025

hugovk commented Jun 16, 2025

Copy link
Copy Markdown
Member

Thanks, this reverts the 3.14 async generator back to 3.12/3.13 behaviour:

>>> (x async for x in None)
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    (x async for x in None)
                      ^^^^
TypeError: 'async for' requires an object with __aiter__ method, got NoneType

And what about changes on main branch? Maybe some thread on d.p.o. should be started?

Yes, please do for main/3.15.

hugovk merged commit 2ef78a8 into python:3.14 Jun 16, 2025

Copy link
Copy Markdown
Member Author

I've started related d.p.o. thread here: https://discuss.python.org/t/generator-expressions-behavior-with-non-iterable-objects/96329.

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants


Back | FazBrowse Home | New Git URL