gh-131738: optimize builtin any/all/tuple calls with a generator expression arg by iritkatriel · Pull Request #131737 · python/cpython · GitHub

This PR implements the optimization described in faster-cpython/ideas#545.

Issue: optimize any/all/tuple calls with a generator expression arg #131738

I don't think LOAD_COMMON_CONST is safe right now, since users could have messed with the builtins module. Probably better to stash the funcs in interp->callable_cache like other builtins used for optimisation?

Very cool! Any plans to do this for set or list?

dict/max/min/sum are trickier, but may be worth exploring too.

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

This should fix the JIT builds:

diff --git a/Tools/jit/_targets.py b/Tools/jit/_targets.py
index aa2b56abf44..73a4210eee0 100644
--- a/Tools/jit/_targets.py
+++ b/Tools/jit/_targets.py
@@ -522,7 +522,14 @@ def get_target(host: str) -> _COFF | _ELF | _MachO:
         args = ["-fms-runtime-lib=dll"]
         target = _COFF(host, args=args)
     elif re.fullmatch(r"x86_64-.*-linux-gnu", host):
-        args = ["-fno-pic", "-mcmodel=medium", "-mlarge-data-threshold=0"]
+        args = [
+            # Jump tables generate R_X86_64_32S relocations, which assume a
+            # signed 32-bit address space:
+            "-fno-jump-tables",
+            "-fno-pic",
+            "-mcmodel=medium",
+            "-mlarge-data-threshold=0",
+        ]
         target = _ELF(host, args=args)
     else:
         raise ValueError(host)

The overall approach looks good.

Do you have stats for this PR?
If there is enough of an increase in the execution counts for CALL_INTRINSIC_1 then we should move CALL_INTRINSIC_1 INTRINSIC_LIST_TO_TUPLE into a new LIST_TO_TUPLE instruction.

dict/max/min/sum are trickier, but may be worth exploring too.

The latter three would be nice wins.

Perf results are here: https://github.com/faster-cpython/benchmarking-public/blob/main/results/bm-20250326-3.14.0a6%2B-50dd66b/README.md

Do you have stats for this PR? If there is enough of an increase in the execution counts for CALL_INTRINSIC_1 then we should move CALL_INTRINSIC_1 INTRINSIC_LIST_TO_TUPLE into a new LIST_TO_TUPLE instruction.

I have made the requested changes; please review again.

Thanks for making the requested changes!

@brandtbucher: please review the changes made to this pull request.

Looks good. You can revert the JIT change, it's not needed anymore.

(Also I'm running JIT benchmarks just because I'm curious, but no need to block on those results at all.)

Is this needed?

Yes, I had to remove this include from another header so it needs to be included when needed.

(Also, I think you need to make regen-cases.)

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>

…r expression arg (python#131737)

## Summary Closes #12912 by only applying the tuple generator behavior of the rule on Python 3.14+, where tuple generator expression got optimized in (ref: python/cpython#131737). Rendering the concerns on that issue no longer relevant on those Python versions. This was especially problematic for `C409` as only some behaviour is in preview, not the rule itself, so you couldn't use `tool.ruff.lint.explicit-preview-rules = true` ## Test Plan New snapshot should pass on CI --- Note: Code changed by Claude Sonnet 4.6. Reviewed by human, although I don't know much about Rust (but this seems a simple enough change)

iritkatriel added 2 commits March 25, 2025 20:28

add CONSTANT_BUILTIN_ALL/ANY/TUPLE oparg for LOAD_COMMON_CONST

8de7276

implement all/any/tuple optimization

df6bccf

iritkatriel requested a review from markshannon as a code owner March 25, 2025 20:35

bedevere-app Bot added the awaiting core review label Mar 25, 2025

iritkatriel changed the title optimize any/all/tuple calls with a generator expression arg gh-131738: optimize builtin any/all/tuple calls with a generator expression arg Mar 25, 2025

bedevere-app Bot mentioned this pull request Mar 25, 2025

optimize any/all/tuple calls with a generator expression arg #131738

Closed

📜🤖 Added by blurb_it.

045cdc9

iritkatriel commented Mar 25, 2025

View reviewed changes

Comment thread Misc/NEWS.d/next/Core_and_Builtins/2025-03-25-20-38-06.gh-issue-131738.eCb0OQ.rst Outdated

iritkatriel added 3 commits March 25, 2025 20:50

add comment

d142f19

fix warning

cb4eddb

fixup

c3bb232

use interp->callable_cache

a217fc4

iritkatriel requested a review from ericsnowcurrently as a code owner March 25, 2025 22:15

brandtbucher requested changes Mar 25, 2025

View reviewed changes

Comment thread Lib/opcode.py

Comment thread Python/bytecodes.c Outdated

Comment thread Python/codegen.c Outdated

Comment thread Python/codegen.c

Comment thread Python/codegen.c Outdated

Comment thread Python/codegen.c Outdated

bedevere-app Bot added awaiting changes and removed awaiting core review labels Mar 25, 2025

tweaks from review

9f71f16

markshannon reviewed Mar 26, 2025

View reviewed changes

Comment thread Python/bytecodes.c Outdated

iritkatriel added 3 commits March 26, 2025 17:47

avoid duplicating name lookup

10ef702

remove pop

222ef3b

fix build

ef78fd1

iritkatriel requested a review from savannahostrowski as a code owner March 26, 2025 18:05

use table for common consts

e4b3ccd

iritkatriel added 2 commits March 26, 2025 21:39

typo

7ae52ab

Merge remote-tracking branch 'upstream/main' into tuple

a495ef4

iritkatriel commented Mar 26, 2025

View reviewed changes

Comment thread Python/bytecodes.c Outdated

Update Python/bytecodes.c

50dd66b

bedevere-app Bot added awaiting change review and removed awaiting changes labels Mar 27, 2025

bedevere-app Bot requested a review from brandtbucher March 27, 2025 11:10

brandtbucher approved these changes Mar 27, 2025

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting change review labels Mar 27, 2025

iritkatriel and others added 2 commits March 28, 2025 10:07

Update Tools/jit/_targets.py

d5071b3

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>

regen-cases

44b9e8d

iritkatriel enabled auto-merge (squash) March 28, 2025 10:13

iritkatriel merged commit 2c8f329 into python:main Mar 28, 2025

bedevere-app Bot removed the awaiting merge label Mar 28, 2025

davfsa mentioned this pull request Mar 31, 2025

SIM110: Recommend proper usage of all and any astral-sh/ruff#16904

Open

diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 1, 2025

pythongh-131738 : optimize builtin any/all/tuple calls with a generato…

8a283af

…r expression arg (python#131737)

opk12 mentioned this pull request Jul 12, 2025

New Rule: Prefer list comprehension over generator comprehensions to create tuples astral-sh/ruff#11839

Open

Avasam mentioned this pull request Mar 27, 2026

C409 now makes code slower astral-sh/ruff#12912

Closed

stestagg mentioned this pull request Apr 29, 2026

Segfault with async generator expressions #149122

Closed

Avasam mentioned this pull request Jun 6, 2026

Drop C409 tuple comprehension preview behaviour astral-sh/ruff#25707

Merged

		#include "pycore_warnings.h" // _PyErr_WarnUnawaitedCoroutine()


		#include "opcode_ids.h" // RESUME, etc

gh-131738: optimize builtin any/all/tuple calls with a generator expression arg by iritkatriel · Pull Request #131737 · python/cpython · GitHub

Repository navigation

gh-131738: optimize builtin any/all/tuple calls with a generator expression arg - #131737

gh-131738: optimize builtin any/all/tuple calls with a generator expression arg#131737iritkatriel merged 17 commits intopython:mainpython/cpython:mainfrom iritkatriel:tupleiritkatriel/cpython:tupleCopy head branch name to clipboard

Conversation

iritkatriel commented Mar 25, 2025 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

TeamSpen210 commented Mar 25, 2025

Uh oh!

brandtbucher left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-app Bot commented Mar 25, 2025

Uh oh!

brandtbucher commented Mar 26, 2025

Uh oh!

Uh oh!

markshannon commented Mar 26, 2025

Uh oh!

rhettinger commented Mar 26, 2025

Uh oh!

Uh oh!

iritkatriel commented Mar 27, 2025

Uh oh!

iritkatriel commented Mar 27, 2025

Uh oh!

iritkatriel commented Mar 27, 2025

Uh oh!

bedevere-app Bot commented Mar 27, 2025

Uh oh!

brandtbucher left a comment

Choose a reason for hiding this comment

Uh oh!

brandtbucher Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

iritkatriel Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

brandtbucher commented Mar 27, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

gh-131738: optimize builtin any/all/tuple calls with a generator expression arg#131737
iritkatriel merged 17 commits into
python:mainfrom
iritkatriel:tuple

iritkatriel commented Mar 25, 2025 •
edited by bedevere-app Bot
Loading

brandtbucher left a comment •
edited
Loading