GH-126491: GC: Mark objects reachable from roots before doing cycle collection by markshannon · Pull Request #127110 · python/cpython · GitHub

This is an updated version of #126502

It differs as follows:

The removal of lazy dict untracking has already been merged, so isn't present
More care is taken to ensure the consistency of the spaces
Some adjustments to the "work to do" calculation

Issue: Mark all objects reachable from roots as live before doing main cyclic GC pass #126491

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 698abb3 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

!buildbot Android

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 6d8a0d4 🤖

The command will test the builders whose names match following regular expression: Android

The builders matched are:

AMD64 Android PR
aarch64 Android PR

!buildbot Android

🤖 New build scheduled with the buildbot fleet by @markshannon for commit d9632c6 🤖

The command will test the builders whose names match following regular expression: Android

The builders matched are:

AMD64 Android PR
aarch64 Android PR

!buildbot Android

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 0702959 🤖

The command will test the builders whose names match following regular expression: Android

The builders matched are:

AMD64 Android PR
aarch64 Android PR

!buildbot iOS

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 0702959 🤖

The command will test the builders whose names match following regular expression: iOS

The builders matched are:

iOS ARM64 Simulator PR

🤖 New build scheduled with the buildbot fleet by @markshannon for commit eaea41e 🤖

The command will test the builders whose names match following regular expression: Android

The builders matched are:

AMD64 Android PR
aarch64 Android PR

!buildbot iOS

🤖 New build scheduled with the buildbot fleet by @markshannon for commit eaea41e 🤖

The command will test the builders whose names match following regular expression: iOS

The builders matched are:

iOS ARM64 Simulator PR

!buildbot iOS

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 79ab26c 🤖

The command will test the builders whose names match following regular expression: iOS

The builders matched are:

iOS ARM64 Simulator PR

!buildbot Android

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 79ab26c 🤖

The command will test the builders whose names match following regular expression: Android

The builders matched are:

AMD64 Android PR
aarch64 Android PR

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 79ab26c 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

Performance shows a ~3% speedup. The results appear noisier than usual, so might be worth repeating, but they are roughly inline with results for earlier versions of this PR.

Maybe this can be rolled into the collecting field (like 0 for not collecting, 1 for MARK, 2 for COLLECT)?

Perhaps, but not very cleanly.
The phase depends on where we are in a full heap scavenge. The collecting flag will flip many times during a scavenge, for incremental collections.

All the buildbots that failed were failing on main due to unrelated issues (the Windows 10 bot may have been fixed since)

… doing cycle collection (pythonGH-127110)" This reverts commit a8dd821.

…e doing cycle collection (pythonGH-127110)" This reverts commit 2923163.

… doing cycle collection (pythonGH-127110)" This reverts commit a8dd821.

…ycle collection (pythonGH-127110) * Mark almost all reachable objects before doing collection phase * Add stats for objects marked * Visit new frames before each increment * Update docs * Clearer calculation of work to do.

markshannon added 11 commits November 21, 2024 09:29

Change the young generation to 'not visited'. WIP

2b0a2c5

Tidy up visited tracking and add consistency asserts

754bb68

Partially apply mark-first-gc patch

ac3ad26

Rename get_heap_size

5c3a5ef

Set visited on entry frame

0ac1c07

Apply more diff, but comment out marking step

bb51b19

Refactor work to do calculation

be636c9

Turn on marking. Explain issue with tuple untracking

287f381

Add news

7e920d6

Add stats for marking

0b46494

Do not touch permanent generation

698abb3

markshannon requested a review from ericsnowcurrently as a code owner November 21, 2024 16:37

bedevere-app Bot mentioned this pull request Nov 21, 2024

Mark all objects reachable from roots as live before doing main cyclic GC pass #126491

Open

bedevere-app Bot added the awaiting core review label Nov 21, 2024

markshannon added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 21, 2024

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 21, 2024

Move informative failure message in test

6d8a0d4

Increase threshold a bit

d9632c6

Increase number of iterations

0702959

markshannon added 2 commits November 22, 2024 11:53

Return number of iterations to 20k

6063dc8

Fix compiler warning

67b1e05

Add missing macro

79ab26c

markshannon added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Nov 22, 2024

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Nov 22, 2024

iritkatriel reviewed Nov 25, 2024

View reviewed changes

iritkatriel approved these changes Dec 2, 2024

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Dec 2, 2024

markshannon merged commit a8dd821 into python:main Dec 2, 2024

bedevere-app Bot removed the awaiting merge label Dec 2, 2024

markshannon mentioned this pull request Dec 2, 2024

GH-126491: Lower heap size limit with faster marking #127519

Merged

mpage added a commit to mpage/cpython that referenced this pull request Dec 3, 2024

Revert "pythonGH-126491 : GC: Mark objects reachable from roots before…

2923163

… doing cycle collection (pythonGH-127110)" This reverts commit a8dd821.

mpage added a commit to mpage/cpython that referenced this pull request Dec 4, 2024

Reapply "pythonGH-126491 : GC: Mark objects reachable from roots befor…

4049072

…e doing cycle collection (pythonGH-127110)" This reverts commit 2923163.

mpage added a commit to mpage/cpython that referenced this pull request Dec 4, 2024

Revert "pythonGH-126491 : GC: Mark objects reachable from roots before…

0f5d11c

… doing cycle collection (pythonGH-127110)" This reverts commit a8dd821.

markshannon deleted the mark-first-2 branch January 10, 2025 16:23

nascheme mentioned this pull request Apr 28, 2025

Incremental "mark alive" pass for cyclic GC #126511

Closed

AA-Turner mentioned this pull request May 7, 2025

Increase the heap size limit to 27_500 in test_incremental_gc_handles_fast_cycle_creation #133561

Closed

sergey-miryanov mentioned this pull request Nov 24, 2025

GH-141890: Fix GC double-counting of marked objects #141891

Closed

pablogsal mentioned this pull request Oct 11, 2026

gh-159137: Restore known-root marking for the GIL garbage collector #159162

Closed

                   Py_ssize_t work_to_do;
                   /* Which of the old spaces is the visited space */
                   int visited_space;
+                  int phase;

GH-126491: GC: Mark objects reachable from roots before doing cycle collection by markshannon · Pull Request #127110 · python/cpython · GitHub

Repository navigation

GH-126491: GC: Mark objects reachable from roots before doing cycle collection - #127110

GH-126491: GC: Mark objects reachable from roots before doing cycle collection#127110markshannon merged 18 commits intopython:mainpython/cpython:mainfrom faster-cpython:mark-first-2faster-cpython/cpython:mark-first-2Copy head branch name to clipboard

Conversation

markshannon commented Nov 21, 2024 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Nov 21, 2024

Uh oh!

markshannon commented Nov 22, 2024

Uh oh!

bedevere-bot commented Nov 22, 2024

Uh oh!

markshannon commented Nov 22, 2024

Uh oh!

bedevere-bot commented Nov 22, 2024

Uh oh!

markshannon commented Nov 22, 2024

Uh oh!

bedevere-bot commented Nov 22, 2024

Uh oh!

markshannon commented Nov 22, 2024

Uh oh!

bedevere-bot commented Nov 22, 2024

Uh oh!

bedevere-bot commented Nov 22, 2024

Uh oh!

markshannon commented Nov 22, 2024

Uh oh!

bedevere-bot commented Nov 22, 2024

Uh oh!

markshannon commented Nov 22, 2024

Uh oh!

bedevere-bot commented Nov 22, 2024

Uh oh!

markshannon commented Nov 22, 2024

Uh oh!

bedevere-bot commented Nov 22, 2024

Uh oh!

bedevere-bot commented Nov 22, 2024

Uh oh!

markshannon commented Nov 23, 2024

Uh oh!

iritkatriel Nov 25, 2024

Choose a reason for hiding this comment

Uh oh!

markshannon Dec 2, 2024

Choose a reason for hiding this comment

Uh oh!

markshannon commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GH-126491: GC: Mark objects reachable from roots before doing cycle collection#127110
markshannon merged 18 commits into
python:mainfrom
faster-cpython:mark-first-2

markshannon commented Nov 21, 2024 •
edited by bedevere-app Bot
Loading

markshannon commented Dec 2, 2024 •
edited
Loading