gh-153946: Add API to defer automatic garbage collection by pablogsal · Pull Request #154188 · python/cpython · GitHub

Issue: Add a private API to defer automatic garbage collection #153946

I ran this a few times on release builds pinned to one CPU, comparing aeedae8 with this branch. These timings are the median of the medians from three independent runs, with 15-20 samples and 10 iterations per sample.

Workload	main	This PR	Change	Automatic collections
Parse eight large stdlib files into AST objects	96.22 ms	78.20 ms	18.7% faster	8 per batch -> 0
marshal.loads(), 50,000 records	12.73 ms	10.49 ms	17.6% faster	1 per call -> 0

I also ran perf stat -r 5 on the same CPython workloads. The AST case went from 26.25B to 24.14B instructions, 8.1% fewer, and task clock dropped 18.5%. The marshal case went from 4.25B to 3.57B instructions, 15.9% fewer, and task clock dropped 17.8%.

I tried scheduling a collection directly from _PyGC_ResumeAutomaticCollection() as well. That made AST conversion about 3% slower than main, so this version keeps counting and waits for the next normal scheduling point. I think this is the right behavior here.

Do we need a similar check before this?

cpython/Python/gc.c

Lines 1486 to 1497 in fe0cc2b

    
           /* update collection and allocation counters */ 
        
           if (generation+1 < NUM_GENERATIONS) { 
        
               gcstate->generations[generation+1].count += 1; 
        
           } 
        
           for (i = 0; i <= generation; i++) { 
        
               gcstate->generations[i].count = 0; 
        
           } 
        
           /* merge younger generations with one we are currently collecting */ 
        
           for (i = 0; i < generation; i++) { 
        
               gc_list_merge(GEN_HEAD(gcstate, i), GEN_HEAD(gcstate, generation)); 
        
           }

This is a very shy question, so please treat it with a huge grain of salt. My hunch is that GC callbacks may still cause race, since they're just regular Python code and may release GIL:

cpython/Python/gc.c

Lines 1467 to 1472 in fe0cc2b

    
           GC_STAT_ADD(generation, collections, 1); 
        
           struct gc_generation_stats stats = { 0 }; 
        
           if (reason != _Py_GC_REASON_SHUTDOWN) { 
        
               invoke_gc_callback(tstate, "start", generation, &stats); 
        
           }

I interrogated Claude Opus to show me what is the actual place for invoking and returning from the callbacks, and that's how I arrived at 1486.

@pablogsal Another odd question: what about a fork() in a free-threading build?

Let's say that there are two threads. There's a fork(), and the children inherit automatic_collection_pause_count. Since these are separate processes, there's no way to recover, unless there's a clean up after fork? (I would have to think, but I'm not sold on naive zeroing, similar to reset_asyncio_state or reset_remotedebug_data)

Truth be told, this is likely a problem with collecting too...

If that's the case, I can take a stab.

cc @nascheme

@pablogsal

Quickly reusing a test from 35cfb62:

    @unittest.skipIf(_testinternalcapi is None, "requires _testinternalcapi")
    @support.requires_fork()
    def test_defer_automatic_collection_resume_after_fork(self):
        was_enabled = gc.isenabled()
        gc.enable()
        try:
            with gc_threshold(1, 0, 0):
                _testinternalcapi.defer_automatic_gc()
                try:
                    with warnings.catch_warnings():
                        warnings.filterwarnings(
                            "ignore",
                            message="This process .* use of fork.*",
                            category=DeprecationWarning,
                        )
                        pid = os.fork()
                    if pid == 0:
                        _testinternalcapi.resume_automatic_gc()
                        before = self.total_collections()
                        objects = [[] for _ in range(10_000)]
                        os._exit(self.total_collections() <= before)
                    support.wait_process(pid, exitcode=0)
                finally:
                    _testinternalcapi.resume_automatic_gc()
        finally:
            if not was_enabled:
                gc.disable()

Getting:

maurycy@gimel cpython (agent/defer-automatic-gc 488c75b*) % ./python.exe -m test test_gc -m test_defer_automatic_collection_resume_after_fork -v
== CPython 3.16.0a0 (heads/agent/defer-automatic-gc-dirty:488c75b3c5d, Aug 26 2026, 12:48:18) [Clang 21.0.0 (clang-2100.1.1.101)]
== macOS-26.6.2-arm64-arm-64bit-Mach-O little-endian
== Python build: release with_assert
== cwd: /Users/maurycy/src/github.com/maurycy/cpython/build/test_python_worker_76137æ
== CPU count: 10
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 3262216598
0:00:00 load avg: 2.79 mem: 29.6 MiB Run 1 test sequentially in a single process
0:00:00 load avg: 2.79 mem: 29.6 MiB [1/1] test_gc
test_defer_automatic_collection_resume_after_fork (test.test_gc.GCTests.test_defer_automatic_collection_resume_after_fork) ... Assertion failed: (gcstate->automatic_collection_pause_count > 0), function _PyGC_ResumeAutomaticCollection, file gc.c, line 1798.
Fatal Python error: Aborted

Current thread 0x00000001f8ec6180 (most recent call first):
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/test_gc.py", line 216 in test_defer_automatic_collection_resume_after_fork
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/unittest/case.py", line 637 in _callTestMethod
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/unittest/case.py", line 691 in run
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/unittest/case.py", line 747 in __call__
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/unittest/runner.py", line 256 in run
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/single.py", line 84 in _run_suite
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/single.py", line 42 in run_unittest
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/single.py", line 162 in test_func
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/single.py", line 118 in regrtest_runner
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/single.py", line 165 in _load_run_test
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/single.py", line 212 in _runtest_env_changed_exc
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/single.py", line 321 in _runtest
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/single.py", line 350 in run_single_test
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/main.py", line 395 in run_test
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/main.py", line 430 in run_tests_sequentially
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/main.py", line 589 in _run_tests
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/main.py", line 624 in run_tests
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/main.py", line 817 in main
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/libregrtest/main.py", line 825 in main
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/__main__.py", line 2 in <module>
  File "<frozen runpy>", line 87 in _run_code
  File "<frozen runpy>", line 201 in _run_module_as_main

Current thread's C stack trace (most recent call first):
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _Py_DumpStack+0x44 [0x100597808]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at faulthandler_fatal_error+0x1ec [0x1005b49d8]
  Binary file "/usr/lib/system/libsystem_platform.dylib", at _sigtramp+0x38 [0x18c491744]
  Binary file "/usr/lib/system/libsystem_pthread.dylib", at pthread_kill+0x128 [0x18c4878d8]
  Binary file "/usr/lib/system/libsystem_c.dylib", at abort+0x94 [0x18c38d978]
  Binary file "/usr/lib/system/libsystem_c.dylib", at err+0x0 [0x18c38cbd4]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at gc_collect_main.cold.1+0x0 [0x100672530]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyGC_AfterFork+0x0 [0x100539d00]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/Modules/_testinternalcapi.cpython-316-darwin.so", at resume_automatic_gc+0x10 [0x1063096b8]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_VectorcallTstate+0x9c [0x10039d2d4]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _Py_VectorCallInstrumentation_StackRefSteal+0x120 [0x1004edaa8]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyEval_EvalFrameDefault+0x2c04 [0x1004f1860]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyEval_Vector+0x2cc [0x1004ed4dc]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_VectorcallTstate+0x9c [0x10039d2d4]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_VectorcallPrepend+0x168 [0x10039f0dc]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyEval_EvalFrameDefault+0x46ac [0x1004f3308]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyEval_Vector+0x2cc [0x1004ed4dc]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_VectorcallDictTstate+0x160 [0x10039d490]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_Call_Prepend+0xb0 [0x10039e3e0]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at call_method+0x7c [0x10044af98]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_MakeTpCall+0x178 [0x10039d6d8]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _Py_VectorCall_StackRefSteal+0x108 [0x1004ed6ec]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyEval_EvalFrameDefault+0x60e4 [0x1004f4d40]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyEval_Vector+0x2cc [0x1004ed4dc]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_VectorcallTstate+0x9c [0x10039d2d4]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_VectorcallPrepend+0x168 [0x10039f0dc]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyEval_EvalFrameDefault+0x3fe8 [0x1004f2c44]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyEval_Vector+0x2cc [0x1004ed4dc]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_VectorcallDictTstate+0x160 [0x10039d490]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_Call_Prepend+0xb0 [0x10039e3e0]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at call_method+0x7c [0x10044af98]
  Binary file "/Users/maurycy/src/github.com/maurycy/cpython/python.exe", at _PyObject_MakeTpCall+0x178 [0x10039d6d8]
  <truncated rest of calls>

Extension modules: _testcapi, _testinternalcapi (total: 2)
FAIL
restoring automatic collection

======================================================================
FAIL: test_defer_automatic_collection_resume_after_fork (test.test_gc.GCTests.test_defer_automatic_collection_resume_after_fork)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/test_gc.py", line 220, in test_defer_automatic_collection_resume_after_fork
    support.wait_process(pid, exitcode=0)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/maurycy/src/github.com/maurycy/cpython/Lib/test/support/__init__.py", line 2496, in wait_process
    raise AssertionError(f"process {pid} exited with code {exitcode2}, "
                         f"but exit code {exitcode} is expected")
AssertionError: process 76140 exited with code -6, but exit code 0 is expected

----------------------------------------------------------------------
Ran 1 test in 0.017s

FAILED (failures=1)
test test_gc failed
0:00:00 load avg: 2.79 mem: 31.6 MiB [1/1/1] test_gc failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_gc

Total duration: 48 ms
Total tests: run=1 (filtered) failures=1
Total test files: run=1/1 (filtered) failed=1
Result: FAILURE
[2] maurycy@gimel cpython (agent/defer-automatic-gc 488c75b*) %

bedevere-app Bot mentioned this pull request Jul 19, 2026

Add a private API to defer automatic garbage collection #153946

Open

maurycy reviewed Jul 20, 2026

View reviewed changes

maurycy reviewed Jul 25, 2026

View reviewed changes

maurycy mentioned this pull request Jul 31, 2026

gh-75537: Add gc.ensure_disabled() context manager #154999

Closed

pablogsal added 3 commits August 26, 2026 00:52

pythongh-153946 : Defer automatic GC in object graph builders

c9173ae

pythongh-153946 : Refine automatic GC deferral

9319acd

pythongh-153946 : Fix NEWS entry filename

920df6a

pablogsal force-pushed the agent/defer-automatic-gc branch from fe0cc2b to 920df6a Compare August 25, 2026 23:52

pablogsal added 3 commits August 26, 2026 01:06

pythongh-153946 : Reset automatic GC deferral after fork

35cfb62

pythongh-153946 : Suppress expected fork warning in test

488c75b

pythongh-153946 : Preserve active GC deferrals after fork

c723f52

+                  // Close the race with a deferral that started before the world stopped.
+                  if (state->reason == _Py_GC_REASON_HEAP &&
+                      _Py_atomic_load_int(
+                          &state->gcstate->automatic_collection_pause_count)) {
+                      _PyEval_StartTheWorld(interp);
+                      return false;
+                  }

               struct _gc_runtime_state {
                   /* Is automatic collection enabled? */
                   int enabled;
+                  int automatic_collection_pause_count;

gh-153946: Add API to defer automatic garbage collection by pablogsal · Pull Request #154188 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-153946: Add API to defer automatic garbage collection - #154188

gh-153946: Add API to defer automatic garbage collection#154188
pablogsal wants to merge 6 commits into
python:mainfrom
pablogsal:agent/defer-automatic-gc

pablogsal commented Jul 19, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

Uh oh!

pablogsal commented Jul 19, 2026

Uh oh!

maurycy Jul 20, 2026

Uh oh!

maurycy Jul 25, 2026 •
edited
Loading

Uh oh!

Uh oh!

maurycy Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	/* update collection and allocation counters */
	if (generation+1 < NUM_GENERATIONS) {
	gcstate->generations[generation+1].count += 1;
	}
	for (i = 0; i <= generation; i++) {
	gcstate->generations[i].count = 0;
	}

	/* merge younger generations with one we are currently collecting */
	for (i = 0; i < generation; i++) {
	gc_list_merge(GEN_HEAD(gcstate, i), GEN_HEAD(gcstate, generation));
	}

	GC_STAT_ADD(generation, collections, 1);

	struct gc_generation_stats stats = { 0 };
	if (reason != _Py_GC_REASON_SHUTDOWN) {
	invoke_gc_callback(tstate, "start", generation, &stats);
	}

gh-153946: Add API to defer automatic garbage collection by pablogsal · Pull Request #154188 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-153946: Add API to defer automatic garbage collection - #154188

gh-153946: Add API to defer automatic garbage collection#154188pablogsal wants to merge 6 commits intopython:mainpython/cpython:mainfrom pablogsal:agent/defer-automatic-gcpablogsal/cpython:agent/defer-automatic-gcCopy head branch name to clipboard

Conversation

pablogsal commented Jul 19, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal commented Jul 19, 2026

Uh oh!

maurycy Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

maurycy Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maurycy Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gh-153946: Add API to defer automatic garbage collection#154188
pablogsal wants to merge 6 commits into
python:mainfrom
pablogsal:agent/defer-automatic-gc

pablogsal commented Jul 19, 2026 •
edited by bedevere-app Bot
Loading

maurycy Jul 25, 2026 •
edited
Loading