FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Fix memory leaks during interpreter and thread teardown by VladimirKras · Pull Request #8966 · RustPython/RustPython · GitHub

Repository navigation

Fix memory leaks during interpreter and thread teardown - #8966

Draft
VladimirKras wants to merge 1 commit into
RustPython:mainfrom
VladimirKras:fix/vm-memory-leaks
Draft

VladimirKras wants to merge 1 commit into
RustPython:mainfrom
VladimirKras:fix/vm-memory-leaks

Conversation

VladimirKras commented Oct 4, 2026 •
edited
Loading

Copy link
Copy Markdown

This draft reports memory retained after embedded interpreter teardown and OS-thread exit. The focus is repeated creation and shutdown of interpreters, rather than memory growth during continued execution in one long-lived interpreter. It includes small reproducers, allocation measurements, and candidate fixes to help maintainers investigate the teardown ownership problems and choose an appropriate solution.

The measured lifecycle is: create an interpreter on a dedicated OS thread, execute a small workload, destroy the interpreter, let the thread exit, join it, then sample memory. Interpreter teardown releases the interpreter-owned state; OS-thread exit also destroys its thread-local freelists. Both parts of that shutdown lifecycle matter here.

Please review the leak locations and reproducers first. The implementation here is exploratory, especially the freelist lifecycle change; its design would benefit from maintainer review before considering it for merge.

Observed problems

Location Observation on unpatched main
PyFunction The separately owned builtins reference is omitted from traversal. A function/builtins dictionary cycle survives collection, including when globals and builtins alias.
PyMemberDescriptor The declaring type is strongly owned without participating in GC traversal. Slotted classes survive collection; importlib's slotted KeyedRef class can retain its module graph.
PyWeak The owned callback is omitted from traversal. A weakref/proxy, callback and closure cycle survives collection even when the referent remains externally alive.
FreeList<T>::drop Thread teardown deliberately skips payload destructors and frees only raw object allocations. Dictionary buffers and other payload storage remain allocated across repeated thread/interpreter creation and shutdown.

The GC snippets isolate ownership edges that can keep object graphs alive after interpreter teardown. The slice snippet isolates a reference retained by a cached payload; allocation scans separately demonstrate payload buffers surviving OS-thread exit. These focused snippets use explicit collection to reproduce the ownership defects without a full embedding harness. Each fails on unpatched main (7536730ed6afb948ffa335fc113a0fcd9af0cdbf) and passes under CPython 3.14 and the candidate patch:

  • extra_tests/snippets/stdlib_gc_function_builtins.py
  • extra_tests/snippets/stdlib_gc_member_descriptor.py
  • extra_tests/snippets/stdlib_gc_weakref_callback.py
  • extra_tests/snippets/stdlib_gc_slice_payload.py

For example, run cargo run -- extra_tests/snippets/stdlib_gc_slice_payload.py. Deleting a slice should release its stop argument; the weakref assertion demonstrates that the cached payload retains it instead.

Candidate approach

The GC changes report the omitted strong-reference edges. Weakref callbacks are visited under the existing stripe lock while referents remain weak.

For freelists, the patch destroys payloads before publishing cached entries, while thread-local deallocation state is available. Reuse writes a fresh payload, and thread teardown releases only the remaining header/allocation using its complete layout. Custom freelists such as the MemoryError pool preserve their initialized-payload contract.

This is intended to respect the reason for skipping destructors during TLS teardown. It also changes when payload destruction occurs and adds a payload-state contract to freelist reuse. Maintainers may prefer a different ownership or cleanup design; the reproducers and measurements should be useful independently of that choice.

Measurements and validation

Across repeated interpreter creation/shutdown cycles, RSS sampled after the interpreter was destroyed and its thread joined originally grew about 1.72 MiB per cycle. With all candidate fixes, a 1,000-cycle run has a tail RSS slope of about 0.0044 MiB/cycle. RSS includes reachable allocations and allocator retention, so this is not a claim of zero memory growth.

Before the freelist change, macOS leaks --atExit --noContent reported 372,640 leaked bytes after ten bare interpreter creation/shutdown cycles and 410,144 bytes after ten creation/shutdown cycles in an embedding harness. With the complete patch, scans report zero leaked blocks/bytes across 100 bare interpreter creation/shutdown cycles, 100 JSON-workload interpreter creation/shutdown cycles in the embedding harness, and additional object and native-call workloads. These are workload-specific results using an optimized RustPython dependency in a dev-profile embedding binary.

  • Rust workspace tests and doctests: 1,346 passed, 18 ignored.
  • Separately configured C API tests: 116 passed, 4 ignored.
  • Release test_gc and test_weakref: 194 tests run, 28 skipped; both modules pass.
  • Python snippets: 471 passed, 17 failed. The exact same 17 failures were reproduced individually against unpatched main. Existing tests were not changed or disabled.
  • Workspace and separate C API Clippy checks, and pre-commit hooks, pass.

AI assistance: Codex (GPT-6) investigated the ownership graph, wrote the candidate changes and regression tests, and ran the checks. This draft is prepared for the contributor's review; no human review or human-use verification is claimed yet.

Traverse function builtins, member descriptor declaring types, and weakref
callbacks so unreachable cycles can be collected. Keep referents weak and
read callbacks under their existing stripe lock.

Destroy thread-local freelist payloads before caching, while TLS is still
available. Reinitialize the payload on reuse and release cached headers
with the complete allocation layout on thread exit. Preserve initialized
payloads for custom freelists such as the MemoryError pool.

Add regressions for live references, cycle collection, weakref callbacks,
and releasing a deleted slice's stop argument.

Assisted-by: Codex:gpt-6

coderabbitai Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

VladimirKras changed the title Fix embedded VM leaks in GC traversal and thread-local freelists Report reproducible embedded-VM memory leaks with candidate fixes Oct 4, 2026
VladimirKras changed the title Report reproducible embedded-VM memory leaks with candidate fixes Report memory retained after embedded interpreter and thread teardown Oct 4, 2026
VladimirKras changed the title Report memory retained after embedded interpreter and thread teardown Fix memory leaks during interpreter and thread teardown Oct 4, 2026

codspeed Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will improve performance by 70.49%

⚡ 1 improved benchmark
✅ 61 untouched benchmarks
⏩ 4 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ rustpython[20000] 14.2 ms 8.3 ms +70.49%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing VladimirKras:fix/vm-memory-leaks (0db3b7e) with main (7536730)

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant


Back | FazBrowse Home | New Git URL