Mimalloc was added as an allocator in #90815. The --disable-gil builds need further integration with mimalloc, as well as some modifications to mimalloc to support thread-safe garbage collection in --disable-gil builds and the dictionary accesses that mostly avoid locking.
These changes can be split up across multiple PRs.
Currently, when mimalloc is enabled, all allocations go to the default heap. This is fine for PyMem_Malloc calls, but we need separate heaps for PyObject_Malloc and PyObject_GC_New. We should associate some mi_heap_ts with each PyThreadState. Every PyThreadState needs four heaps: one for PyMem_Malloc, one for non-GC objects (via PyObject_Malloc), one for GC objects with managed dicts (extra pre-header) and one for GC objects without a managed dict. We need some way to know which heap to use in _PyObject_MiMalloc. There's not a great way to do this, but I suggest adding something like a "current pyobject heap" variable to PyThreadState. It should generally point to the PyObject_Malloc heap, but PyObject_GC_New should temporarily override it to point to the correct GC heap when called
--disable-gil should imply --with-mimalloc and require mimalloc (i.e., disallow changing the allocator with PYTHONMALLOC).
We should tag each mi_heap_t and mi_page_t with a number identifying which type of allocation it's associated with. This is important for when pages are abandoned (i.e., when a thread exits with live blocks remaining) and the page is no longer associated with a heap. The GC still needs to identify which of those pages store GC-enabled objects. (see colesbury/nogil-3.12@d447b69)
When claiming a page from an abandoned segment, mimalloc should associate it with the correct heap from the current thread. In other words, pages that store GC-enabled objects should only be put back in the correct GC heap.
Activity
pythongh-112532: Require mimalloc in `--disable-gil` builds
gh-112532: Require mimalloc in `--disable-gil` builds (gh-112883)
pythongh-112532: Use separate mimalloc heaps for GC objects
pythongh-112532: Use separate mimalloc heaps for GC objects
pythongh-112532: Use separate mimalloc heaps for GC objects
pythongh-112532: Use separate mimalloc heaps for GC objects
pythongh-112532: Isolate abandoned segments by interpreter
gh-112532: Use separate mimalloc heaps for GC objects (gh-113263)
pythongh-112532: Fix peg generator for mimalloc build
gh-112532: Fix peg generator build.py for mimalloc build (gh-113492)
pythongh-112532: Isolate abandoned segments by interpreter
28 remaining items
pythongh-112532: Improve mimalloc page visiting (python#114133)
colesbury commented on Feb 14, 2024
I think think this is done now. There are still a few QSBR-related mimalloc changes, but those will be tracked in #115103.
pythongh-112532: Require mimalloc in `--disable-gil` builds (pythongh…
pythongh-112532: Use separate mimalloc heaps for GC objects (pythongh…
pythongh-112532: Fix peg generator build.py for mimalloc build (pytho…
pythongh-112532: Isolate abandoned segments by interpreter (python#11…
pythongh-112532: Tag mimalloc heaps and pages (python#113742)
pythongh-112532: Fix memory block count for free-threaded build (pyth…
pythongh-112532: Improve mimalloc page visiting (python#114133)
theap: let a parked thread hand its heaps to the scavenger
Re-apply CPython-specific changes to mimalloc v2.5.2
Re-apply CPython-specific changes to mimalloc v2.5.2
gh-159111: Update vendored mimalloc to v2.5.2 (gh-159112)
[3.15] pythongh-159111: Update vendored mimalloc to v2.5.2 (pythongh-…