Python Python (private heap) Python (Python memory manager) Python (sharing) (segmentation) (preallocation) (caching)

(raw memory allocator) Python /Python

It is important to understand that the management of the Python heap is performed by the interpreter itself and that the user has no control over it, even if she regularly manipulates object pointers to memory blocks inside that heap. The allocation of heap space for Python objects and other internal buffers is performed on demand by the Python memory manager through the Python/C API functions listed in this document.

Python C : malloc() calloc() realloc() free() C Python C Python C :

PyObject *res;
char *buf = (char *) malloc(BUFSIZ); /* for I/O */

if (buf == NULL)
    return PyErr_NoMemory();
...Do some I/O operation involving buf...
res = PyBytes_FromString(buf);
free(buf); /* malloc'ed */
return res;

I/O C Python

Python Python Python C Python Python Python Python C I/O Python

PYTHONMALLOC Python

PYTHONMALLOCSTATS pymalloc

GIL

: malloc(), calloc(), realloc(), free() 0 malloc(1) ( calloc(1, 1))

3.4 .

void* PyMem_RawMalloc(size_t n)

n void* NULL

0 PyMem_RawMalloc(1) NULL

void* PyMem_RawCalloc(size_t nelem, size_t elsize)

elsize nelem void* NULL

0 PyMem_RawCalloc(1, 1) NULL

3.5 .

void* PyMem_RawRealloc(void *p, size_t n)

p n

p NULL PyMem_RawMalloc(n) n NULL

p NULL p PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawCalloc()

PyMem_RawRealloc() NULL p

void PyMem_RawFree(void *p)

Frees the memory block pointed to by p, which must have been returned by a previous call to PyMem_RawMalloc(), PyMem_RawRealloc() or PyMem_RawCalloc(). Otherwise, or if PyMem_RawFree(p) has been called before, undefined behavior occurs.

p NULL

Python ANSI C 0 :

pymalloc

GIL

3.6 : malloc() pymalloc

void* PyMem_Malloc(size_t n)

n void* NULL

0 PyMem_Malloc(1) NULL

void* PyMem_Calloc(size_t nelem, size_t elsize)

elsize nelem void* NULL

0 PyMem_Calloc(1, 1) NULL

3.5 .

void* PyMem_Realloc(void *p, size_t n)

p n

p NULL PyMem_Malloc(n) n NULL

p NULL p PyMem_Malloc(), PyMem_Realloc() PyMem_Calloc()

PyMem_Realloc() NULL p

void PyMem_Free(void *p)

p p PyMem_Malloc() PyMem_Realloc() PyMem_Calloc() PyMem_Free(p)

p NULL

TYPE C

TYPE* PyMem_New(TYPE, size_t n)

PyMem_Malloc() (n * sizeof(TYPE)) TYPE*

TYPE* PyMem_Resize(void *p, TYPE, size_t n)

PyMem_Realloc() (n * sizeof(TYPE)) TYPE* p NULL

C p p

void PyMem_Del(void *p)

PyMem_Free()

C API Python Python

  • PyMem_MALLOC(size)
  • PyMem_NEW(type, size)
  • PyMem_REALLOC(ptr, size)
  • PyMem_RESIZE(ptr, type, size)
  • PyMem_FREE(ptr)
  • PyMem_DEL(ptr)

Object allocators

Python ANSI C 0 :

pymalloc

GIL

void* PyObject_Malloc(size_t n)

n void* NULL

Requesting zero bytes returns a distinct non-NULL pointer if possible, as if PyObject_Malloc(1) had been called instead. The memory will not have been initialized in any way.

void* PyObject_Calloc(size_t nelem, size_t elsize)

elsize nelem void* NULL

Requesting zero elements or elements of size zero bytes returns a distinct non-NULL pointer if possible, as if PyObject_Calloc(1, 1) had been called instead.

3.5 .

void* PyObject_Realloc(void *p, size_t n)

p n

If p is NULL, the call is equivalent to PyObject_Malloc(n); else if n is equal to zero, the memory block is resized but is not freed, and the returned pointer is non-NULL.

Unless p is NULL, it must have been returned by a previous call to PyObject_Malloc(), PyObject_Realloc() or PyObject_Calloc().

If the request fails, PyObject_Realloc() returns NULL and p remains a valid pointer to the previous memory area.

void PyObject_Free(void *p)

Frees the memory block pointed to by p, which must have been returned by a previous call to PyObject_Malloc(), PyObject_Realloc() or PyObject_Calloc(). Otherwise, or if PyObject_Free(p) has been called before, undefined behavior occurs.

p NULL

3.4 .

PyMemAllocatorEx

4:
void *ctx
void* malloc(void *ctx, size_t size)
void* calloc(void *ctx, size_t nelem, size_t elsize) 0
void* realloc(void *ctx, void *ptr, size_t new_size)
void free(void *ctx, void *ptr)

3.5 : PyMemAllocator PyMemAllocatorEx calloc

PyMemAllocatorDomain

:

PYMEM_DOMAIN_RAW

:

PYMEM_DOMAIN_MEM

:

PYMEM_DOMAIN_OBJ

:

void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)

void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)

0 NULL

PYMEM_DOMAIN_RAW : GIL

(1) PyMem_SetupDebugHooks()

void PyMem_SetupDebugHooks(void)

Python

0xCB 0xDB

:

On error, the debug hooks use the tracemalloc module to get the traceback where a memory block was allocated. The traceback is only displayed if tracemalloc is tracing Python memory allocations and the memory block was traced.

These hooks are installed by default if Python is compiled in debug mode. The PYTHONMALLOC environment variable can be used to install debug hooks on a Python compiled in release mode.

3.6 : This function now also works on Python compiled in release mode. On error, the debug hooks now use tracemalloc to get the traceback where a memory block was allocated. The debug hooks now also check if the GIL is held when functions of PYMEM_DOMAIN_OBJ and PYMEM_DOMAIN_MEM domains are called.

pymalloc

Python (512) pymalloc pymalloc 256 KB "" 512 PyMem_RawMalloc() PyMem_RawRealloc()

pymalloc is the default allocator of the PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) and PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) domains.

:

  • Windows VirtualAlloc() VirtualFree()
  • mmap() munmap()
  • malloc() free()

pymalloc

3.4 .

PyObjectArenaAllocator

3:
void *ctx
void* alloc(void *ctx, size_t size) size
void free(void *ctx, size_t size, void *ptr)
PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)

PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)

Python I/O :

PyObject *res;
char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */

if (buf == NULL)
    return PyErr_NoMemory();
/* ...Do some I/O operation involving buf... */
res = PyBytes_FromString(buf);
PyMem_Free(buf); /* allocated with PyMem_Malloc */
return res;

:

PyObject *res;
char *buf = PyMem_New(char, BUFSIZ); /* for I/O */

if (buf == NULL)
    return PyErr_NoMemory();
/* ...Do some I/O operation involving buf... */
res = PyBytes_FromString(buf);
PyMem_Del(buf); /* allocated with PyMem_New */
return res;

API (Fatal) 

char *buf1 = PyMem_New(char, BUFSIZ);
char *buf2 = (char *) malloc(BUFSIZ);
char *buf3 = (char *) PyMem_Malloc(BUFSIZ);
...
PyMem_Del(buf3);  /* Wrong -- should be PyMem_Free() */
free(buf2);       /* Right -- allocated via malloc() */
free(buf1);       /* Fatal -- should be PyMem_Del()  */

Python PyObject_New() PyObject_NewVar() PyObject_Del() Python

C