Move _Py_RefTotal to PyInterpreterState · Issue #102304 · python/cpython · GitHub

(See gh-100227.)

_Py_RefTotal holds the current global total number of refcounts. It only exists if Py_REF_DEBUG is defined (implied by Py_DEBUG). It is exposed by sys.gettotalrefcount() and set by Py_INCREF(), Py_DECREF(), etc. and _Py_NewReference().

Modications to _Py_RefTotal are currently protected by the GIL so it should be moved to PyInterpreterState. For various aspects of compatibility, it makes sense to keep the _Py_RefTotal symbol around (and correct) and keep returning the global total from sys.gettotalrefcount().

Also, _Py_RefTotal is used by stable ABI extensions only where Py_REF_DEBUG is defined (unlikely) and only where built against 3.9 or earlier. Just in case, though, we must still keep the global variable around, so any solution here must respect that.

Linked PRs

From #100227 (comment):

FYI, I have a plan for _Py_RefTotal that should preserve stable ABI compatibility without breaking interpreter isolation (and without relying on granular locks). Basically:

deprecate _Py_Reftotal but keep exporting the symbol (for stable ABI compatibility)

add _PyRuntimeState.object_state.last_legacy_reftotal

add _PyRuntimeState.object_state.reftotal

update everywhere to use the reftotal field (for simplicity, add #define _Py_RefTotal _PyRuntime.object_state.reftotal)

We would update _Py_GetRefTotal() to something like this:
Py_ssize_t
_Py_GetRefTotal(void)
{
    // _Py_GetLegacyRefTotal() returns the value of the actual `_Py_RefTotal` global.
    Py_ssize_t legacy = _Py_GetLegacyRefTotal();
    _PyRuntimeState.object_state.reftotal += legacy - _PyRuntimeState.object_state.last_legacy_reftotal;
    _PyRuntimeState.object_state.last_legacy_reftotal = legacy;
    return _PyRuntimeState.object_state.reftotal;
}
All this assumes that folks are not using _Py_RefTotal directly.

For a per-interpreter GIL, I see two options:

add a granular lock around modifying _PyRuntimeState.object_state.reftotal, which would hurt the performance of incref/decref, but only on debug builds

switch to PyInterpreterState.object_state.reftotal and aggregate the values from all interpreters in _Py_GetRefTotal()

Alternately, from #100227 (comment):

_Py_RefTotal is only defined if Py_REF_DEBUG is defined.
All updates to _Py_RefTotal can be made atomic (which will have terrible performance with multiple parallel interpreters) to preserve the API.

I vote to make it atomic instead of per interpreter. I use this while debugging refleaks and making it per interp will effectively make it impossible to use. This is only used in debug builds so I am not too much concerned about performance.

gh-102304: Consolidate Direct Usage of _Py_RefTotal (gh-102514)

gh-102304: Fix Py_INCREF() for limited C API 3.9 (#105550)

pythongh-102304: Fix Py_INCREF() for limited C API 3.9 (pythonGH-105550)

pythongh-102304: Remove Py_INCREF() doc change

gh-102304: Remove Py_INCREF() doc change (#105552)

pythongh-102304: Fix Py_INCREF() for limited C API 3.9 (python#105550)

pythongh-102304: Remove Py_INCREF() doc change (python#105552)

[3.12] gh-102304: Fix Py_INCREF() for limited C API 3.9 (#105553)

pythongh-102304: Rename _Py_IncRefTotal_DO_NOT_USE_THIS()

gh-102304: Rename _Py_IncRefTotal_DO_NOT_USE_THIS() (#107193)

pythongh-102304: Rename _Py_IncRefTotal_DO_NOT_USE_THIS() (pythonGH-1…

[3.12] gh-102304: Rename _Py_IncRefTotal_DO_NOT_USE_THIS() (GH-107193) (

pythongh-102304: Rename _Py_IncRefTotal_DO_NOT_USE_THIS() (python#107193

[3.12] pythongh-102304: doc: Add links to Stable ABI and Limited C API (

[3.11] pythongh-102304: doc: Add links to Stable ABI and Limited C API (

[3.11] gh-102304: Add links to the Stable ABI and Limited C API docs (#…

Move _Py_RefTotal to PyInterpreterState · Issue #102304 · python/cpython · GitHub

Repository navigation

Move _Py_RefTotal to PyInterpreterState #102304

Linked PRs

62 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Move _Py_RefTotal to PyInterpreterState · Issue #102304 · python/cpython · GitHub

Repository navigation

Move _Py_RefTotal to PyInterpreterState #102304

Description

Linked PRs

Activity

ericsnowcurrently commented on Feb 27, 2023

ericsnowcurrently commented on Feb 27, 2023

kumaraditya303 commented on Mar 3, 2023

62 remaining items

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions