Commit bd0e4cb
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -15,15 +15,50 @@ See also :ref:`Remove functions <remove-funcs>`. | |||
| 15 | 15 | Borrowed references | |
| 16 | 16 | =================== | |
| 17 | 17 | ||
| 18 | - A borrowed reference is a pointer which doesn't "hold" a reference. If the | ||
| 19 | - object is destroyed, the borrowed reference becomes a dangling pointer: pointer | ||
| 20 | - pointing to freed memory which might be reused by a new object. | ||
| 18 | + Problem caused by borrowed references | ||
| 19 | + ------------------------------------- | ||
| 21 | 20 | ||
| 22 | - Borrowed references can lead to bugs and crashes when misused. The | ||
| 23 | - :ref:`Specialized list for small integers <specialized-list>` optimization is | ||
| 21 | + A borrowed reference is a pointer which doesn't "hold" a reference. If the | ||
| 22 | + object is destroyed, the borrowed reference becomes a `dangling pointer | ||
| 23 | + <https://en.wikipedia.org/wiki/Dangling_pointer>`_: point to freed memory which | ||
| 24 | + might be reused by a new object. Borrowed references can lead to bugs and | ||
| 25 | + crashes when misused. Recent example of CPython bug: `bpo-25750: crash in | ||
| 26 | + type_getattro() <https://bugs.python.org/issue25750>`_. | ||
| 27 | + | ||
| 28 | + Borrowed references are a problem whenever there is no reference to borrow: | ||
| 29 | + they assume that a reified object already exists (and thus have a positive | ||
| 30 | + refcout), so that it is just borrowed. | ||
| 31 | + | ||
| 32 | + :ref:`Tagged pointers <tagged-pointer>` are an example of this: since there is | ||
| 33 | + no concrete ``PyObject*`` to represent the integer, it cannot easily be | ||
| 34 | + manipulated. | ||
| 35 | + | ||
| 36 | + PyPy has a similar problem with list strategies: if there is a list containing | ||
| 37 | + only integers, it is stored as a compact C array of longs, and the W_IntObject | ||
| 38 | + is only created when an item is accessed (most of the time the W_IntObject is | ||
| 39 | + optimized away by the JIT, but this is another story). | ||
| 40 | + | ||
| 41 | + But for cpyext, this is a problem: ``PyList_GetItem()`` returns a borrowed | ||
| 42 | + reference, but there is no any concrete ``PyObject*`` to return! The current | ||
| 43 | + ``cpyext`` solution is very bad: basically, the first time ``PyList_GetItem()`` | ||
| 44 | + is called, the *whole* list is converted to a list of ``PyObject*``, just to | ||
| 45 | + have something to return: see `cpyext get_list_storage() | ||
| 46 | + <https://bitbucket.org/pypy/pypy/src/b9bbd6c0933349cbdbfe2b884a68a16ad16c3a8a/pypy/module/cpyext/listobject.py#lines-28>`_. | ||
| 47 | + | ||
| 48 | + See also the :ref:`Specialized list for small integers <specialized-list>` | ||
| 49 | + optimization: same optimization applied to CPython. This optimization is | ||
| 24 | 50 | incompatible with borrowed references, since the runtime cannot guess when the | |
| 25 | 51 | temporary object should be destroyed. | |
| 26 | 52 | ||
| 53 | + | ||
| 54 | + If ``PyList_GetItem()`` returned a strong reference, the ``PyObject*`` could | ||
| 55 | + just be allocated on the fly and destroy it when the user decref it. Basically, | ||
| 56 | + by putting borrowed references in the API, we are fixing in advance the data | ||
| 57 | + structure to use! | ||
| 58 | + | ||
| 59 | + C API using borrowed references | ||
| 60 | + ------------------------------- | ||
| 61 | + | ||
| 27 | 62 | CPython 3.7 has many functions and macros which return or use borrowed | |
| 28 | 63 | references. For example, ``PyTuple_GetItem()`` returns a borrowed reference, | |
| 29 | 64 | whereas ``PyTuple_SetItem()`` stores a borrowed reference (store an item into a | |
@@ -33,7 +68,7 @@ CPython contains ``Doc/data/refcounts.dat`` (file is edited manually) which | |||
| 33 | 68 | documents how functions handle reference count. | |
| 34 | 69 | ||
| 35 | 70 | Functions | |
| 36 | - --------- | ||
| 71 | + ^^^^^^^^^ | ||
| 37 | 72 | ||
| 38 | 73 | * ``PyDict_GetItem()`` | |
| 39 | 74 | * ``PyDict_GetItemWithError()`` | |
@@ -56,6 +91,7 @@ Functions | |||
| 56 | 91 | * ``PyMethod_Class()`` | |
| 57 | 92 | * ``PyMethod_Function()`` | |
| 58 | 93 | * ``PyMethod_Self()`` | |
| 94 | + * ``PyModule_AddObject()``: steal a reference on success, but doesn't on error | ||
| 59 | 95 | * ``PyModule_GetDict()`` | |
| 60 | 96 | * ``PyNumber_Check()`` | |
| 61 | 97 | * ``PyObject_Init()`` | |
@@ -80,7 +116,8 @@ Macros | |||
| 80 | 116 | * ``PyTuple_SET_ITEM()`` | |
| 81 | 117 | * ``PyWeakref_GET_OBJECT()`` | |
| 82 | 118 | ||
| 83 | - Border line: | ||
| 119 | + Border line | ||
| 120 | + ^^^^^^^^^^^ | ||
| 84 | 121 | ||
| 85 | 122 | * ``Py_SETREF()``, ``Py_XSETREF()``: the caller has to manually increment the | |
| 86 | 123 | reference counter of the new value | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -64,6 +64,8 @@ decrease and test in ``Py_DECREF()``) the reference count has been proposed, | |||
| 64 | 64 | but experiment showed a slowdown of 20% on single threaded microbenchmarks. | |
| 65 | 65 | ||
| 66 | 66 | ||
| 67 | + .. _tagged-pointer: | ||
| 68 | + | ||
| 67 | 69 | Tagged pointers: doable | |
| 68 | 70 | ======================= | |
| 69 | 71 | ||
0 commit comments