| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
|
According to NEWS when these were added, they were unstable. Presumably because they mess with internals that might change. Then shouldn't they be named PyUnstable_...? Or, @markshannon have you changed your mind about their status? |
Sorry, something went wrong.
|
It's too late for 3.12 right? |
Sorry, something went wrong.
That's a question for @Yhg1s, Python 3.12 release manager. This change adds 2 functions to the public C API. I would prefer to fix the API in Python 3.12, but if it's too late, it's too late :-) |
Sorry, something went wrong.
|
If it's too late for Python 3.12, that's ok. The private functions can be used. But I will update my PR to document the change in What's new in Python 3.13 instead ;-) Just tell what you think about this one @Yhg1s ;-) |
Sorry, something went wrong.
|
No, I think it's too late for 3.12. |
Sorry, something went wrong.
What do you mean when you say that _PyObject_VisitManagedDict() and _PyObject_ClearManagedDict() are unstable? Are you referring to the risk of bugs? Or the risk that the API may change soon? Are you referring to the What's New in Python 3.12 doc? * Extension classes wanting to add a ``__dict__`` or weak reference slot should use :c:macro:`Py_TPFLAGS_MANAGED_DICT` and :c:macro:`Py_TPFLAGS_MANAGED_WEAKREF` instead of ``tp_dictoffset`` and ``tp_weaklistoffset``, respectively. The use of ``tp_dictoffset`` and ``tp_weaklistoffset`` is still supported, but does not fully support multiple inheritance (:gh:`95589`), and performance may be worse. Classes declaring :c:macro:`Py_TPFLAGS_MANAGED_DICT` should call :c:func:`!_PyObject_VisitManagedDict` and :c:func:`!_PyObject_ClearManagedDict` to traverse and clear their instance's dictionaries. To clear weakrefs, call :c:func:`PyObject_ClearWeakRefs`, as before. The "does not fully support multiple inheritance" part is scary. The PyTypeObject.tp_dictoffset documentation strongly suggests moving towards the Py_TPFLAGS_MANAGED_DICT API: .. c:member:: Py_ssize_t PyTypeObject.tp_dictoffset While this field is still supported, :c:macro:`Py_TPFLAGS_MANAGED_DICT` should be used instead, if at all possible. (...) **Inheritance:** This field is inherited by subtypes. A subtype should not override this offset; doing so could be unsafe, if C code tries to access the dictionary at the previous offset. To properly support inheritance, use :c:macro:`Py_TPFLAGS_MANAGED_DICT`. |
Sorry, something went wrong.
Ok, the release manager spoke, I re-targeted my PR to Python 3.13 :-) It's ok to use the private API in Python 3.12. |
Sorry, something went wrong.
It's literally what the NEWS entry for 3.12a1 says: .. date: 2022-07-25-15-54-27 .. gh-issue: 92678 .. nonce: ziZpxz .. section: C API Adds unstable C-API functions ``_PyObject_VisitManagedDict`` and ``_PyObject_ClearManagedDict`` to allow C extensions to allow the VM to manage their object's dictionaries. |
Sorry, something went wrong.
So @gvanrossum is worried by the Changelog entry adding these 2 APIs as "unstable". @markshannon: You added these functions 1 year ago in Python 3.12 as commit de388c0. Do you still consider these APIs as "unstable"? Are you ok to make them public? Well, see the issue and previous comments for the rationale ;-) |
Sorry, something went wrong.
|
For me, it's unfortunate that Python 3.12 broke type inheritance with the new "managed dict" change, but doesn't offer a public API to switch to it, while the documentation strongly advices using this new API. That's why I asked to make this API public in Python 3.12. But this issue is waiting for 1 month, and Python 3.12 release is following its process towards the final release in one month. It's unclear to me if inheritance was broken in Python 3.11 or 3.12. I recall that pybind11 got a regression related to that, and as I wrote. It now uses a the private _PyObject_GetDictPtr() API to visit and clear the managed dict: #107073 (comment) So it worked around the lack _PyObject_VisitManagedDict() and _PyObject_ClearManagedDict() in Python 3.11. Well, even if I dislike the implementation, it's good to know that there is a way to implement PyObject_VisitManagedDict() and PyObject_ClearManagedDict() on Python 3.11. It also means that there is a need for public API for that, since pybind11 already uses it :-) |
Sorry, something went wrong.
If we decide adding PyObject_VisitManagedDict() to the public C API, what I can is to provide a public PyObject_VisitManagedDict() function for Python 3.11 and 3.12 in the https://github.com/python/pythoncapi-compat/ project using _PyObject_GetDictPtr(). So projects can use the new public API since Python 3.11 :-) So I don't have to be pushy with our busy Python 3.12 release manager to get it merged after beta1 (after rc1!!!) ;-) |
Sorry, something went wrong.
|
@dean0x7d @wjakob: Hi, you modified pybin11 to travere and clear the dictionary in pybind11 (commit) with such code: /// dynamic_attr: Allow the garbage collector to traverse the internal instance `__dict__`.
extern "C" inline int pybind11_traverse(PyObject *self, visitproc visit, void *arg) {
PyObject *&dict = *_PyObject_GetDictPtr(self);
Py_VISIT(dict);
return 0;
}
/// dynamic_attr: Allow the GC to clear the dictionary.
extern "C" inline int pybind11_clear(PyObject *self) {
PyObject *&dict = *_PyObject_GetDictPtr(self);
Py_CLEAR(dict);
return 0;
}
I see that pybind11 uses Py_TPFLAGS_MANAGED_DICT on Python 3.11. Would it help you to have public function PyObject_VisitManagedDict() and PyObject_ClearManagedDict()? Do you have an opinion on the API? Is it good? |
Sorry, something went wrong.
|
The proposal looks fine to me. A tangent: a more general solution would be to expose the dictionary in a public (perhaps even stable) API interface, but not like the existing mechanism. _PyObject_GetDictPtr is problematic because it exposes CPython implementation details. If the signature was PyObject_GetDict() instead (returning PyObject* and not PyObject **) then that is enough information to find and break reference cycles. Those two pybind11 functions you highlighted could then be implemented as /// dynamic_attr: Allow the garbage collector to traverse the internal instance `__dict__`.
extern "C" inline int pybind11_traverse(PyObject *self, visitproc visit, void *arg) {
PyObject *dict = PyObject_GetDict(self);
Py_VISIT(dict);
return 0;
}
/// dynamic_attr: break a reference cycle involving this object
extern "C" inline int pybind11_clear(PyObject *self) {
PyObject *dict = PyObject_GetDict(self);
if (dict) {
PyDict_Clear(dict);
}
return 0;
} |
Sorry, something went wrong.
|
In short, you ask to make _PyObject_GetDict() public? Yeah, I am curious if it would make sense to make this function public as well. By the way, this function is documented as: "should be treated as part of the public API"! /* Helper to get a pointer to an object's __dict__ slot, if any.
* Creates the dict from inline attributes if necessary.
* Does not set an exception.
*
* Note that the tp_dictoffset docs used to recommend this function,
* so it should be treated as part of the public API.
*/ |
Sorry, something went wrong.
|
@vstinner Ah, interesting—I did not know about this function. The fact that it creates the dictionary if non-existent makes it inappropriate for what I had suggested (for use in garbage collection hooks). |
Sorry, something went wrong.
|
Oops, sorry for the noise :-( I tried a Git tool which messed up my PRs. I fixed my PR. |
Sorry, something went wrong.
Can we discuss exposing _PyObject_GetDict() separetly? Exposing just PyObject_VisitManagedDict() and PyObject_ClearManagedDict() already looks controversial :-) |
Sorry, something went wrong.
|
@markshannon @gvanrossum @encukou: this issue is languishing since July 22. I plan to merge my PR next Friday. If you need more time for review, please speak up. Once this PR will be merged, I consider adding a compatible layer for Python 3.11 and 3.12 in pythoncapi-compat. Later, I consider making _PyObject_GetDict() public, but apparently this function needs to be adjusted before being made public. |
Sorry, something went wrong.
|
I understand that exposing these functions would be helpful to fix garbage collection of managed dictionaries in extension libraries like pybind11. But unless there is a big rush to push this out for a release (which I don't think is the case here, right?), isn't it better to refine and have a perfect API? In this case, it somehow seems suboptimal to expose two highly specific public functions as public API if they are likely to be subsumed by a more general one later one. |
Sorry, something went wrong.
I don't think that anyone has a plan to replace these two convenient functions with something like _PyObject_GetDict(). _PyObject_GetDict() cannot be used directly in traverse and clear functions, IMO it's less convenient to use. See my Doc/c-api/typeobj.rst changes in this PR: PyObject_VisitManagedDict() and PyObject_ClearManagedDict() are convenient to use. For me, _PyObject_GetDict() fits other use cases, no? |
Sorry, something went wrong.
|
Both Mark and I are on vacation this week. Please wait. |
Sorry, something went wrong.
So, any update on these functions? Should they be exposed? Is it the new official way to have a dict in an object? Or should we burry them deeper in the internal C API? |
Sorry, something went wrong.
|
I've commented on the issue on my thinking as to why we should add these to the public API. |
Sorry, something went wrong.
| debugging aid you may want to visit it anyway just so the :mod:`gc` module's | ||
| :func:`~gc.get_referents` function will include it. | ||
|
|
||
| Heap types (:c:macro:`Py_TPFLAGS_HEAPTYPE`) must visit their type with:: |
There was a problem hiding this comment.
This doesn't seem relevant PyObject_VisitManagedDict, but no harm in adding it.
Sorry, something went wrong.
There was a problem hiding this comment.
This doc was forgotten since Python 3.8, it requires to fix an important bug.
Sorry, something went wrong.
|
Thanks for the review @gvanrossum and @markshannon, I really wasn't sure about this one. Now we should visit the _PyObject_GetDict() case, hum. |
Sorry, something went wrong.
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️Hi! The buildbot ARM64 macOS 3.x has failed when building commit fc2cb86. What do you need to do:
You can take a look at the buildbot page here: https://buildbot.python.org/all/#builders/725/builds/5825 Failed tests:
Summary of the results of the build (if available): == Click to see traceback logsremote: Enumerating objects: 26, done.
remote: Counting objects: 4% (1/24)
remote: Counting objects: 8% (2/24)
remote: Counting objects: 12% (3/24)
remote: Counting objects: 16% (4/24)
remote: Counting objects: 20% (5/24)
remote: Counting objects: 25% (6/24)
remote: Counting objects: 29% (7/24)
remote: Counting objects: 33% (8/24)
remote: Counting objects: 37% (9/24)
remote: Counting objects: 41% (10/24)
remote: Counting objects: 45% (11/24)
remote: Counting objects: 50% (12/24)
remote: Counting objects: 54% (13/24)
remote: Counting objects: 58% (14/24)
remote: Counting objects: 62% (15/24)
remote: Counting objects: 66% (16/24)
remote: Counting objects: 70% (17/24)
remote: Counting objects: 75% (18/24)
remote: Counting objects: 79% (19/24)
remote: Counting objects: 83% (20/24)
remote: Counting objects: 87% (21/24)
remote: Counting objects: 91% (22/24)
remote: Counting objects: 95% (23/24)
remote: Counting objects: 100% (24/24)
remote: Counting objects: 100% (24/24), done.
remote: Compressing objects: 6% (1/16)
remote: Compressing objects: 12% (2/16)
remote: Compressing objects: 18% (3/16)
remote: Compressing objects: 25% (4/16)
remote: Compressing objects: 31% (5/16)
remote: Compressing objects: 37% (6/16)
remote: Compressing objects: 43% (7/16)
remote: Compressing objects: 50% (8/16)
remote: Compressing objects: 56% (9/16)
remote: Compressing objects: 62% (10/16)
remote: Compressing objects: 68% (11/16)
remote: Compressing objects: 75% (12/16)
remote: Compressing objects: 81% (13/16)
remote: Compressing objects: 87% (14/16)
remote: Compressing objects: 93% (15/16)
remote: Compressing objects: 100% (16/16)
remote: Compressing objects: 100% (16/16), done.
remote: Total 26 (delta 8), reused 8 (delta 8), pack-reused 2
From https://github.com/python/cpython
* branch main -> FETCH_HEAD
Note: switching to 'fc2cb86d210555d509debaeefd370d5331cd9d93'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at fc2cb86d21 gh-107073: Make PyObject_VisitManagedDict() public (#108763)
Switched to and reset branch 'main'
In file included from ./Modules/_tkinter.c:52:
In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99:
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
int (*free_private)(); /* called to free private storage */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
struct _XImage *(*create_image)();
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
XID (*resource_alloc)(); /* allocator function */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
int (*synchandler)(); /* Synchronization handler */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
Bool (*event_vec[128])(); /* vector for wire to event */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
Status (*wire_vec[128])(); /* vector for event to wire */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
Bool (**error_vec)(); /* vector for wire to error */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
int (*savedsynchandler)(); /* user synchandler when Xlib usurps */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
typedef void (*XIMProc)();
^
void
In file included from ./Modules/tkappinit.c:17:
In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99:
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
int (*free_private)(); /* called to free private storage */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
struct _XImage *(*create_image)();
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
XID (*resource_alloc)(); /* allocator function */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
int (*synchandler)(); /* Synchronization handler */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
Bool (*event_vec[128])(); /* vector for wire to event */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
Status (*wire_vec[128])(); /* vector for event to wire */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
Bool (**error_vec)(); /* vector for wire to error */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
int (*savedsynchandler)(); /* user synchandler when Xlib usurps */
^
void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
typedef void (*XIMProc)();
^
void
9 warnings generated.
9 warnings generated.
make: *** [buildbottest] Error 5 |
Sorry, something went wrong.
Oh, @markshannon replied that we should not expose it: #107073 (comment) @wjakob: If you want, you can now use the pythoncapi-project to use PyObject_VisitManagedDict() and PyObject_ClearManagedDict() on Python 3.11 and Python 3.12, implemented using _PyObject_GetDictPtr(), see: python/pythoncapi-compat@a594354 |
Sorry, something went wrong.
|
@vstinner — those two functions would be usable for pybind11 (@rwgk, @henryiii). For nanobind, I'm still using old-style dictionaries built into each instance :-/. I don't this functionality is sufficiently exposed through limited API / stable ABI interfaces to be able to switch to managed dicts. Is it the plan that stable ABI interfaces can also use them? |
Sorry, something went wrong.
If you want a function to be exposed in the limited C API, please open a new issue for that. |
Sorry, something went wrong.
`PyObject_VisitManagedDict` and `PyObject_ClearManagedDict` were made public in python/cpython#108763. Both are available from `pythoncapi_compat.h`.
Make PyObject_VisitManagedDict() and PyObject_ClearManagedDict() functions public in Python 3.13 C API. * Rename _PyObject_VisitManagedDict() to PyObject_VisitManagedDict(). * Rename _PyObject_ClearManagedDict() to PyObject_ClearManagedDict(). * Document these functions.
| Back | FazBrowse Home | New Git URL |
Make PyObject_VisitManagedDict() and PyObject_ClearManagedDict() functions public in Python 3.13 C API.
EDIT: Target Python 3.13, not Python 3.12.
📚 Documentation preview 📚: https://cpython-previews--108763.org.readthedocs.build/