Repository navigation
Pass opaque state to PyContext_AddWatcher callback #127124
Description
Feature or enhancement
Proposal:
I would like to change the Context watcher C API (new to v3.14, see #119333) to pass an opaque pointer to the callback. Specifically, change:
cpython/Include/cpython/context.h
Lines 39 to 55 in e8bb053
to:
/*
* Context object watcher callback function. arg is the same pointer passed to
* PyContext_AddWatcher when the callback was registered. The object passed to
* the callback is event-specific; see PyContextEvent for details.
*
* if the callback returns with an exception set, it must return -1. Otherwise
* it should return 0
*/
typedef int PyContext_WatchCallback(
void *arg, PyContextEvent event, PyObject *obj);
/*
* Register a per-interpreter callback that will be invoked for context events.
* arg is an optional opaque pointer that is passed back to the callback; it
* can be used to manage state if desired.
*
* Returns a handle that may be passed to PyContext_ClearWatcher on success,
* or -1 and sets and error if no more handles are available.
*/
PyAPI_FUNC(int) PyContext_AddWatcher(
PyContext_WatchCallback *callback, void *arg);The original idea was for the callback to get any required state from the current context, but:
- The config/state is not guaranteed to be available in the context. For example, the Python code can do ctx = contextvars.Context() instead of ctx = contextvars.copy_context(). Or it can do copy_context(), but early during initialization and squirrel the context away for later use. The latter seems plausible for 3rd party libraries (where the user might not have much control).
- The same watcher callback cannot be registered multiple times simultaneously, each with its own config/state. This came up while I was refactoring TestContextObjectWatchers to add some more tests for a change I’m working on.
I’m not sure how likely either is to come up in normal use, but I think it is worth changing the design before the 3.14 release cements it.
Has this already been discussed elsewhere?
I have already discussed this feature proposal on Discourse
Links to previous discussion of this feature:
https://discuss.python.org/t/v3-14a1-design-limitations-of-pycontext-addwatcher/68177/4
cc @fried
Activity
pythongh-127124: Add opaque pointer to context watcher callback
ZeroIntensity commented on Nov 22, 2024
void *arg is a bit of a headache, and I'm not sure we have any precedent for it in the C API. There's a few things we need to worry about:
These aren't unsolvable problems, but I think the approach would be much easier if we just allowed passing a Python object, which has answered all of these questions. If the user really wants some C data, they can use a capsule.
fixup! pythongh-127124: Add opaque pointer to context watcher callback
rhansen commented on Nov 24, 2024
I can do PyObject * instead of void *. I suggested void * because that's what I'm used to with pure C and thought it might be easier for developers connecting C code to Python. I was unaware of capsules; thanks for mentioning them.
Spitballing: Instead of a function pointer + object, WDYT about taking a callable object? That would be more awkward to use from C, unless there's a utility function I'm unaware of that wraps a C function pointer inside a callable.
ZeroIntensity commented on Nov 24, 2024
That might be better, it would unlock some more use cases. You can use PyCFunction_New to wrap a C function into a Python callable, but yeah it's slightly more painful to use.
pythongh-127124: Pass optional state to context watcher callback
pythongh-127124: Pass optional state to context watcher callback
pythongh-127124: Change context watcher callback to a callable object
fixup! pythongh-127124: Pass optional state to context watcher callback
fixup! pythongh-127124: Change context watcher callback to a callable…
fixup! pythongh-127124: Change context watcher callback to a callable…