PEP 689 -- Add an unstable C-API tier · Issue #91744 · python/cpython · GitHub

On python-dev, it became clear that it would be useful to have a “semi-stable” “unstable” C-API tier which will stay stable within a minor release.

This API will go in a new directory (Include/unstable/) - see Nick's reply
You'll need to #define Py_USING_UNSTABLE_API.
(name still up for bikeshedding).
Since we're nearing Beta and there's no rush to break things, in 3.11
you only get a warning if you try to use it without the opt-in #define.
In 3.12 it'll fail.
The functions will be renamed to drop the leading underscore. The old
names will be available, and may be removed
whenever the API changes. (Ideally, the underscore should always mark
API that's fully private with no guarantees at all.)
Get SC approval (this is now PEP-689 that will need acceptance)
The API will be stable during a minor release. (As usual, for extreme
cases, exceptions are possible with SC approval.)
Docs:
- Update https://devguide.python.org/c-api/
- Update https://docs.python.org/3.10/c-api/stable.html
- Add individual reference entries for the API
- Document the new opt-in macro
All ununstable API should be tested
- PEP-523 PyCode_* functions: tested in test_code
- ...

What should be here:

Functions added in PEP-523
PyCode_New, PyCode_NewWithPosOnlyArgs
Ideally anything documented as subject to change between minor
releases. (To be kind to users, if something is added later we should
again have one release of compiler warnings before requiring the opt-in.
Unless that API just changed and users would get errors anyway.)
Cinder's dict writing API looks like it would feel at home here: add support for watching writes to selected dictionaries #91052 (comment)
PyBytesObject.ob_shash sounds like a good candidate: see https://discuss.python.org/t/15108
non-opaque access to frame structs and any other key APIs needed to
implement alternate eval loops with comparable performance to the default
eval loop (unless & until we can figure out stable public APIs that can
deliver equivalent performance) - see Nick's reply
C APIs that provide access to compiled code whether in AST or opcode
form (the API itself may be stable, but the compiled code isn't, so this is
kinda covered by your last point) - see Nick's reply
_Py_HashDouble, see comment below

First PR: #91789

This API will go in a new directory (Include/semi-stable/) - see Nick's reply

For the directory name, I suggest to omit - or _ and just put the two words as semistable, it's easier to type.

C compilers don't handle well header files with the same names in different directories. For Include/cpython/, all header files fail with #error if they are included directly. IMO we can do the same here to ensure that Include/semistable/ is not added to the search paths by mistake.

Another option is to add a prefix to each file, like pycore_ for the internal C API. For example, Include/internal/pycore_object.h is related to Include/object.h. But that's more annoying and confusing.

I would prefer that C extensions just include the main <Python.h> and that's it (no change). I would prefer that C extensions don't include directly <semistable/frame.h> for example.

UPDATE: I completed my list, I forgot some removed functions.

Document the new opt-in macro

When building Python itself, does the Py_BUILD_CORE macro imply defining the Py_USING_SEMI_STABLE_API macro?

For me, the internal API (Py_BUILD_CORE) is a superset of semi-stable API (Py_USING_SEMI_STABLE_API).

APIs that provide access to compiled code whether in AST or opcode form (the API itself may be stable, but the compiled code isn't, so this is kinda covered by your last point)

In Python 3.10, Python-ast.h was moved to the internal C API as pycore_ast.h, and many API related to AST have been moved to the internal C API:

PyST_GetScope()
PySymtable_Build()
PySymtable_BuildObject()
PySymtable_Free()
Py_SymtableString()
Py_SymtableStringObject()
PyAST_Compile()
PyAST_CompileEx()
PyAST_CompileObject()
PyFuture_FromAST()
PyFuture_FromASTObject()
PyParser_ASTFromFile()
PyParser_ASTFromFileObject()
PyParser_ASTFromFilename()
PyParser_ASTFromString()
PyParser_ASTFromStringObject()
PyArena_New()
PyArena_Free()
PyArena_Malloc()
PyArena_AddPyObject()
see: https://docs.python.org/dev/whatsnew/3.10.html#id4

Are there remaining "AST related" functions in the C API?

Obviously, compile() can still be called with PyCF_ONLY_AST flag, or call ast.parse(). But this API returns Python objects, not C structures.

I would be nice to move the PyTypeObject to the semi-stable API, but IMO it's not possible to remove it from the default API because way too many C extensions use it. Modify all these extensions to define Py_USING_SEMI_STABLE_API is just annoying.

OK, I'll do semistable/, and I already have #error in my WIP.

When building Python itself, does the Py_BUILD_CORE macro imply defining the Py_USING_SEMI_STABLE_API macro?

Yes.

Are there remaining "AST related" functions in the C API?

@ncoghlan?

I would be nice to move the PyTypeObject to the semi-stable API

Definitely not in 3.11.

The old underscore-prefixed names can be deprecated static inline functions, rather than #defines as in the original plan.

I forgot (or simply didn't notice) that the AST APIs had been migrated to the internal API.

I still think they'd be good candidates for being declared semi-stable - part of the rationale for the new tier is reducing the temptation for 3rd parties to include the internal API headers, and these were considered useful enough to be public APIs prior to 3.10

Which projects use the C API of AST functions removed in Python 3.10? What is the advantage of adding back this removed API?

These functions were undocumented, excluded from the limited C API and had no test. The removal is related to the PEG parser PEP 617 which removed the old parser. The implementation modified the C code and in C, it was no longer possible to create struct _mod (mod_ty) which were arguments of the removed AST functions.

In Python 3.8, you needed the parser to get a node* object to then get a mod_ty object. But the whole C API of the parser has been removed in Python 3.10: the struct _node (node) type has been removed.

In Python 3.10 and 3.11, it's possible to browse AST using the Python API. Just use the Python ast module. IMO it's enough. There is no need to add a public C API for that.

The old underscore-prefixed names can be deprecated static inline functions, rather than #defines as in the original plan.

If there is no plan to deprecate the old names, macros are fine. Like the current code in Include/cpython/abstract.h:

// Backwards compatibility aliases for API that was provisional in Python 3.8
#define _PyObject_Vectorcall PyObject_Vectorcall
#define _PyObject_VectorcallMethod PyObject_VectorcallMethod
#define _PyObject_FastCallDict PyObject_VectorcallDict
#define _PyVectorcall_Function PyVectorcall_Function
#define _PyObject_CallOneArg PyObject_CallOneArg
#define _PyObject_CallMethodNoArgs PyObject_CallMethodNoArgs
#define _PyObject_CallMethodOneArg PyObject_CallMethodOneArg

By the way, IMO it's time to deprecate these aliases :-)

Then nice advantage of a static inline function is that it can be decorated with Py_DEPRECATED(3.11).

Haha, _PyInterpreterState_GetEvalFrameFunc and _PyInterpreterState_SetEvalFrameFunc, are not mentioned in PEP 523 at all, but their docs say See PEP 523...

I guess they fall under Functions added in PEP-523. Off with their underscores!

Haha, _PyInterpreterState_GetEvalFrameFunc and _PyInterpreterState_SetEvalFrameFunc, are not mentioned in PEP 523 at all, but their docs say See PEP 523...

These functions should be made public in the semi-stable API. The PyInterperterState structure was made opaque in Python 3.8 (released in 2019), PEP 523 was approved in 2016. I added these functions to keep PEP 523 usable. Otherwise, you had to include the internal C API to use the PEP, hum...

@vstinner I see the function is documented to take PyFrameObject: https://docs.python.org/3.10/c-api/init.html?highlight=pyframeevalfunction#c._PyFrameEvalFunction
But it actually takes a _PyInterpreterFrame. Should that be moved to semi-stable API too?

I found the commit that changed it and asked there: #88756 (comment)

If you ask me, I suggest to move these functions and type to the semi-stable API:

_PyFrameEvalFunction type
_PyInterpreterState_GetEvalFrameFunc()
_PyInterpreterState_SetEvalFrameFunc()
PyCode_New()
PyCode_NewWithPosOnlyArgs()

I would prefer to not expose _PyEval_EvalFrameDefault() directly. It should be the default value of _PyInterpreterState_GetEvalFrameFunc(). But if someone considers that it should be exposed, I'm also fine with that.

So, What's New was updated to say:

:c:func:_PyFrameEvalFunction now takes _PyInterpreterFrame* as its second parameter, instead of PyFrameObject*. See :pep:523 for more details of how to use this function pointer type.

But that's pretty confusing, PEP 523 does not mention _PyInterpreterFrame!

If I understand correctly, _PyInterpreterFrame needs to stay private, and _PyFrameEvalFunction needs to take it so _PyEval_EvalFrameDefault can stay fast.
But if users can access it, it needs to be exposed (as an opaque structure, of course).

Would it be appropriate to also put _PyFrame_GetFrameObject in the semi-stable API, and say use PyFrame_GetFrameObject to convert _PyInterpreterFrame to PyFrameObject?

You should ask @markshannon and users of these API.

Since NumPy uses it, does it make sense to move _Py_HashDouble into the semi stable API? We also seem to use _PyDict_GetItemStringWithError, although that is easy to just ship on our own (we do for older versions anyway). This is a quick search, I may be missing things. Or maybe just put them into the public API directly?

_PyDict_GetItemStringWithError is used in only a few places in CPython, frequently for static strings. IMO it would be better for CPython to reevaluate using it.
_Py_HashDouble looks OK for semi-stable API, but I wouldn't make it fully public -- its handling of NaN, which is the reason it takes an extra PyObject*, is a detail we might want to change.

I'm porting a debugger that uses a _PyFrameEvalFunction and ran into the problem of _PyInterpreterFrame being opaque (unless I use the internal headers). I think I can work around this, but for some purposes (including, I think, the motivating case for adding eval_frame of a jit compiler) access to the frame to be evaluated is needed. A variant of _PyFrame_GetFrameObject could be used or it may be possible to maintain the old signature and have a frame passed -- for performance, the core would need to bypass the eval_frame if it is set to the default

@jpe, What information do you need to get out of the frame? It may make more sense to offer APIs (stable or not) that give you the specific things you are looking for.

As of now, I need the global and locals. But for something like a jit compiler (I think eval_frame was added for pyjion), you'd need more. As I mentioned I can work around this, but I think that eval_frame currently isn't very useful and either should be deprecated / removed or made to work reasonably.

If your issue is that PEP 623 isn't the right API, I think you're preaching to the choir, but I'm not sure how much PEP 689 can help -- it isn't the place to design a new API.

My issue is I don't think the eval_frame code in 3.11 is very useful as it stands and think it should be either deprecated / removed or made as useful as it was in 3.10. I suspect if left as it is, it will either not be used or that internal api's will be need to be used.

Noted. Unfortunately our expert on that topic (@markshannon) is currently on vacation. I think it's too late for code changes in 3.11, so you are likely going to be using internal APIs (which might become "unstable but supported APIs" in 3.12, if PEP 689 is accepted).

I'm deferring the PEP for now.

Oh, too bad, I liked this PEP. But yeah, it's a lot of work :-(

PEP 689 -- Add an unstable C-API tier · Issue #91744 · python/cpython · GitHub

Repository navigation

PEP 689 -- Add an unstable C-API tier #91744

5 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

PEP 689 -- Add an unstable C-API tier · Issue #91744 · python/cpython · GitHub

Repository navigation

PEP 689 -- Add an unstable C-API tier #91744

Description

Activity

vstinner commented on Apr 20, 2022

vstinner commented on Apr 20, 2022

vstinner commented on Apr 20, 2022

encukou commented on Apr 20, 2022

encukou commented on Apr 20, 2022

ncoghlan commented on Apr 20, 2022

vstinner commented on Apr 20, 2022

vstinner commented on Apr 20, 2022

encukou commented on Apr 20, 2022

vstinner commented on Apr 20, 2022

encukou commented on Apr 21, 2022

encukou commented on Apr 21, 2022

vstinner commented on Apr 21, 2022

5 remaining items

encukou commented on Apr 22, 2022

vstinner commented on Apr 22, 2022

seberg commented on Apr 27, 2022

encukou commented on Apr 28, 2022

jpe commented on Jul 12, 2022

gvanrossum commented on Jul 12, 2022

jpe commented on Jul 12, 2022

gvanrossum commented on Jul 12, 2022

jpe commented on Jul 12, 2022

gvanrossum commented on Jul 12, 2022

encukou commented on Aug 10, 2022

vstinner commented on Aug 11, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions