Repository navigation
Expose _PyInterpreterFrame_GetLine in the private API #96803
Description
In 3.11 we are using now an opaque pointer to _PyInterpreterFrame in the eval function. The problem is that for PEP 523's APIs, we changed the type of _PyFrameEvalFunction to be PyObject *(*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag) but there is no way to do anything with _PyInterpreterFrame because is opaque. Even if the extension defines PY_BUILD_CORE the symbols are not exposed in the executable/libpython, which means that including the headers is insufficient.
This is a problem because it really limits what users of PEP 523 can do with the frame object that they receive, even if is opaque. Most profilers using PEP 523 API do it for speed reasons and the only thing they need is to get the line number, but that is now impossible because the pointer is opaque and we don't offer any exposed APIs.
Linked PRs
- GH-96803: Add three C-API functions to make _PyInterpreterFrame less opaque for users of PEP 523. #96849
- GH-96803: Document and test new unstable internal frame API functions #104211
- GH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h #107188
- [3.12] GH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h (GH-107188) #107195
Activity
pablogsal commented on Sep 13, 2022
CC: @markshannon @brandtbucher @iritkatriel @ericsnowcurrently
ericsnowcurrently commented on Sep 13, 2022
CC @mdboom
markshannon commented on Sep 14, 2022
I see your point, but I don't think that _PyInterpreterFrame_GetLine() is the right API.
I think the pair of _PyInterpreterFrame_GetCode() and _PyInterpreterFrame_GetLasti() would be better.
Getting the offset is much cheaper than computing the line number.
The line number can be computed, if needed.
Other information to be extracted from the code object, such as the filename and function name.
pablogsal commented on Sep 15, 2022
I started exposing _PyInterpreterFrame_GetCode and _PyInterpreterFrame_GetLasti but then I realized that neither _PyInterpreterFrame_GetCode and _PyInterpreterFrame_GetLasti are in 3.12 so exposing those is going to be more chaotic than just exposing _PyInterpreterFrame_GetLine. Notice that if the application defines PY_BUILD_CORE it can already access f_code and f_lasti manually from the struct.
As there can be arbitrary logic in _PyInterpreterFrame_GetLine I still think is a better idea to export _PyInterpreterFrame_GetLine.
pythongh-96803: Export _PyInterpreterFrame_GetLine as a private API s…
markshannon commented on Sep 15, 2022
It still think that the pair of _PyInterpreterFrame_GetCode and _PyInterpreterFrame_GetLasti is a better interface.
If you think that doing that would be too disruptive for 3.11, we can add them for 3.12, and just expose _PyInterpreterFrame_GetLine for 3.11.
pablogsal commented on Sep 15, 2022
In 3.12 they are already exposed if the user calls PY_BUILD_CORE because one is a macro and the other can be basically substituted by frame->f_code because the struct definition is available if PY_BUILD_CORE is defined. The problem here is having to re-implement _PyInterpreterFrame_GetLine.
Exposing _PyInterpreterFrame_GetCode and _PyInterpreterFrame_GetLasti doesn't give the user anything that cannot be achieved by just calling frame->f_code and frame->f_lasti after defining PY_BUILD_CORE
markshannon commented on Sep 15, 2022
_PyInterpreterFrame_GetLine is quite simple once we have the other two functions. There is no harm in exposing it as well.
All that does is expose the existence of a field call f_code. It doesn't say anything about its meaning or when it is valid.
In general, frame->f_code cannot be substituted for _PyInterpreterFrame_GetCode() as the meaning of frame->f_code is not defined. E.g. we might make it such that f_code is only valid for some frames faster-cpython/ideas#111
pablogsal commented on Sep 15, 2022
Ok, I am convinced. I will expose _PyInterpreterFrame_GetCode and _PyInterpreterFrame_GetLasti
pythongh-96803: Export _PyInterpreterFrame_GetLine as a private API s…
3 remaining items
markshannon commented on Sep 15, 2022
Doesn't that undermine the whole point as using the _PyInterpreterFrame in the first place, i.e. efficiency?
Also, incomplete _PyInterpreterFrames have a code object and lasti, but we cannot safely produce a frame object for them
GH-96803: Add three C-API functions to make _PyInterpreterFrame less …
erlend-aasland commented on May 5, 2023
gh-96849 added new APIs, but no documentation was added. Mentioning it here, so we don't forget before we close this issue.
Merge branch 'main' into inlinecomp2
pythonGH-96803: Add three C-API functions to make _PyInterpreterFrame…
encukou commented on May 9, 2023
Note that even unstable API needs tests as well as documentation.
GH-96803: Document and test new unstable internal frame API functions (…
Merge branch 'main' into nestedcomps
pythonGH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h
pythonGH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h
GH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h (#10…
pythonGH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h (
[3.12] GH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python…
pythonGH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h (