gh-117398: datetime: Access C-API via `PyInterpreterState` by neonene · Pull Request #118357 · python/cpython · GitHub

An alternative PR to #117413, which also makes the datetime C-API available to the isolated subinterpreters.

The pointer to the PyDateTime_CAPI struct is stored in PyInterpreterState right after the PyCapsule_Import() call. This version does not change Modules/_testcapi/datetime.c for now. Also, the overhead should be even across interpreters. However, this extends the PyDateTime_CAPI struct.

cc @pganssle @vstinner @ericsnowcurrently @encukou @erlend-aasland

Issue: Isolate the _datetime extension module #117398

I would prefer to not leak such implementation detail at the ABI level, and so use a regular function rather than a static inline function. I would prefer that _pydatetime_capi is private.

How can I resolve non-Windows errors? I'm not sure how to use private struct members here.

Error:  _testcapi failed to import: /home/runner/work/cpython/cpython-builddir/
build/lib.linux-x86_64-3.13-pydebug/_testcapi.cpython-313d-x86_64-linux-gnu.so:
undefined symbol: _PyDateTimeAPI_Get

I added another capsule which points to the _PyDateTimeAPI_Import function in _datetimemodule.c. Any buildable alternative is welcome.

I have mixed this patch into the multi-phase init PR: #118337.

@encukou Is there any chance that users could test this or #117413 in the beta phase? If these approaches work, I think _detetime can be isolated in a regular way.

Such change is too late for beta1. It's better to wait for the beginning of the 3.14 dev cycle.

Such change is too late for beta1.

How do you think of the situation?

In my experience, such change is likely to cause regression and need more time to be stabilized. I prefer to wait Wednesday.

Otherwise, 3.13 should take specific workarounds for the crash during after the interned-strings cleanup.

A workaround is ready for review: #118531

Otherwise, 3.13 should take specific workarounds for the crash during after the interned-strings cleanup.

Which crash? This PR is related to issue gh-117398 which doesn't mention any crash.

FYI, I asked @Yhg1s if we could get an exception for landing this after beta 1: #103092 (comment). He gave us a similar exception for 3.12. Thomas might not do that this time, but at least I've asked.

FYI, I asked @Yhg1s if we could get an exception for landing this after beta 1: #103092 (comment). He gave us a similar exception for 3.12. Thomas might not do that this time, but at least I've asked.

This particular change is just one that adds a new capsule C API, which can then be used when in the future the datetime module switches to multi-phase init and better subinterpreter support, right? Given that it's of mild impact and it gives us an API that allows us to deprecate the old one and use multi-phase init for datetime, I'm okay with this going into beta 2.

It doesn't sound safe to me to announce that the _datetime extension is compatible with subinterpreters before static types are converted to heap types. I suggest to discuss the plan in the issue: #117398 (comment)

@vstinner I think this should be tested from 3.13.0b1+. The change is easy to diagnose or revert. If an inevitable problem is found, the case will support your exceptional approach (mix _datetime types into PyInterpreterState) whose rationale is currently not proposed.

EDIT: Ultimately, PyInterpreterState does not need to be used, as the alternative PR proves.

IMO Py_MOD_PER_INTERPRETER_GIL_SUPPORTED cannot be added before other changes are done, such as converting static types to heap types.

I prefer to do the conversion in a different order, with multiple steps: #117398 (comment)

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

IMO Py_MOD_PER_INTERPRETER_GIL_SUPPORTED cannot be added before other changes are done, such as converting static types to heap types.

This PR does not add the flag in _datetimemodule.c. The test case in _testmultiphase.c is a kind of _zoneinfo multi-phase init module with the flag.

set capi-struct pointer to interp state

4c1325e

neonene requested review from abalkin, ericsnowcurrently, gvanrossum, markshannon and pganssle as code owners April 27, 2024 23:21

bedevere-app Bot mentioned this pull request Apr 27, 2024

Isolate the _datetime extension module #117398

Closed

bedevere-app Bot added the awaiting review label Apr 27, 2024

neonene added 7 commits April 28, 2024 08:34

globals-to-fix.tsv

fecd192

make exit code individual

2bdf01c

fix gil-disabled test

bead431

add an assertion in datetime_destructor()

d361ee4

fix assert()

3b721bc

add a struct for internal use

53b0219

Merge branch 'main' into capsule2

7333d39

neonene marked this pull request as draft April 29, 2024 23:09

bedevere-app Bot removed the awaiting review label Apr 29, 2024

This comment was marked as resolved.

Sign in to view

vstinner reviewed Apr 30, 2024

View reviewed changes

try hiding internal info

8b10d70

vstinner reviewed Apr 30, 2024

View reviewed changes

Comment thread Modules/_testmultiphase.c Outdated

Comment thread Modules/_testmultiphase.c Outdated

Comment thread Modules/_testmultiphase.c Outdated

Comment thread Modules/_testmultiphase.c Outdated

neonene added 4 commits May 1, 2024 04:40

apply suggested changes

fb0f5ef

ignored.tsv

7de45ac

Merge branch 'main' into capsule2

98564cf

globals-to-fix.tsv

f412c55

neonene marked this pull request as ready for review April 30, 2024 20:58

bedevere-app Bot added the awaiting review label Apr 30, 2024

neonene added 3 commits May 1, 2024 06:26

globals-to-fix.tsv

c00ed1f

ignored.tsv again

e4986f5

run main interp twice in test

39e17c7

neonene added 2 commits May 1, 2024 07:08

add a comment to a test

7aa369e

add an internal capsule

0769c96

neonene added 2 commits May 1, 2024 19:19

remove typedef

de31431

Merge branch 'main' into capsule2

c814186

neonene mentioned this pull request May 1, 2024

gh-117398: datetime: Make use of C-API capsules for sub-interpreters #117413

Closed

neonene mentioned this pull request May 3, 2024

Isolate Stdlib Extension Modules #103092

Closed

25 tasks

Merge branch 'main' into capsule2

92d4205

neonene added 2 commits May 6, 2024 00:54

add Py_MOD_GIL_NOT_USED to the test

8dadc49

Merge branch 'main' into capsule2

6527a3b

Merge branch 'main' into capsule2

ca89e9b

vstinner requested changes May 9, 2024

View reviewed changes

bedevere-app Bot added awaiting changes and removed awaiting review labels May 9, 2024

neonene mentioned this pull request May 10, 2024

datetime extension module isolation python/steering-council#243

Closed

neonene closed this May 23, 2024

neonene deleted the capsule2 branch May 23, 2024 14:04

+              static PyDateTime_CAPI *(*_get_pydatetime_capi)(void) = _get_pydatetime_capi_dummy;
+              #define PyDateTimeAPI _get_pydatetime_capi()
+              static inline void pydatetime_import(void) {

gh-117398: datetime: Access C-API via `PyInterpreterState` by neonene · Pull Request #118357 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-117398: datetime: Access C-API via PyInterpreterState - #118357

gh-117398: datetime: Access C-API via PyInterpreterState#118357neonene wants to merge 24 commits intopython:mainpython/cpython:mainfrom neonene:capsule2Copy head branch name to clipboard

Conversation

neonene commented Apr 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

vstinner Apr 30, 2024

Choose a reason for hiding this comment

Uh oh!

neonene Apr 30, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

neonene commented May 1, 2024

Uh oh!

neonene commented May 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented May 1, 2024

Uh oh!

neonene commented May 1, 2024

Uh oh!

vstinner commented May 2, 2024

Uh oh!

neonene commented May 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neonene commented May 3, 2024

Uh oh!

vstinner commented May 3, 2024

Uh oh!

ericsnowcurrently commented May 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Yhg1s commented May 5, 2024

Uh oh!

vstinner commented May 5, 2024

Uh oh!

neonene commented May 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-app Bot commented May 9, 2024

Uh oh!

neonene commented May 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-117398: datetime: Access C-API via PyInterpreterState#118357
neonene wants to merge 24 commits into
python:mainfrom
neonene:capsule2

neonene commented Apr 27, 2024 •
edited
Loading

neonene commented May 1, 2024 •
edited
Loading

neonene commented May 2, 2024 •
edited
Loading

ericsnowcurrently commented May 3, 2024 •
edited
Loading

neonene commented May 8, 2024 •
edited
Loading