gh-116322: Add Py_mod_gil module slot by swtaarrs · Pull Request #116882 · python/cpython · GitHub

This PR adds the ability to enable the GIL if it was disabled at interpreter startup, and modifies the multi-phase module initialization path to enable the GIL when loading a module, unless that module's spec includes a slot indicating it can run safely without the GIL.

PEP 703 called the constant for the slot Py_mod_gil_not_used; I went with Py_MOD_GIL_NOT_USED for consistency with gh-104148.

A warning will be issued up to once per interpreter for the first GIL-using module that is loaded. If -v is given, a shorter message will be printed to stderr every time a GIL-using module is loaded (including the first one that issues a warning).

Issue: Allow C extensions to declare compatibility with free-threading #116322

📚 Documentation preview 📚: https://cpython-previews--116882.org.readthedocs.build/

The code here is ready, but I'll keep it as a draft until gh-116749 and gh-116329 are done, since this doesn't do anything when the GIL is enabled by default. I changed the default locally to test this as it works as intended.

I think it does make sense to land this now after all. The code won't function as intended until the GIL is disabled by default, but it will be nice to have the module slot available while I work on gh-116738, so I can tag modules as they're ready.

This is structured such that there shouldn't be any behavioral differences until the GIL is disabled by default, except when a --disable-gil build is run with -v, when it will print a single line for every non-free-threading-safe module that's loaded.

I'll take a look on Monday.

mostly LGTM

I've left a few comments and I have some feedback on the relationship between Py_mod_multiple_interpreters and Py_mod_gil.

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@ericsnowcurrently I don't think there should be any code changes as a result of our big sub-thread. Please let me know if that's wrong!

Before this is ready for another review, I need to make some other changes. It looks like the consensus in Discord is that we should enable the GIL when calling a module init function, because we don't know before the call if it's a single-phase init module that needs the GIL. Then if it's a multi-phase init module that provides Py_GIL_NOT_USED, we can disable the GIL and continue, otherwise we leave the GIL on for the rest of the interpreter's lifetime. I'll make an issue soon to nail down how we handle single-phase init modules.

The only nontrivial (but not too complicated) part of this will be adding a function to disable the GIL safely. It should also move the GIL-related logic up out of PyModule_FromDefAndSpec2(), which will be a nice side-effect.

I don't think there should be any code changes as a result of our big sub-thread.

Yeah, I think you're right.

More details: - Fix a race while enabling the GIL by checking if the GIL was enabled between a no-op call to `_PyEval_AcquireLock()` and the thread attaching, and trying again if it was. - Enable the GIL before running a module init function, since we can't know if it's a single-phase init module that doesn't support free-threading. Look at the state of the module after initialization to determine if it's safe to disable the GIL again. - Add `PyModule_SetGIL()`, which can be used by single-phase init modules to declare that they support running without the GIL. - Change `gil->enabled` from a simple on/off switch to a count of active requests to enable the GIL. This allows us to support multiple interleaved imports that each independently track whether the GIL should remain enabled. See the big comment in `pycore_ceval.h` for more details.

I have made the requested changes; please review again.

This should be ready for review again. It grew in scope a bit, and now includes support for single-phase init modules (gh-117526) (some details are in the commit message). With the infrastructure needed to properly support multi-phase init modules, supporting single-phase init modules is just a few extra lines, but I can pull it out into a separate PR if anyone wants.

One specific thing I'd like input on is the timing of the _PyImport_CheckGILForModule() call in PyModule_FromDefAndSpec2(): I put it before module creation to disable the GIL as soon as possible for modules that don't need the GIL. The downside of this is that if the module does need the GIL and a subsequent part of the import process fails, the GIL will remain enabled even though we didn't actually load the module. Maybe this should be rare enough that it's not worth worrying about.

!buildbot nogil

🤖 New build scheduled with the buildbot fleet by @colesbury for commit 77d1652 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

x86-64 MacOS Intel ASAN NoGIL PR
AMD64 Ubuntu NoGIL Refleaks PR
ARM64 MacOS M1 NoGIL PR
AMD64 Windows Server 2022 NoGIL PR
AMD64 Ubuntu NoGIL PR
ARM64 MacOS M1 Refleaks NoGIL PR
x86-64 MacOS Intel NoGIL PR

I found a few more modules that I missed during some testing, and will push a commit to add those shortly.

!buildbot nogil

🤖 New build scheduled with the buildbot fleet by @swtaarrs for commit d1fe0cc 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

x86-64 MacOS Intel ASAN NoGIL PR
AMD64 Ubuntu NoGIL Refleaks PR
ARM64 MacOS M1 NoGIL PR
AMD64 Windows Server 2022 NoGIL PR
AMD64 Ubuntu NoGIL PR
ARM64 MacOS M1 Refleaks NoGIL PR
x86-64 MacOS Intel NoGIL PR

Ehm, can this please be renamed to PyUnstable as suggested before? We have such naming schemes for a reason.

Sorry, I'll get a PR ready. I went with this name because having both Unstable and Experimental in the name felt redundant, and in this comment, @ericsnowcurrently expressed a preference for PyModule_ExperimentalSetGIL() over PyUnstable_Module_SetGIL().

Does anyone else have strong feelings on whether it should be PyUnstable_Module_ExperimentalSetGIL() or PyUnstable_Module_SetGIL()?

PR here: #118645

IMO, the “experimental” is redundant. If we need “Experimental” too, we should rethink the entire PEP-689 naming convention.

This PR adds the ability to enable the GIL if it was disabled at interpreter startup, and modifies the multi-phase module initialization path to enable the GIL when loading a module, unless that module's spec includes a slot indicating it can run safely without the GIL. PEP 703 called the constant for the slot `Py_mod_gil_not_used`; I went with `Py_MOD_GIL_NOT_USED` for consistency with pythongh-104148. A warning will be issued up to once per interpreter for the first GIL-using module that is loaded. If `-v` is given, a shorter message will be printed to stderr every time a GIL-using module is loaded (including the first one that issues a warning).

Should be Py_GIL_DISABLED.

This means: "Do not re-enable the GIL when importing rl.readline." See python/cpython#116322 See python/cpython#116882

Add Py_mod_gil module slot

fa92cb4

swtaarrs added the topic-free-threading label Mar 15, 2024

bedevere-app Bot mentioned this pull request Mar 15, 2024

Allow C extensions to declare compatibility with free-threading #116322

Closed

Merge remote-tracking branch 'upstream/main' into cpython-mod-gil

7edf796

swtaarrs marked this pull request as ready for review March 22, 2024 21:54

swtaarrs requested a review from ericsnowcurrently as a code owner March 22, 2024 21:54

bedevere-app Bot added the awaiting review label Mar 22, 2024

swtaarrs requested a review from colesbury March 22, 2024 21:54

ericsnowcurrently requested changes Mar 25, 2024

View reviewed changes

bedevere-app Bot removed the awaiting review label Mar 25, 2024

bedevere-app Bot added the awaiting changes label Mar 25, 2024

swtaarrs added 2 commits April 3, 2024 11:53

Some review comments from Eric

450aa41

Merge remote-tracking branch 'upstream/main' into cpython-mod-gil

6bbd281

swtaarrs mentioned this pull request Apr 3, 2024

Add a mechanism for single-phase init modules to declare free-threading compatibility #117526

Closed

swtaarrs commented Apr 4, 2024

View reviewed changes

Comment thread Python/ceval_gil.c Outdated

swtaarrs added 2 commits April 15, 2024 15:05

Merge remote-tracking branch 'upstream/main' into cpython-mod-gil

f7c3e50

swtaarrs requested review from brettcannon, kumaraditya303, ncoghlan and warsaw as code owners April 15, 2024 22:11

bedevere-app Bot added awaiting change review and removed awaiting changes labels Apr 15, 2024

AlexWaygood removed request for AlexWaygood, FFY00, erlend-aasland, kumaraditya303 and pablogsal May 1, 2024 23:08

swtaarrs added 3 commits May 1, 2024 16:10

Fix 'gil_slot set but not used' warning

99bc5cb

Patch generator for Python-ast.c instead of just the file itself

6882c13

Update limited API version for _scproxy.c

da63df1

swtaarrs requested a review from isidentical as a code owner May 1, 2024 23:17

swtaarrs added 2 commits May 1, 2024 16:22

Fix _testconsole.c for Windows build

2998def

Fix winsound.c for Windows build

77d1652

colesbury approved these changes May 2, 2024

View reviewed changes

Mark more modules as not using the GIL

d1fe0cc

colesbury merged commit c2627d6 into python:main May 3, 2024

bedevere-app Bot removed the awaiting merge label May 3, 2024

swtaarrs mentioned this pull request May 3, 2024

gh-116322: Enable the GIL while loading C extension modules #118560

Merged

Yhg1s reviewed May 5, 2024

View reviewed changes

mikeshardmind mentioned this pull request May 11, 2024

I think this is all that's needed AbstractUmbra/audioop#7

Merged

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this pull request Jul 25, 2024

pythongh-116882 : Fix typo in the #ifdef check

30a9ee1

serhiy-storchaka reviewed Jul 25, 2024

View reviewed changes

colesbury added a commit to colesbury/cpython that referenced this pull request Jul 25, 2024

pythongh-116882 : Fix typo in moduleobject.c

ae70061

stefanholek added a commit to stefanholek/rl that referenced this pull request Oct 14, 2024

Mark readline module in no-gil builds.

65457c6

This means: "Do not re-enable the GIL when importing rl.readline." See python/cpython#116322 See python/cpython#116882

cpcloud mentioned this pull request Oct 8, 2025

build: enable free-threading compatibility NVIDIA/cuda-python#1103

Merged

encukou mentioned this pull request Dec 16, 2025

gh-140550: Update xxlimited with 3.15 limited API & PEP 697 #142827

Merged


		.. versionadded:: 3.9

		.. c:function:: int PyModule_ExperimentalSetGIL(PyObject module, void gil)

                       }
                   }
                   m->md_def = module;
+              #ifdef Py_GIL_DISABLE

gh-116322: Add Py_mod_gil module slot by swtaarrs · Pull Request #116882 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-116322: Add Py_mod_gil module slot - #116882

gh-116322: Add Py_mod_gil module slot#116882colesbury merged 24 commits intopython:mainpython/cpython:mainfrom swtaarrs:cpython-mod-gilswtaarrs/cpython:cpython-mod-gilCopy head branch name to clipboard

Conversation

swtaarrs commented Mar 15, 2024 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swtaarrs commented Mar 15, 2024

Uh oh!

swtaarrs commented Mar 22, 2024

Uh oh!

ericsnowcurrently commented Mar 23, 2024

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-app Bot commented Mar 25, 2024

Uh oh!

swtaarrs commented Apr 3, 2024

Uh oh!

Uh oh!

ericsnowcurrently commented Apr 5, 2024

Uh oh!

swtaarrs commented Apr 15, 2024

Uh oh!

colesbury commented May 2, 2024

Uh oh!

bedevere-bot commented May 2, 2024

Uh oh!

swtaarrs commented May 3, 2024

Uh oh!

swtaarrs commented May 3, 2024

Uh oh!

bedevere-bot commented May 3, 2024

Uh oh!

Yhg1s May 5, 2024

Choose a reason for hiding this comment

Uh oh!

swtaarrs May 6, 2024

Choose a reason for hiding this comment

Uh oh!

encukou May 6, 2024

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Jul 25, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

gh-116322: Add Py_mod_gil module slot#116882
colesbury merged 24 commits into
python:mainfrom
swtaarrs:cpython-mod-gil

swtaarrs commented Mar 15, 2024 •
edited by github-actions Bot
Loading