FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-99113: A Per-Interpreter GIL! by ericsnowcurrently · Pull Request #104210 · python/cpython · GitHub

/ cpython Public

gh-99113: A Per-Interpreter GIL! - #104210

Merged
ericsnowcurrently merged 2 commits into
python:mainfrom
ericsnowcurrently:per-interpreter-gil
May 8, 2023
ConversationCommits (2)ChecksFiles changed
Merged

gh-99113: A Per-Interpreter GIL!#104210
ericsnowcurrently merged 2 commits into
python:mainfrom
ericsnowcurrently:per-interpreter-gil

Conversation

ericsnowcurrently commented May 5, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

This is the culmination of PEP 684 (and of my 8-year long multi-core Python project)!

Each subinterpreter may now be created with its own GIL (via Py_NewInterpreterFromConfig()). If not so configured then the interpreter will share with the main interpreter--the status quo since the subinterpreters were added decades ago. The main interpreter always has its own GIL and subinterpreters from Py_NewInterpreter() will always share with the main interpreter.

(FYI, I've split up the original gh-99114 into multiple PRs, with this one being the final one in the chain.)

bedevere-bot mentioned this pull request May 5, 2023
Closed
ericsnowcurrently mentioned this pull request May 5, 2023
Closed
3 tasks
ericsnowcurrently added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 6, 2023

bedevere-bot commented May 6, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 289d050 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 6, 2023
ericsnowcurrently mentioned this pull request May 6, 2023
Merged
ericsnowcurrently marked this pull request as ready for review May 6, 2023 22:03

ericsnowcurrently commented May 6, 2023

Copy link
Copy Markdown
Member Author

FYI, there are plenty of little things to clean up after this lands, particularly for PEP 684. There are also some corner-case crashes I need to chase down, related to extreme races.

Also, I still can't believe I've finally reached this point with this project!!! 😭

erlend-aasland approved these changes May 7, 2023
View reviewed changes

erlend-aasland left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM 🎉 BTW, I think the massive top-level comment in Python/ceval_gil.c needs an overhaul now.

kumaraditya303 approved these changes May 8, 2023
View reviewed changes

kumaraditya303 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM is short for the amount of effort you put into it.

ericsnowcurrently merged commit 5c9ee49 into python:main May 8, 2023

ericsnowcurrently commented May 8, 2023

Copy link
Copy Markdown
Member Author

Thanks for all the great work and support, everyone!

There are plenty of small things to wrap up, but this is definitely the climax.

carljm added a commit to carljm/cpython that referenced this pull request May 9, 2023
Merge branch 'main' into inlinecomp2
76077cd 
* main: (47 commits)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)
  pythongh-103193: Improve `getattr_static` test coverage (python#104286)
  Trim trailing whitespace and test on CI (python#104275)
  pythongh-102500: Remove mention of bytes shorthand (python#104281)
  pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)
  pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)
  pythongh-104273: Remove redundant len() calls in argparse function (python#104274)
  pythongh-64660: Don't hardcode Argument Clinic return converter result variable name (python#104200)
  pythongh-104265 Disallow instantiation of `_csv.Reader` and `_csv.Writer` (python#104266)
  pythonGH-102613: Improve performance of `pathlib.Path.rglob()` (pythonGH-104244)
  pythongh-103650: Fix perf maps address format (python#103651)
  pythonGH-89812: Churn `pathlib.Path` methods (pythonGH-104243)
  ...
adr26 mentioned this pull request May 9, 2023
Open
carljm added a commit to carljm/cpython that referenced this pull request May 9, 2023
Merge branch 'main' into fixsendfamily
a614216 
* main: (29 commits)
  pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)
  pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)
  pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)
  require-pr-label.yml: Add missing "permissions:" (python#104309)
  pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)
  pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)
  pythongh-103193: Improve `getattr_static` test coverage (python#104286)
  Trim trailing whitespace and test on CI (python#104275)
  pythongh-102500: Remove mention of bytes shorthand (python#104281)
  pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)
  pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)
  ...
carljm added a commit to carljm/cpython that referenced this pull request May 9, 2023
Merge branch 'main' into supermon
c48eee3 
* main: (156 commits)
  pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)
  pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)
  pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)
  pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)
  pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)
  pythongh-104240: return code unit metadata from codegen (python#104300)
  pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)
  pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)
  pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)
  require-pr-label.yml: Add missing "permissions:" (python#104309)
  pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)
  pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  ...
carljm added a commit to carljm/cpython that referenced this pull request May 9, 2023
Merge branch 'main' into fixloadsuperspec
cf11908 
* main: (35 commits)
  pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)
  pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)
  pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)
  pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)
  pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)
  pythongh-104240: return code unit metadata from codegen (python#104300)
  pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)
  pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)
  pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)
  require-pr-label.yml: Add missing "permissions:" (python#104309)
  pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)
  pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  ...
ericsnowcurrently mentioned this pull request May 9, 2023
Closed
This was referenced May 30, 2023
Open
Open
XuehaiPan mentioned this pull request Nov 12, 2023
Closed
1 task

aotto1968 commented Aug 29, 2024

Copy link
Copy Markdown

well , i changed my library to use this, but end up in ...

I started using Py_NewInterpreterFromConfig to add an PER-THREAD interpreter with my
library. The library is thread safe and supports many languages
-> C C++ JAVA C# VB.Net Perl Python GO TCL Ruby Php
other languages with thread support
-> C C++ JAVA C# TCL GO
works fine.
even python works fine without thread support.

Task: now I started thread support in python, I had to change to Multi-phase initialization
isolate the python data on an per-thread-level and finally gent an easy thread example working
-> ok

    static PyInterpreterConfig config = {
        .use_main_obmalloc = 0,
        .allow_fork = 0,
        .allow_exec = 0,
        .allow_threads = 1,
        .allow_daemon_threads = 0,
        .check_multi_interp_extensions = 1,
        .gil = PyInterpreterConfig_OWN_GIL,
    };

    PyThreadState *tstate = NULL;
    PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);

NOW I start with an more complex example and this stuff fails with a python INTERNAL core
that I can't debug. The stack-trace comes from my MqDisasterSignal handler.

-> question: what can I do to debug this issue?
-> debugging with valgrind is not possible because with valgrind this example works fine.


Debug memory block at address p=0x7f11884def70: API '^@'
    18302063728033398269 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0x00 *** OUCH
        at p-6: 0x00 *** OUCH
        at p-5: 0x00 *** OUCH
        at p-4: 0x00 *** OUCH
        at p-3: 0x00 *** OUCH
        at p-2: 0x00 *** OUCH
        at p-1: 0x00 *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=0xfdfe7d0f864bed6d are X> {PRINT               :pid(60831):tid(0x7f118819c700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1cb99b8     ):MqDisasterSignal              }: BackTrace {
:pid(60831):tid(0x7f118819c700):   [ library              : filename                       : lineno ] function
:pid(60831):tid(0x7f118819c700):   [ -------              : --------                       : ------ ] --------
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 705    ] MqDisasterSignal
:pid(60831):tid(0x7f118819c700):   [ unknown              : unknown                        : 0      ] unknown
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2408   ] _PyObject_DebugDumpAddress
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2326   ] _PyMem_DebugCheckAddress
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2159   ] _PyMem_DebugRawFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 685    ] PyMem_RawFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 1853   ] _PyObject_Free
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2163   ] _PyMem_DebugRawFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2296   ] _PyMem_DebugFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 830    ] PyObject_Free
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1569   ] dictresize
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1194   ] insertion_resize
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1261   ] insertdict
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1865   ] _PyDict_SetItem_Take2
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1883   ] PyDict_SetItem
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7618   ] add_subclass
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7352   ] type_ready_add_subclasses
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7515   ] type_ready
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7553   ] PyType_Ready
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3795   ] type_new_impl
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3929   ] type_new
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 1664   ] type_call
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 240    ] _PyObject_MakeTpCall
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3949   ] type_vectorcall
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 133    ] _PyObject_FastCallDictTstate
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 157    ] PyObject_VectorcallDict
:pid(60831):tid(0x7f118819c700):   [ system               : Python/bltinmodule.c           : 208    ] builtin___build_class__
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/methodobject.c         : 438    ] cfunction_vectorcall_FASTCALL_KEYWORDS
:pid(60831):tid(0x7f118819c700):   [ system               : Include/internal/pycore_call.h : 92     ] _PyObject_VectorcallTstate
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 325    ] PyObject_Vectorcall
:pid(60831):tid(0x7f118819c700):   [ system               : Python/bytecodes.c             : 2714   ] _PyEval_EvalFrameDefault
:pid(60831):tid(0x7f118819c700):   [ system               : nclude/internal/pycore_ceval.h : 89     ] _PyEval_EvalFrame
:pid(60831):tid(0x7f118819c700):   [ system               : Python/ceval.c                 : 1683   ] _PyEval_Vector
:pid(60831):tid(0x7f118819c700):   [ system               : Python/ceval.c                 : 578    ] PyEval_EvalCode
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1722   ] run_eval_code_obj
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1743   ] run_mod
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1643   ] pyrun_file
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 433    ] _PyRun_SimpleFileObject
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 466    ] PyRun_SimpleFileExFlags
:pid(60831):tid(0x7f118819c700):   [ theLink              : py/MqFactoryC_py.c             : 194    ] py_mqmsgque_sFactoryCTor
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/MqFactoryS_mq.c              : 289    ] MqFactoryInvoke_RT
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 269    ] MqSysServerThreadMain
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 324    ] sSysServerThreadInit
:pid(60831):tid(0x7f118819c700):   [ unknown              : unknown                        : 0      ] unknown
:pid(60831):tid(0x7f118819c700): }
:pid(60831):tid(0x7f118819c700): ^[[1;36mglobal data -----------------------------------------------------^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mruntime  : MkRuntimeRLS=0x1cb93b0, MqRuntimeRLS=0x7f11879539c8^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mgeDat    : geDat=0x7f1188499de0, geMax=16, geCur=2^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mevent    : event=0x7f1187954610, DataLNum=16, DataLCur=0, geNum=1, fdmax=0^[[0;m
.../NHI1/bin/Nhi1Exec: line 870: 60831 Aborted                 '.../x86_64-suse-linux-gnu/debug/bin/python3' '.../NHI1/example/py/server.py' '--thread' '--uds' '--file' '/tmp/test.uds'

ericsnowcurrently commented Jul 7, 2025

Copy link
Copy Markdown
Member Author

@aotto1968, sorry I didn't see this sooner. Your best option is to open a new issue. As to the described problem, we can talk about it on the new issue, but I'm guessing it has to do with using PyGILState_Ensure(), which still doesn't handle subinterpreters quite right in a couple of corner cases.

ericsnowcurrently commented Jul 7, 2025

Copy link
Copy Markdown
Member Author

Please CC me on the new issue.

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

erlend-aasland erlend-aasland approved these changes
kumaraditya303 kumaraditya303 approved these changes
markshannon Awaiting requested review from markshannon

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

Back | FazBrowse Home | New Git URL