gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw memory allocator by chgnrdv · Pull Request #109808 · python/cpython · GitHub

Fixes #109795

When new thread is starting, but Python is finalizing, PyMem_Free can't be called on new thread bootstate, because this thread doesn't own the GIL. We should use raw memory allocator to allocate/free bootstate to avoid crashes/other unexpected behaviour.

Issue: Starting new thread during finalization leads to call to PyMem_Free without holding the GIL #109795

…mory allocator When new thread is starting, but Python is finalizing, PyMem_Free can't be called on new thread bootstate, because this created thread doesn't own the GIL. We should use raw memory allocator to allocate/free bootstate to avoid crashes/other unexpected behaviour.

fixed news entry

Windows (x86) failed due to #109739.

…of github.com:chgnrdv/cpython into _thread-start-new-thread-use-raw-malloc-for-bootstate

Co-authored-by: Victor Stinner <vstinner@python.org>

…bootstate

Oh. Please remove this Changelog entry. I don't think that it's worth it. It doesn't impact directly users and it's a recent regression, no?

…e-109795.gI7KW6.rst

Thanks @chgnrdv for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

Sorry, @chgnrdv and @vstinner, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 1b8f2366b38c87b0450d9c15bdfdd4c4a2fc3a01 3.11

…e using raw memory allocator (python#109808) (cherry picked from commit 1b8f236)

GH-109852 is a backport of this pull request to the 3.11 branch.

I didn't backport PR gh-109135 yet, since I'm waiting PR gh-109133 to be merged. I will merge this fix at the same time than backporting gh-109135.

Merged, thanks!

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 RHEL8 LTO 3.x has failed when building commit 1b8f236.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/338/builds/5153) and take a look at the build logs.
Check if the failure is related to this commit (1b8f236) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/338/builds/5153

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1004, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1100, in get_code
  File "<frozen importlib._bootstrap_external>", line 1199, in get_data
TypeError: descriptor 'close' for '_io.BufferedReader' objects doesn't apply to a '_io.FileIO' object


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto/build/Lib/threading.py", line 1059, in _bootstrap_inner
    self.run()
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto/build/Lib/threading.py", line 996, in run
    self._target(*self._args, **self._kwargs)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto/build/Lib/test/test_interpreters.py", line 483, in task
    interp = interpreters.create()
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto/build/Lib/test/support/interpreters.py", line 25, in create
    id = _interpreters.create(isolated=isolated)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: interpreter creation failed
k

…te usin… (#109852) gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw memory allocator (#109808) (cherry picked from commit 1b8f236) Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com>

…e using raw memory allocator (python#109808)

…e using raw memory allocator (python#109808) (cherry picked from commit 1b8f236)

) (#110342) * gh-108987: Fix _thread.start_new_thread() race condition (#109135) Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time. Move _PyThreadState_CheckConsistency() from thread_run() to _PyThreadState_Bind(). (cherry picked from commit 517cd82) * gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw memory allocator (#109808) (cherry picked from commit 1b8f236) --------- Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com>

…e using raw memory allocator (python#109808)

bedevere-app Bot added the awaiting review label Sep 24, 2023

bedevere-app Bot mentioned this pull request Sep 24, 2023

Starting new thread during finalization leads to call to PyMem_Free without holding the GIL #109795

Closed

blurb-it Bot and others added 2 commits September 24, 2023 22:26

📜🤖 Added by blurb_it.

9665998

Update 2023-09-24-22-26-56.gh-issue-109795.gI7KW6.rst

9b32580

fixed news entry

chgnrdv changed the title gh-109795: _thread.start_new_thread: allocate/free thread bootstate using raw memory allocator gh-109795: _thread.start_new_thread: allocate thread bootstate using raw memory allocator Sep 24, 2023

AlexWaygood requested a review from vstinner September 24, 2023 23:10

vstinner reviewed Sep 25, 2023

View reviewed changes

Comment thread Modules/_threadmodule.c

chgnrdv added 2 commits September 25, 2023 12:43

added comment that refers the changes

e5c0f92

Merge branch '_thread-start-new-thread-use-raw-malloc-for-bootstate' …

b2f5e37

…of github.com:chgnrdv/cpython into _thread-start-new-thread-use-raw-malloc-for-bootstate

vstinner reviewed Sep 25, 2023

View reviewed changes

Comment thread Modules/_threadmodule.c Outdated

chgnrdv and others added 2 commits September 25, 2023 17:22

Update Modules/_threadmodule.c

0fdd943

Co-authored-by: Victor Stinner <vstinner@python.org>

Merge branch 'main' into _thread-start-new-thread-use-raw-malloc-for-…

7c776a7

…bootstate

vstinner reviewed Sep 25, 2023

View reviewed changes

Delete Misc/NEWS.d/next/Core and Builtins/2023-09-24-22-26-56.gh-issu…

1d9c95a

…e-109795.gI7KW6.rst

vstinner added the skip news label Sep 25, 2023

vstinner merged commit 1b8f236 into python:main Sep 25, 2023

bedevere-app Bot removed the awaiting review label Sep 25, 2023

vstinner added the needs backport to 3.11 only security fixes label Sep 25, 2023

miss-islington assigned vstinner Sep 25, 2023

vstinner pushed a commit to vstinner/cpython that referenced this pull request Sep 25, 2023

pythongh-109795 : _thread.start_new_thread : allocate thread bootstat…

75e36a4

…e using raw memory allocator (python#109808) (cherry picked from commit 1b8f236)

vstinner mentioned this pull request Sep 25, 2023

[3.11] gh-109795: _thread.start_new_thread: allocate thread bootstate usin… #109852

Merged

bedevere-app Bot removed the needs backport to 3.11 only security fixes label Sep 25, 2023

csm10495 pushed a commit to csm10495/cpython that referenced this pull request Sep 28, 2023

pythongh-109795 : _thread.start_new_thread : allocate thread bootstat…

f3d62b3

…e using raw memory allocator (python#109808)

vstinner pushed a commit to vstinner/cpython that referenced this pull request Oct 4, 2023

pythongh-109795 : _thread.start_new_thread : allocate thread bootstat…

b4da602

…e using raw memory allocator (python#109808) (cherry picked from commit 1b8f236)

Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024

pythongh-109795 : _thread.start_new_thread : allocate thread bootstat…

c709195

…e using raw memory allocator (python#109808)

gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw memory allocator by chgnrdv · Pull Request #109808 · python/cpython · GitHub

Repository navigation

gh-109795: _thread.start_new_thread: allocate thread bootstate using raw memory allocator - #109808

gh-109795: _thread.start_new_thread: allocate thread bootstate using raw memory allocator#109808
vstinner merged 8 commits into
python:mainfrom
chgnrdv:_thread-start-new-thread-use-raw-malloc-for-bootstate

chgnrdv commented Sep 24, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

Uh oh!

chgnrdv commented Sep 24, 2023

Uh oh!

Uh oh!

Uh oh!

vstinner Sep 25, 2023

Uh oh!

miss-islington commented Sep 25, 2023

Uh oh!

miss-islington commented Sep 25, 2023

Uh oh!

bedevere-app Bot commented Sep 25, 2023

Uh oh!

vstinner commented Sep 25, 2023

Uh oh!

vstinner commented Sep 25, 2023

Uh oh!

bedevere-bot commented Sep 25, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -0,0 +1 @@
		:func:`!_thread.start_new_thread` now uses raw memory allocator to allocate new thread bootstate. This helps to avoid crashes in case when new thread gets started at interpreter finalization.

gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw memory allocator by chgnrdv · Pull Request #109808 · python/cpython · GitHub

Repository navigation

gh-109795: _thread.start_new_thread: allocate thread bootstate using raw memory allocator - #109808

gh-109795: _thread.start_new_thread: allocate thread bootstate using raw memory allocator#109808vstinner merged 8 commits intopython:mainpython/cpython:mainfrom chgnrdv:_thread-start-new-thread-use-raw-malloc-for-bootstateCopy head branch name to clipboard

Conversation

chgnrdv commented Sep 24, 2023 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chgnrdv commented Sep 24, 2023

Uh oh!

Uh oh!

Uh oh!

vstinner Sep 25, 2023

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Sep 25, 2023

Uh oh!

miss-islington commented Sep 25, 2023

Uh oh!

bedevere-app Bot commented Sep 25, 2023

Uh oh!

vstinner commented Sep 25, 2023

Uh oh!

vstinner commented Sep 25, 2023

Uh oh!

bedevere-bot commented Sep 25, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-109795: _thread.start_new_thread: allocate thread bootstate using raw memory allocator#109808
vstinner merged 8 commits into
python:mainfrom
chgnrdv:_thread-start-new-thread-use-raw-malloc-for-bootstate

chgnrdv commented Sep 24, 2023 •
edited by bedevere-app Bot
Loading