FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-124375: Avoid calling `_PyMem_ProcessDelayed` on other thread states by colesbury · Pull Request #124459 · python/cpython · GitHub

/ cpython Public

gh-124375: Avoid calling _PyMem_ProcessDelayed on other thread states - #124459

Merged
colesbury merged 3 commits into
python:mainfrom
colesbury:gh-124375-py-mem-process-delayed
Oct 15, 2024
ConversationCommits (3)ChecksFiles changed
Merged

gh-124375: Avoid calling _PyMem_ProcessDelayed on other thread states#124459
colesbury merged 3 commits into
python:mainfrom
colesbury:gh-124375-py-mem-process-delayed

Conversation

colesbury commented Sep 24, 2024 •
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Contributor

This fixes a crash when running the PyO3 test suite on the free-threaded build. The qsbr field is initialized after the PyThreadState is added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is always initialized (possibly empty) and protected by the stop the world pause.

pythongh-124375: Avoid calling _PyMem_ProcessDelayed on other threa…
bc567fc 
…d states

This fixes a crash when running the PyO3 test suite on the free-threaded
build. The `qsbr` field is initialized after the `PyThreadState` is
added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is
always initialized (possibly empty) and protected by the stop the world
pause.
colesbury requested a review from DinoV September 24, 2024 18:55
bedevere-app Bot mentioned this pull request Sep 24, 2024
Closed

colesbury commented Sep 24, 2024

Copy link
Copy Markdown
Contributor Author

I spent a while trying to get a unit test to reproduce the issue without success. I think the period between when HEAD_UNLOCK is called and when qsbr is initialized is pretty short and hard to exploit in a test case:

cpython/Python/pystate.c

Lines 1589 to 1600 in 17b3bc9

HEAD_UNLOCK(runtime);
if (!used_newtstate) {
// Must be called with lock unlocked to avoid re-entrancy deadlock.
PyMem_RawFree(new_tstate);
}
else {
}
#ifdef Py_GIL_DISABLED
// Must be called with lock unlocked to avoid lock ordering deadlocks.
_Py_qsbr_register(tstate, interp, qsbr_idx);
#endif

colesbury added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Sep 24, 2024

bedevere-bot commented Sep 24, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @colesbury for commit bc567fc 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Sep 24, 2024

colesbury commented Sep 24, 2024

Copy link
Copy Markdown
Contributor Author

cc @ngoldbaum

ngoldbaum commented Sep 24, 2024

Copy link
Copy Markdown
Contributor

Thanks for this! I'll try to test this with PyO3 tomorrow morning.

ngoldbaum commented Sep 25, 2024

Copy link
Copy Markdown
Contributor

Thankfully it seems the 3.14 ABI hasn't yet diverged enough from 3.13 that it's painful to test this with PyO3. It looks like this does fix the segfault - I'm completely unable to trigger it on this PR branch and I can trigger it very quickly with the same test approaches (looping on cargo test --lib -- --test-threads=1000) on 3.13.0rc2 and the current main branch.

DinoV approved these changes Sep 26, 2024
View reviewed changes

DinoV left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM!

colesbury enabled auto-merge (squash) October 15, 2024 15:31
colesbury merged commit 54c6fcb into python:main Oct 15, 2024

miss-islington-app Bot commented Oct 15, 2024

Copy link
Copy Markdown

Thanks @colesbury for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 15, 2024
pythongh-124375: Avoid calling _PyMem_ProcessDelayed on other threa…
93e51c9 
…d states (pythonGH-124459)

This fixes a crash when running the PyO3 test suite on the free-threaded
build. The `qsbr` field is initialized after the `PyThreadState` is
added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is
always initialized (possibly empty) and protected by the stop the world
pause.
(cherry picked from commit 54c6fcb)

Co-authored-by: Sam Gross <colesbury@gmail.com>

bedevere-app Bot commented Oct 15, 2024

Copy link
Copy Markdown

GH-125540 is a backport of this pull request to the 3.13 branch.

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Oct 15, 2024
colesbury added a commit that referenced this pull request Oct 15, 2024
[3.13] gh-124375: Avoid calling _PyMem_ProcessDelayed on other thre…
4d83f6d 
…ad states (GH-124459) (#125540)

This fixes a crash when running the PyO3 test suite on the free-threaded
build. The `qsbr` field is initialized after the `PyThreadState` is
added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is
always initialized (possibly empty) and protected by the stop the world
pause.
(cherry picked from commit 54c6fcb)

Co-authored-by: Sam Gross <colesbury@gmail.com>
colesbury deleted the gh-124375-py-mem-process-delayed branch December 4, 2024 20:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

DinoV DinoV approved these changes

Assignees

No one assigned

Labels

skip news topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

Back | FazBrowse Home | New Git URL