FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-133171: Make the executor_list thread-safe by aisk · Pull Request #140038 · python/cpython · GitHub

  1. python
  2. cpython

Repository navigation

gh-133171: Make the executor_list thread-safe - #140038

Closed
aisk wants to merge 7 commits into
python:mainfrom
aisk:executor-list-lock
ConversationCommits (7)ChecksFiles changed
Closed

gh-133171: Make the executor_list thread-safe#140038
aisk wants to merge 7 commits into
python:mainfrom
aisk:executor-list-lock

Conversation

aisk commented Oct 13, 2025 •
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Member

The idea is simple and naive: just add a PyMutex to the PyInterpreterState for the executor_list under the nogil build, and lock or unlock the mutex in all functions that operate on the executor_list.

Since I'm not very familiar with the related code, there might be a better way to do this, so any feedback would be appreciated!

bedevere-app Bot mentioned this pull request Oct 13, 2025
Open
1 of 7 tasks
aisk added the skip news label Oct 13, 2025
aisk changed the title gh-133171: Add lock on operations to executor_list gh-133171: Make the executor_list thread-safe Oct 13, 2025
ZeroIntensity reviewed Oct 14, 2025
View reviewed changes

ZeroIntensity left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I'm not entirely sure that a mutex is safe here, because we're locking code paths that call Py_DECREF. @Fidget-Spinner, can these paths escape?

Comment thread Include/internal/pycore_interp_structs.h

Fidget-Spinner commented Oct 14, 2025

Copy link
Copy Markdown
Member

In theory, Py_DECREF(executor) could clear the (future) constants pool that we plan to add, which would hold arbitrary constant objects, which can escape.

corona10 commented Oct 15, 2025 •
edited
Loading

Copy link
Copy Markdown
Member

I am not sure if making executor_list operations thread-safe at this point is good idea.
There is no way to verify that we make the thread-safe until we actually integrate the whole pipeline. I do believe that we have to do more things before we start to work on this task.

Fidget-Spinner commented Nov 7, 2025

Copy link
Copy Markdown
Member

@corona10 we need the executor list to be thread safe for it to work with the plan here #133171 (comment).

@aisk could you resume the PR please?

Fidget-Spinner reviewed Nov 7, 2025
View reviewed changes
Comment thread Python/optimizer.c
void
_Py_Executors_InvalidateAll(PyInterpreterState *interp, int is_invalidation)
{
EXECUTOR_LIST_LOCK(interp);

Fidget-Spinner Nov 7, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This should be a stop the world event.

Comment thread Python/optimizer.c
}
/* Clearing an executor can deallocate others, so we need to make a list of
* executors to invalidate first */
EXECUTOR_LIST_LOCK(interp);

Fidget-Spinner Nov 7, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This is fine. nice.!

Comment thread Python/optimizer.c

corona10 commented Nov 7, 2025

Copy link
Copy Markdown
Member

I will try to re review this PR in this weekend.

corona10 commented Nov 7, 2025

Copy link
Copy Markdown
Member

Share the DM context between @Fidget-Spinner and I.

Before merging this PR, we should investigate the current blocker #133171 (comment) to verify this PR in practice.

aisk commented Nov 9, 2025

Copy link
Copy Markdown
Member Author

Hi @ZeroIntensity @corona10 @Fidget-Spinner , thanks for the review and feedback. I'll bring this PR up to date while waiting for #133171 (comment) to be resolved.

Fidget-Spinner reviewed Nov 9, 2025
View reviewed changes
Comment thread Python/optimizer.c
Comment on lines +303 to +307
/* Release the lock before calling _Py_ClearExecutorDeletionList
* to avoid deadlock, since it also tries to acquire the same lock */
EXECUTOR_LIST_UNLOCK(interp);
_Py_ClearExecutorDeletionList(interp);
EXECUTOR_LIST_LOCK(interp);

Fidget-Spinner Nov 9, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Do we still need this?

Fidget-Spinner commented Nov 10, 2025

Copy link
Copy Markdown
Member

Sorry, now that I think about it, locking isn't the right choice here. We should instead move the executor list to tstate.

Really sorry for taking up your time!

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Fidget-Spinner Fidget-Spinner left review comments
ZeroIntensity ZeroIntensity left review comments
ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
markshannon Awaiting requested review from markshannon markshannon is a code owner

Assignees

No one assigned

Labels

awaiting review skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

Back | FazBrowse Home | New Git URL