Repository navigation
multi-threading + fork warning when threads are stopped before fork #137109
Description
Bug report
Bug description:
If all threads are stopped before a fork, and then restarted in the parent (and optionally the child) process, the new multi-threading with fork warning is given
import os
import threading
stop = False
def target():
while not stop:
pass
thread = threading.Thread(target=target)
def stop_thread():
global stop
stop = True
thread.join()
def restart_thread():
global thread, stop
stop = False
(thread := threading.Thread(target=target)).start()
os.register_at_fork(before=stop_thread, after_in_child=restart_thread, after_in_parent=restart_thread)
thread.start()
if os.fork() == 0:
os._exit(0)If the thread is not restarted in the parent, no warning is given. I believe warn_about_fork_with_threads should be refactored so that a warning is given to the user only if there are threads detected after the call to PyOS_BeforeFork. If it is not possible to give a warning there, as stated in the comments, perhaps one can just do the thread count check there and take note of the result, then warn after PyOS_AfterFork_Parent, when safe to do so.
CPython versions tested on:
3.13
Operating systems tested on:
macOS
Activity
picnixz commented on Jul 25, 2025
Let's remove the callback in the child. The order of execution would be:
I think the problem is with the fact that thread from the parent process is not entirely joined. And so it's effectively the case that we're still multi-threaded even though it's not the case maybe.
cc @ZeroIntensity as someone who has better threading knowledge.
P403n1x87 commented on Jul 25, 2025
It looks to me that warn_about_fork_with_threads is warning on the new thread in the parent, because it is restarted by PyOS_AfterFork_Parent, before warn_about_fork_with_threads is called. So the warning is about the new thread running in the parent after the fork, rather than the original thread (that has actually been stopped and joined before the call to the fork syscall).
colesbury commented on Jul 28, 2025
Yeah, it'd be better to call the after-fork-parent calls after we check and warn about threads. That would avoid the spurious warning.
gpshead commented on Nov 12, 2025
Code paths that warn_about_fork_with_thread can take depend on PyOS_AfterFork_Parent() having been called already to avoid deadlock.
gpshead commented on Nov 12, 2025
A workaround for this could be for it to be called by PyOS_AfterFork_Parent() itself right before the run_at_forkers(...) call.
gpshead commented on Nov 12, 2025
But that'd still leave deadlocks possible as the Python world it may need to call into could rely on other afterfork parent calls that have been registered.
gpshead commented on Nov 12, 2025
Okay I think I have a solution for the most common case. The warn function can be split up into the OS specific APIs to get the number of threads (which can be called before AfterFork_Parent) and the Python APIs that must wait until after. This won't fix it in situations where to lack platform specific thread detection code or when those APIs fail, but will in the most widely used scenarios. PR coming after testing.
pythongh-137109: refactor warning about threads when forking
gpshead commented on Nov 12, 2025
re: race conditions, it may be OS specific as to if the thread could still show up in OS APIs as existing in the process after the internal C pthread_join() that the Python thread.join() did has returned. I'd like to think not, but that is up to the OS as there are no posix APIs to get a list of threads for a process so it is undefined behavior and up to the kernel implementation as to if it has finished tearing down that state and cleaning it up synchronously or not.
gh-137109: refactor warning about threads when forking (#141438)
7 remaining items
[3.14] pythongh-137109: refactor warning about threads when forking (p…
[3.13] gh-137109: refactor warning about threads when forking (GH-141438
gpshead commented on Nov 24, 2025
The PRs merged should've fixed this for the most common case.
pythongh-137109: refactor warning about threads when forking (python#…
fixed the last three ci found: 3.13 retainers, the fork warning, windows
asked the interpreter about the fork warning in the run being judged
stopped a race in cpython's own bookkeeping failing the fork parity
DEBUG: bracket gen_log.info()/os.fork() and add gc.collect()+sleep probe
DEBUG: measure OS-level thread count, not just PyThreadStates
test: ignore the spurious fork() multi-threading warning on macOS
test: ignore the spurious fork() multi-threading warning on x86_64 macOS
fixed the last three ci found: 3.13 retainers, the fork warning, windows
asked the interpreter about the fork warning in the run being judged
stopped a race in cpython's own bookkeeping failing the fork parity