All commit authors signed the Contributor License Agreement.

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

It might be a good idea to open a PR with just the fix for forkserver under #140867. That issue describes the problem with forkserver better, and the fix for forkserver is much simpler and safer than the fix for fork/spawn. It makes sense to treat them separately.

@zmedico if you want to do that, please go ahead, and feel free to copy the unit test from this PR if it is helpful 🙂

This fix looks good to me.

Tested this patch on Gentoo, on python-3.14.2, applies cleanly and fixes the errors in portage with egencache.

Agreed this looks good, any questions i had seem to have already been raised, described in the descriptions or discussed. I'm always cautious of introducing new locks but I believe this is all done correctly. running it through the buildbots now.

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 5d87b96 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F131440%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

This PR is stale because it has been open for 30 days with no activity.

@duaneg The CI failure for buildbot/AMD64 Debian root PR looks possibly related:

2 re-run tests:
    test.test_multiprocessing_fork.test_misc
    test.test_multiprocessing_spawn.test_misc
1 test failed:
    test.test_multiprocessing_spawn.test_misc

Can you have a look?

Can you have a look?

Certainly! Apologies for the delay, I'm just back from holiday.

The failing test case is test_large_pool, the proximate cause is os.fork failing due to memory exhaustion:

FAIL: test_large_pool (test.test_multiprocessing_spawn.test_misc.MiscTestCase.test_large_pool)
...
Traceback (most recent call last):                                              
  File "<string>", line 1, in <module>                                          
    from multiprocessing.forkserver ...
...
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 329, in main
    pid = os.fork()                                                             
OSError: [Errno 12] Cannot allocate memory                                      
Traceback (most recent call last):                                              
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/build/test_python_4137044æ/@test_4137044_tmpæ", line 4, in <module>
    with multiprocessing.Pool(200) as p:                                        
         ~~~~~~~~~~~~~~~~~~~~^^^^^                                              

The test times out when it is run the first time (for the fork and spawn methods). When rerun it passes for fork but gives that error for spawn. It runs successfully for forkserver.

First thing to note: it is rather odd that spawn method is failing, but the stack trace shows forkserver. In fact, since this test runs in a subprocess and doesn't set the method, it will always use the default multiprocessing method (here forkserver), regardless of which one it should be testing. The test was added to fix #89240, which was a Windows-specific bug, and possibly the author didn't realise or forgot the tests on Linux would run with multiple different methods.

Another thing to note: the tests are being run with -j2, so if the test is running for two methods at the same time it will be spawning up to 400 CPU-bound processes simultaneously.

I suspect we should update the generated code that runs in the subprocess to set the method, or possibly just run it once with the default. We should probably also annotate the test with @support.requires_resource('cpu') (although I'm not sure if that will change things for this buildbot). I am happy to make those changes, either in a separate PR or here if it is deemed more appropriate.

However, that aside, it is very suspicious that this failure crops up after this highly relevant change. Possibly it prevents or excessively delays reaping children, and that is enough to tip things over? It really shouldn't for forkserver, though: unless I'm missing something, for this test case, the change amounts to just one extra uncontested lock.

I shall try and investigate this locally over the next few days, but if anyone has any ideas, or has seen this test failing before, let me know!

I suspect we should update the generated code that runs in the subprocess to set the method, or possibly just run it once with the default. We should probably also annotate the test with @support.requires_resource('cpu') (although I'm not sure if that will change things for this buildbot). I am happy to make those changes, either in a separate PR or here if it is deemed more appropriate.

Do you investigate with the @test.support.bigmemtest decorator ?

Do you investigate with the @test.support.bigmemtest decorator ?

I'm afraid not: unfortunately I haven't had the time to investigate this in any detail, nor am I likely to in the immediate future. I will try and get back to it when I can, though.

This PR is stale because it has been open for 90 days with no activity.