FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-146004: propagate all -X options to multiprocessing child processes by gpshead · Pull Request #146005 · python/cpython · GitHub

/ cpython Public

gh-146004: propagate all -X options to multiprocessing child processes - #146005

Merged
gpshead merged 2 commits into
python:mainfrom
gpshead:propagate-x-options-gh146004
Mar 28, 2026
Merged

gh-146004: propagate all -X options to multiprocessing child processes#146005
gpshead merged 2 commits into
python:mainfrom
gpshead:propagate-x-options-gh146004

Conversation

gpshead commented Mar 16, 2026
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Member

See the issue for details. Our list of -X options to opt-in was overlooking a lot. CI and buildbot tests on these may prove interesting.

vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM.

gpshead marked this pull request as ready for review March 28, 2026 04:32
gpshead enabled auto-merge (squash) March 28, 2026 04:37
gpshead added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Mar 28, 2026

gpshead commented Mar 28, 2026

Copy link
Copy Markdown
Member Author

Auto-backported test will probably fail CI as the test includes new -X options that older releases don't have. I'll clean that up in the backport PR branches.

gpshead merged commit 1efe441 into python:main Mar 28, 2026
52 checks passed

Copy link
Copy Markdown

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 28, 2026
…ocesses (pythonGH-146005)

Propagate all -X command line options to multiprocessing spawned child Python processes.
(cherry picked from commit 1efe441)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>

Copy link
Copy Markdown

Sorry, @gpshead, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 1efe441de7c448852b9ba51fb0db4d355a7157a8 3.13

bedevere-app Bot commented Mar 28, 2026

Copy link
Copy Markdown

GH-146552 is a backport of this pull request to the 3.14 branch.

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Mar 28, 2026
gpshead added a commit that referenced this pull request Mar 28, 2026
…rocesses (GH-146005) (#146552)

gh-146004: propagate all -X options to multiprocessing child processes (GH-146005)

Propagate all -X command line options to multiprocessing spawned child Python processes.
(cherry picked from commit 1efe441)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows Server 2022 NoGIL 3.x (tier-1) has failed when building commit 1efe441.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1241/builds/8204) and take a look at the build logs.
  4. Check if the failure is related to this commit (1efe441) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1241/builds/8204

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "C:\bbarea\3.x.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\test\support\os_helper.py", line 524, in temp_dir
    os.mkdir(path)
    ~~~~~~~~^^^^^^
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\bbarea\\3.x.itamaro-win64-srv-22-aws.x64.nogil\\build\\build\\test_python_13292æ'
Stack (most recent call last):
  File "C:\bbarea\3.x.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\bbarea\3.x.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\bbarea\3.x.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\test\libregrtest\worker.py", line 141, in <module>
    main()
  File "C:\bbarea\3.x.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\test\libregrtest\worker.py", line 136, in main
    with os_helper.temp_cwd(work_dir, quiet=True):
  File "C:\bbarea\3.x.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "C:\bbarea\3.x.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\test\support\os_helper.py", line 597, in temp_cwd
    with temp_dir(path=name, quiet=quiet) as temp_path:

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows PGO Tailcall 3.x (tier-1) has failed when building commit 1efe441.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1870/builds/252) and take a look at the build logs.
  4. Check if the failure is related to this commit (1efe441) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1870/builds/252

Failed tests:

  • test_profiling

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "C:\bbarea\3.x.itamaro-win64-srv-22-aws.tailcall.pgo\build\Lib\test\test_profiling\test_sampling_profiler\test_advanced.py", line 169, in test_native_frames_enabled
    self.assertFalse(any(stack.endswith(";<native>") for stack in stacks))
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: True is not false

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows Server 2022 NoGIL 3.14 (tier-1) has failed when building commit 28d3b9b.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1717/builds/1129) and take a look at the build logs.
  4. Check if the failure is related to this commit (28d3b9b) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1717/builds/1129

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 11, done.        
remote: Counting objects:  10% (1/10)        
remote: Counting objects:  20% (2/10)        
remote: Counting objects:  30% (3/10)        
remote: Counting objects:  40% (4/10)        
remote: Counting objects:  50% (5/10)        
remote: Counting objects:  60% (6/10)        
remote: Counting objects:  70% (7/10)        
remote: Counting objects:  80% (8/10)        
remote: Counting objects:  90% (9/10)        
remote: Counting objects: 100% (10/10)        
remote: Counting objects: 100% (10/10), done.        
remote: Total 11 (delta 9), reused 9 (delta 9), pack-reused 1 (from 1)        
From https://github.com/python/cpython
 * branch                    3.14       -> FETCH_HEAD
Note: switching to '28d3b9bc56df6c2da3e898a0b7a066df5761b04d'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 28d3b9bc56d [3.14] gh-146004: propagate all -X options to multiprocessing child processes (GH-146005) (#146552)
Switched to and reset branch '3.14'

Could Not Find C:\bbarea\3.14.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find C:\bbarea\3.14.itamaro-win64-srv-22-aws.x64.nogil\build\PCbuild\python*.zip

Could Not Find C:\bbarea\3.14.itamaro-win64-srv-22-aws.x64.nogil\build\PCbuild\python*.zip

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.x (tier-1) has failed when building commit 1efe441.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/146/builds/13440) and take a look at the build logs.
  4. Check if the failure is related to this commit (1efe441) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/146/builds/13440

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\support\os_helper.py", line 524, in temp_dir
    os.mkdir(path)
    ~~~~~~~~^^^^^^
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'D:\\buildarea\\3.x.bolen-windows10\\build\\build\\test_python_1152æ'
Stack (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\libregrtest\worker.py", line 141, in <module>
    main()
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\libregrtest\worker.py", line 136, in main
    with os_helper.temp_cwd(work_dir, quiet=True):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\support\os_helper.py", line 597, in temp_cwd
    with temp_dir(path=name, quiet=quiet) as temp_path:


Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\support\os_helper.py", line 524, in temp_dir
    os.mkdir(path)
    ~~~~~~~~^^^^^^
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'D:\\buildarea\\3.x.bolen-windows10\\build\\build\\test_python_11832æ'
Stack (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\libregrtest\worker.py", line 141, in <module>
    main()
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\libregrtest\worker.py", line 136, in main
    with os_helper.temp_cwd(work_dir, quiet=True):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\support\os_helper.py", line 597, in temp_cwd
    with temp_dir(path=name, quiet=quiet) as temp_path:

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows11 Bigmem 3.x (tier-1) has failed when building commit 1efe441.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1079/builds/8289) and take a look at the build logs.
  4. Check if the failure is related to this commit (1efe441) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1079/builds/8289

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 11, done.        
remote: Counting objects:  10% (1/10)        
remote: Counting objects:  20% (2/10)        
remote: Counting objects:  30% (3/10)        
remote: Counting objects:  40% (4/10)        
remote: Counting objects:  50% (5/10)        
remote: Counting objects:  60% (6/10)        
remote: Counting objects:  70% (7/10)        
remote: Counting objects:  80% (8/10)        
remote: Counting objects:  90% (9/10)        
remote: Counting objects: 100% (10/10)        
remote: Counting objects: 100% (10/10), done.        
remote: Compressing objects:  25% (1/4)        
remote: Compressing objects:  50% (2/4)        
remote: Compressing objects:  75% (3/4)        
remote: Compressing objects: 100% (4/4)        
remote: Compressing objects: 100% (4/4), done.        
remote: Total 11 (delta 6), reused 6 (delta 6), pack-reused 1 (from 1)        
From https://github.com/python/cpython
 * branch                    main       -> FETCH_HEAD
Note: switching to '1efe441de7c448852b9ba51fb0db4d355a7157a8'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 1efe441de7c gh-146004: propagate all -X options to multiprocessing child processes (GH-146005)
Switched to and reset branch 'main'

Could Not Find R:\buildarea\3.x.ambv-bb-win11.bigmem\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find R:\buildarea\3.x.ambv-bb-win11.bigmem\build\PCbuild\python*.zip

Could Not Find R:\buildarea\3.x.ambv-bb-win11.bigmem\build\PCbuild\python*.zip

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.14 (tier-1) has failed when building commit 28d3b9b.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1711/builds/890) and take a look at the build logs.
  4. Check if the failure is related to this commit (28d3b9b) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1711/builds/890

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 11, done.        
remote: Counting objects:  14% (1/7)        
remote: Counting objects:  28% (2/7)        
remote: Counting objects:  42% (3/7)        
remote: Counting objects:  57% (4/7)        
remote: Counting objects:  71% (5/7)        
remote: Counting objects:  85% (6/7)        
remote: Counting objects: 100% (7/7)        
remote: Counting objects: 100% (7/7), done.        
remote: Total 11 (delta 7), reused 7 (delta 7), pack-reused 4 (from 2)        
From https://github.com/python/cpython
 * branch                    3.14       -> FETCH_HEAD
Note: switching to '28d3b9bc56df6c2da3e898a0b7a066df5761b04d'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 28d3b9bc56d [3.14] gh-146004: propagate all -X options to multiprocessing child processes (GH-146005) (#146552)
Switched to and reset branch '3.14'

Could Not Find D:\buildarea\3.14.bolen-windows10\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find D:\buildarea\3.14.bolen-windows10\build\PCbuild\python*.zip

Could Not Find D:\buildarea\3.14.bolen-windows10\build\PCbuild\python*.zip

Copy link
Copy Markdown
Member

test_support fails on Windows buildbots:

Warning -- files was modified by test_support
Warning --   Before: []
Warning --   After:  ['tmp/']

I don't see the relationship with the change.

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows11 Non-Debug 3.14 (tier-1) has failed when building commit 28d3b9b.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1684/builds/511) and take a look at the build logs.
  4. Check if the failure is related to this commit (28d3b9b) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1684/builds/511

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 16, done.        
remote: Counting objects:  12% (1/8)        
remote: Counting objects:  25% (2/8)        
remote: Counting objects:  37% (3/8)        
remote: Counting objects:  50% (4/8)        
remote: Counting objects:  62% (5/8)        
remote: Counting objects:  75% (6/8)        
remote: Counting objects:  87% (7/8)        
remote: Counting objects: 100% (8/8)        
remote: Counting objects: 100% (8/8), done.        
remote: Total 16 (delta 7), reused 7 (delta 7), pack-reused 8 (from 3)        
From https://github.com/python/cpython
 * branch                    3.14       -> FETCH_HEAD
Note: switching to '28d3b9bc56df6c2da3e898a0b7a066df5761b04d'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 28d3b9bc56d [3.14] gh-146004: propagate all -X options to multiprocessing child processes (GH-146005) (#146552)
Switched to and reset branch '3.14'

Could Not Find b:\uildarea\3.14.ware-win11.nondebug\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find b:\uildarea\3.14.ware-win11.nondebug\build\PCbuild\python*.zip

Could Not Find b:\uildarea\3.14.ware-win11.nondebug\build\PCbuild\python*.zip

picnixz commented Mar 28, 2026
edited
Loading

Copy link
Copy Markdown
Member

I think the problem is ['-X', 'pycache_prefix=/tmp/pycache']. We should use another temporary folder here.

(The problem is likely because we are using / on Windows platforms)

Copy link
Copy Markdown
Member

I suggest using tempfile.gettempdir().

picnixz commented Mar 28, 2026
edited
Loading

Copy link
Copy Markdown
Member

I suggest just using test.support's temporary directory instead (a fresh one). It doesn't matter that we use a real OS-tempdir

gpshead commented Mar 28, 2026

Copy link
Copy Markdown
Member Author

I won't be able to get to this today. That makes sense as a fix.

picnixz commented Mar 28, 2026

Copy link
Copy Markdown
Member

No worries! Chris handled it already: #146580

Comment thread Lib/test/test_support.py
['-X', 'dev'],
['-Wignore', '-X', 'dev'],
['-X', 'cpu_count=4'],
['-X', 'disable-remote-debug'],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This is documented as

-X disable_remote_debug

with underscores.

Furthermore, -X presite=package.module, -X gil=0,1 and some more aren't listed here.

Not a big issue IMHO, since this just tests that whatever we pass in is returned from args_from_interpreter_flags?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

So an alternative fix for #146580 would be to just remove ['-X', 'pycache_prefix=/tmp/pycache'],?

Sorry, didn't realize earlier, just stumbled over it now ...

clin1234 pushed a commit to clin1234/cpython that referenced this pull request Apr 16, 2026
…ocesses (pythonGH-146005)

Propagate all -X command line options to multiprocessing spawned child Python processes.
ljfp pushed a commit to ljfp/cpython that referenced this pull request Apr 25, 2026
…ocesses (pythonGH-146005)

Propagate all -X command line options to multiprocessing spawned child Python processes.

Copy link
Copy Markdown
Member

Please don't forget about backports.

bedevere-app Bot commented Aug 17, 2026

Copy link
Copy Markdown

GH-155938 is a backport of this pull request to the 3.13 branch.

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Aug 17, 2026
hanlee55 added a commit to hanlee55/cpython that referenced this pull request Aug 17, 2026
…ocesses (pythonGH-146005)

Propagate all -X command line options to multiprocessing spawned child Python processes.
(cherry picked from commit 1efe441)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
hugovk pushed a commit that referenced this pull request Aug 17, 2026
…rocesses (GH-146005) (#155938)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants


Back | FazBrowse Home | New Git URL