gh-109649: Use os.process_cpu_count() by vstinner · Pull Request #110165 · python/cpython · GitHub

Replace os.cpu_count() with os.process_cpu_count() in modules:

compileall
concurrent.futures
multiprocessing

Replace os.cpu_count() with os.process_cpu_count() in programs:

_decimal deccheck.py test
freeze.py
multissltests.py
python -m test (regrtest)
wasm_build.py

Other changes:

test.pythoninfo logs os.process_cpu_count().
regrtest gets os.process_cpu_count() / os.cpu_count() in headers.

Issue: os.process_cpu_count(): get the number CPUs usable by the current process #109649

📚 Documentation preview 📚: https://cpython-previews--110165.org.readthedocs.build/

Replace os.cpu_count() with os.process_cpu_count() in modules: * compileall * concurrent.futures * multiprocessing Replace os.cpu_count() with os.process_cpu_count() in programs: * _decimal deccheck.py test * freeze.py * multissltests.py * python -m test (regrtest) * wasm_build.py Other changes: * test.pythoninfo logs os.process_cpu_count(). * regrtest gets os.process_cpu_count() / os.cpu_count() in headers.

Example: CPU count: 2 (process) / 12 (system)

vstinner@mona$ taskset -c 0-1 ./python -m test 
== CPython 3.13.0a0 (heads/use_process_count:a8514200b8, Oct 1 2023, 01:31:10) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)]
== Linux-6.4.15-200.fc38.x86_64-x86_64-with-glibc2.37 little-endian
== Python build: debug
== cwd: /home/vstinner/python/main/build/test_python_worker_2132029æ
== CPU count: 2 (process) / 12 (system)
(...)

Replace os.cpu_count() with os.process_cpu_count() in programs:

_decimal deccheck.py test
freeze.py
multissltests.py
python -m test (regrtest)
wasm_build.py

multissltests.py and wasm_build.py use the code below to keep support for old Python versions:

    if hasattr(os, 'process_cpu_count'):
        jobs = os.process_cpu_count()
    else:
        jobs = os.cpu_count()

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x SLES 3.x has failed when building commit a46e960.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/540/builds/6704) and take a look at the build logs.
Check if the failure is related to this commit (a46e960) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/540/builds/6704

Failed tests:

test.test_concurrent_futures.test_thread_pool

Failed subtests:

test_idle_thread_reuse - test.test_concurrent_futures.test_thread_pool.ThreadPoolExecutorTest.test_idle_thread_reuse

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_concurrent_futures/test_thread_pool.py", line 49, in test_idle_thread_reuse
    self.assertEqual(len(executor._threads), 1)
AssertionError: 2 != 1

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Debian 3.x has failed when building commit a46e960.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/49/builds/6739) and take a look at the build logs.
Check if the failure is related to this commit (a46e960) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/49/builds/6739

Failed tests:

test.test_multiprocessing_forkserver.test_processes

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 40, done.        
remote: Counting objects:   2% (1/40)        
remote: Counting objects:   5% (2/40)        
remote: Counting objects:   7% (3/40)        
remote: Counting objects:  10% (4/40)        
remote: Counting objects:  12% (5/40)        
remote: Counting objects:  15% (6/40)        
remote: Counting objects:  17% (7/40)        
remote: Counting objects:  20% (8/40)        
remote: Counting objects:  22% (9/40)        
remote: Counting objects:  25% (10/40)        
remote: Counting objects:  27% (11/40)        
remote: Counting objects:  30% (12/40)        
remote: Counting objects:  32% (13/40)        
remote: Counting objects:  35% (14/40)        
remote: Counting objects:  37% (15/40)        
remote: Counting objects:  40% (16/40)        
remote: Counting objects:  42% (17/40)        
remote: Counting objects:  45% (18/40)        
remote: Counting objects:  47% (19/40)        
remote: Counting objects:  50% (20/40)        
remote: Counting objects:  52% (21/40)        
remote: Counting objects:  55% (22/40)        
remote: Counting objects:  57% (23/40)        
remote: Counting objects:  60% (24/40)        
remote: Counting objects:  62% (25/40)        
remote: Counting objects:  65% (26/40)        
remote: Counting objects:  67% (27/40)        
remote: Counting objects:  70% (28/40)        
remote: Counting objects:  72% (29/40)        
remote: Counting objects:  75% (30/40)        
remote: Counting objects:  77% (31/40)        
remote: Counting objects:  80% (32/40)        
remote: Counting objects:  82% (33/40)        
remote: Counting objects:  85% (34/40)        
remote: Counting objects:  87% (35/40)        
remote: Counting objects:  90% (36/40)        
remote: Counting objects:  92% (37/40)        
remote: Counting objects:  95% (38/40)        
remote: Counting objects:  97% (39/40)        
remote: Counting objects: 100% (40/40)        
remote: Counting objects: 100% (40/40), done.        
remote: Compressing objects:  16% (1/6)        
remote: Compressing objects:  33% (2/6)        
remote: Compressing objects:  50% (3/6)        
remote: Compressing objects:  66% (4/6)        
remote: Compressing objects:  83% (5/6)        
remote: Compressing objects: 100% (6/6)        
remote: Compressing objects: 100% (6/6), done.        
remote: Total 40 (delta 34), reused 40 (delta 34), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to 'a46e96076898d126c9f276aef1934195aac34b4e'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at a46e960768 gh-109649: Use os.process_cpu_count() (#110165)
Switched to and reset branch 'main'

make: *** [Makefile:2018: buildbottest] Error 5

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora 3.x has failed when building commit a46e960.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/223/builds/4583) and take a look at the build logs.
Check if the failure is related to this commit (a46e960) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/223/builds/4583

Failed tests:

test.test_concurrent_futures.test_shutdown

Failed subtests:

test_interpreter_shutdown - test.test_concurrent_futures.test_shutdown.ProcessPoolForkserverProcessPoolShutdownTest.test_interpreter_shutdown

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 50, in test_interpreter_shutdown
    self.assertEqual(out.strip(), b"apple")
AssertionError: b'' != b'apple'

Replace os.cpu_count() with os.process_cpu_count() in modules: * compileall * concurrent.futures * multiprocessing Replace os.cpu_count() with os.process_cpu_count() in programs: * _decimal deccheck.py test * freeze.py * multissltests.py * python -m test (regrtest) * wasm_build.py Other changes: * test.pythoninfo logs os.process_cpu_count(). * regrtest gets os.process_cpu_count() / os.cpu_count() in headers.

vstinner requested review from brettcannon and rhettinger as code owners September 30, 2023 23:48

bedevere-app Bot added the awaiting review label Sep 30, 2023

bedevere-app Bot mentioned this pull request Sep 30, 2023

os.process_cpu_count(): get the number CPUs usable by the current process #109649

Closed

vstinner added 2 commits October 1, 2023 01:51

wasm_build.py: keep support with old Python

6c336bc

multissltests.py: keep support with old Python

0cc90be

corona10 approved these changes Sep 30, 2023

View reviewed changes

bedevere-app Bot added awaiting core review and removed awaiting review labels Sep 30, 2023

Complete the doc

6059dee

vstinner merged commit a46e960 into python:main Oct 1, 2023

vstinner deleted the use_process_count branch October 1, 2023 01:14

bedevere-app Bot removed the awaiting core review label Oct 1, 2023

gh-109649: Use os.process_cpu_count() by vstinner · Pull Request #110165 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-109649: Use os.process_cpu_count() - #110165

gh-109649: Use os.process_cpu_count()#110165
vstinner merged 4 commits into
python:mainfrom
vstinner:use_process_count

vstinner commented Sep 30, 2023 •
edited by github-actions Bot
Loading

Uh oh!

Uh oh!

vstinner commented Sep 30, 2023

Uh oh!

vstinner commented Sep 30, 2023

Uh oh!

bedevere-bot commented Oct 1, 2023

Uh oh!

bedevere-bot commented Oct 1, 2023

Uh oh!

bedevere-bot commented Oct 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gh-109649: Use os.process_cpu_count() by vstinner · Pull Request #110165 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-109649: Use os.process_cpu_count() - #110165

gh-109649: Use os.process_cpu_count()#110165vstinner merged 4 commits intopython:mainpython/cpython:mainfrom vstinner:use_process_countvstinner/cpython:use_process_countCopy head branch name to clipboard

Conversation

vstinner commented Sep 30, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Sep 30, 2023

Uh oh!

vstinner commented Sep 30, 2023

Uh oh!

bedevere-bot commented Oct 1, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Oct 1, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Oct 1, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gh-109649: Use os.process_cpu_count()#110165
vstinner merged 4 commits into
python:mainfrom
vstinner:use_process_count

vstinner commented Sep 30, 2023 •
edited by github-actions Bot
Loading