gh-108388: Split test_multiprocessing_spawn - #108396
Merged
vstinner merged 1 commit intoAug 24, 2023
Merged
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration.
Copy link
Copy Markdown
Member
Author
|
Example running 4 slow tests: ./python -m test test_asyncio test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn -j0 --slowest Timing (reference => this PR): 3 min 22 sec => 2 min 22 sec (-1 min) On the main branch: 10 slowest tests: - test_multiprocessing_spawn: 3 min 22 sec - test_asyncio: 2 min 10 sec - test_multiprocessing_forkserver: 1 min 43 sec - test_multiprocessing_fork: 1 min 17 sec Total duration: 3 min 22 sec With this PR + PR #108393 (needed for this test): 40 tests OK.
10 slowest tests:
- test_multiprocessing_spawn.test_processes: 2 min 3 sec
- test_multiprocessing_forkserver.test_processes: 1 min 41 sec
- test_multiprocessing_spawn.test_misc: 1 min 22 sec
- test_multiprocessing_spawn.test_manager: 1 min 2 sec
- test_multiprocessing_forkserver.test_misc: 54.3 sec
- test_asyncio.test_tasks: 52.9 sec
- test_multiprocessing_forkserver.test_manager: 45.5 sec
- test_multiprocessing_fork.test_processes: 43.4 sec
- test_asyncio.test_events: 39.3 sec
- test_multiprocessing_fork.test_misc: 26.3 sec
2 tests skipped:
test_asyncio.test_windows_events test_asyncio.test_windows_utils
Total duration: 2 min 22 sec
All CPUs are well used, the system load peak is 12.33 which is a good thing (the machine has 12 threads/6 cores). Full logs: Detailsvstinner@mona$ ./python -m test test_asyncio test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn -j0 --slowest
0:00:00 load avg: 6.17 Run tests in parallel using 14 child processes
0:00:01 load avg: 6.17 [ 1/42] test_asyncio.test_protocols passed
0:00:01 load avg: 6.17 [ 2/42] test_asyncio.test_futures2 passed
0:00:01 load avg: 6.17 [ 3/42] test_asyncio.test_buffered_proto passed
0:00:01 load avg: 6.17 [ 4/42] test_asyncio.test_context passed
0:00:01 load avg: 6.17 [ 5/42] test_asyncio.test_pep492 passed
0:00:02 load avg: 6.17 [ 6/42] test_asyncio.test_runners passed
0:00:02 load avg: 6.17 [ 7/42] test_asyncio.test_proactor_events passed
0:00:02 load avg: 6.17 [ 8/42] test_asyncio.test_queues passed
0:00:03 load avg: 6.17 [ 9/42] test_asyncio.test_server passed
0:00:03 load avg: 6.17 [10/42] test_asyncio.test_selector_events passed
0:00:03 load avg: 6.17 [11/42] test_asyncio.test_eager_task_factory passed
0:00:04 load avg: 6.80 [12/42] test_asyncio.test_locks passed
0:00:04 load avg: 6.80 [13/42] test_asyncio.test_threads passed
0:00:06 load avg: 6.80 [14/42] test_asyncio.test_timeouts passed
0:00:06 load avg: 6.80 [15/42] test_asyncio.test_transports passed
0:00:07 load avg: 6.80 [16/42] test_asyncio.test_streams passed
0:00:07 load avg: 6.80 [17/42] test_asyncio.test_windows_events skipped
test_asyncio.test_windows_events skipped -- Windows only
0:00:07 load avg: 6.80 [18/42] test_asyncio.test_base_events passed
0:00:07 load avg: 6.80 [19/42] test_asyncio.test_sslproto passed
0:00:07 load avg: 6.80 [20/42] test_asyncio.test_windows_utils skipped
test_asyncio.test_windows_utils skipped -- Windows only
0:00:08 load avg: 6.80 [21/42] test_asyncio.test_unix_events passed
0:00:09 load avg: 7.70 [22/42] test_asyncio.test_waitfor passed
0:00:10 load avg: 7.70 [23/42] test_asyncio.test_sock_lowlevel passed
0:00:10 load avg: 7.70 [24/42] test_asyncio.test_sendfile passed
0:00:11 load avg: 7.70 [25/42] test_asyncio.test_taskgroups passed
0:00:12 load avg: 7.70 [26/42] test_asyncio.test_futures passed
0:00:18 load avg: 8.68 [27/42] test_asyncio.test_ssl passed
0:00:28 load avg: 10.42 [28/42] test_multiprocessing_fork.test_threads passed
0:00:29 load avg: 11.02 [29/42] test_asyncio.test_subprocess passed
Unknown child process pid 313199, will report returncode 255
Loop <_UnixSelectorEventLoop running=False closed=True debug=False> that handles pid 313199 is closed
0:00:32 load avg: 11.02 [30/42] test_multiprocessing_fork.test_manager passed -- running: test_asyncio.test_events (32.0 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310261) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=313364) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
0:00:32 load avg: 11.02 [31/42] test_multiprocessing_forkserver.test_threads passed -- running: test_asyncio.test_events (32.9 sec)
0:00:34 load avg: 11.50 [32/42] test_multiprocessing_fork.test_misc passed -- running: test_asyncio.test_events (34.6 sec), test_asyncio.test_tasks (31.7 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=313266) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
0:00:40 load avg: 12.02 [33/42] test_asyncio.test_events passed (39.3 sec) -- running: test_multiprocessing_forkserver.test_misc (30.9 sec), test_asyncio.test_tasks (37.1 sec), test_multiprocessing_fork.test_processes (32.4 sec), test_multiprocessing_forkserver.test_manager (31.1 sec)
0:00:51 load avg: 12.33 [34/42] test_multiprocessing_spawn.test_threads passed -- running: test_multiprocessing_forkserver.test_processes (40.9 sec), test_multiprocessing_spawn.test_processes (32.8 sec), test_multiprocessing_forkserver.test_misc (42.1 sec), test_multiprocessing_spawn.test_misc (38.9 sec), test_asyncio.test_tasks (48.2 sec), test_multiprocessing_fork.test_processes (43.5 sec), test_multiprocessing_spawn.test_manager (39.7 sec), test_multiprocessing_forkserver.test_manager (42.2 sec)
0:00:51 load avg: 12.33 [35/42] test_multiprocessing_fork.test_processes passed (43.4 sec) -- running: test_multiprocessing_forkserver.test_processes (41.5 sec), test_multiprocessing_spawn.test_processes (33.5 sec), test_multiprocessing_forkserver.test_misc (42.7 sec), test_multiprocessing_spawn.test_misc (39.5 sec), test_asyncio.test_tasks (48.8 sec), test_multiprocessing_spawn.test_manager (40.4 sec), test_multiprocessing_forkserver.test_manager (42.9 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310329) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310329) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
0:00:55 load avg: 12.06 [36/42] test_multiprocessing_forkserver.test_manager passed (45.5 sec) -- running: test_multiprocessing_forkserver.test_processes (45.0 sec), test_multiprocessing_spawn.test_processes (37.0 sec), test_multiprocessing_forkserver.test_misc (46.2 sec), test_multiprocessing_spawn.test_misc (43.0 sec), test_asyncio.test_tasks (52.3 sec), test_multiprocessing_spawn.test_manager (43.9 sec)
0:00:56 load avg: 12.06 [37/42] test_asyncio.test_tasks passed (52.9 sec) -- running: test_multiprocessing_forkserver.test_processes (46.3 sec), test_multiprocessing_spawn.test_processes (38.2 sec), test_multiprocessing_forkserver.test_misc (47.5 sec), test_multiprocessing_spawn.test_misc (44.3 sec), test_multiprocessing_spawn.test_manager (45.1 sec)
0:01:04 load avg: 10.75 [38/42] test_multiprocessing_forkserver.test_misc passed (54.3 sec) -- running: test_multiprocessing_forkserver.test_processes (53.8 sec), test_multiprocessing_spawn.test_processes (45.7 sec), test_multiprocessing_spawn.test_misc (51.8 sec), test_multiprocessing_spawn.test_manager (52.6 sec)
0:01:14 load avg: 9.63 [39/42] test_multiprocessing_spawn.test_manager passed (1 min 2 sec) -- running: test_multiprocessing_forkserver.test_processes (1 min 4 sec), test_multiprocessing_spawn.test_processes (56.1 sec), test_multiprocessing_spawn.test_misc (1 min 2 sec)
0:01:35 load avg: 12.00 [40/42] test_multiprocessing_spawn.test_misc passed (1 min 22 sec) -- running: test_multiprocessing_forkserver.test_processes (1 min 24 sec), test_multiprocessing_spawn.test_processes (1 min 16 sec)
0:01:52 load avg: 13.34 [41/42] test_multiprocessing_forkserver.test_processes passed (1 min 41 sec) -- running: test_multiprocessing_spawn.test_processes (1 min 34 sec)
0:02:22 load avg: 8.84 running: test_multiprocessing_spawn.test_processes (2 min 4 sec)
0:02:22 load avg: 8.84 [42/42] test_multiprocessing_spawn.test_processes passed (2 min 3 sec)
== Tests result: SUCCESS ==
40 tests OK.
10 slowest tests:
- test_multiprocessing_spawn.test_processes: 2 min 3 sec
- test_multiprocessing_forkserver.test_processes: 1 min 41 sec
- test_multiprocessing_spawn.test_misc: 1 min 22 sec
- test_multiprocessing_spawn.test_manager: 1 min 2 sec
- test_multiprocessing_forkserver.test_misc: 54.3 sec
- test_asyncio.test_tasks: 52.9 sec
- test_multiprocessing_forkserver.test_manager: 45.5 sec
- test_multiprocessing_fork.test_processes: 43.4 sec
- test_asyncio.test_events: 39.3 sec
- test_multiprocessing_fork.test_misc: 26.3 sec
2 tests skipped:
test_asyncio.test_windows_events test_asyncio.test_windows_utils
Total duration: 2 min 22 sec
|
Sorry, something went wrong.
gpshead
reviewed
Aug 24, 2023
Comment thread
Lib/test/_test_multiprocessing.py
|
|
||
| def install_tests_in_module_dict(remote_globs, start_method): | ||
| def install_tests_in_module_dict(remote_globs, start_method, | ||
| only_type=None, exclude_types=False): |
Copy link
Copy Markdown
Member
There was a problem hiding this comment.
This function needs proper documentation as to what it does and how it behaves. The new parameters behaviors are unobvious.
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
|
Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12. |
Sorry, something went wrong.
Copy link
Copy Markdown
|
GH-108442 is a backport of this pull request to the 3.12 branch. |
Sorry, something went wrong.
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Aug 24, 2023
Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>
Yhg1s
pushed a commit
that referenced
this pull request
Aug 25, 2023
gh-108388: Split test_multiprocessing_spawn (GH-108396) Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>
Copy link
Copy Markdown
Contributor
|
Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11. |
Sorry, something went wrong.
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Sep 21, 2023
Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>
Copy link
Copy Markdown
|
GH-109688 is a backport of this pull request to the 3.11 branch. |
Sorry, something went wrong.
vstinner
added a commit
to miss-islington/cpython
that referenced
this pull request
Sep 21, 2023
Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>
vstinner
added a commit
that referenced
this pull request
Sep 21, 2023
gh-108388: Split test_multiprocessing_spawn (GH-108396) Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration.