Repository navigation
Improve bigmem tests #75876
Description
| BPO | 31695 |
|---|---|
| Nosy | @pitrou, @vstinner, @serhiy-storchaka |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2017-10-04.19:22:19.577>
labels = ['type-feature', 'tests']
title = 'Improve bigmem tests'
updated_at = <Date 2017-10-07.15:45:09.628>
user = 'https://github.com/serhiy-storchaka'bugs.python.org fields:
activity = <Date 2017-10-07.15:45:09.628>
actor = 'pitrou'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2017-10-04.19:22:19.577>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 31695
keywords = []
message_count = 2.0
messages = ['303729', '303885']
nosy_count = 3.0
nosy_names = ['pitrou', 'vstinner', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31695'
versions = []Linked PRs
- gh-75876: Use functools.wraps() in test.support test decorators #155262
- [3.15] gh-75876: Use functools.wraps() in test.support test decorators (GH-155262) #155290
- [3.14] gh-75876: Use functools.wraps() in test.support test decorators (GH-155262) #155291
- gh-75876: Run bigmem tests in a subprocess #155302
- gh-75876: Correct the memory use declared by some bigmem tests #155307
- [3.13] gh-75876: Use functools.wraps() in test.support test decorators (GH-155262) #155311
- [3.15] gh-75876: Correct the memory use declared by some bigmem tests (GH-155307) #155312
- [3.14] gh-75876: Correct the memory use declared by some bigmem tests (GH-155307) #155313
- [3.13] gh-75876: Correct the memory use declared by some bigmem tests (GH-155307) #155314
- [3.14] gh-75876: Run bigmem tests in a subprocess (GH-155302) #156013
- [3.13] gh-75876: Run bigmem tests in a subprocess (GH-155302) #156014
- gh-75876: Limit the address space of a bigmem test #156023
- gh-75876: Report the peak memory use of a bigmem test #156024
- [3.15] gh-75876: Run bigmem tests in a subprocess (GH-155302) #159087
Activity
serhiy-storchaka commented on Oct 4, 2017
I think that the process of running bigmem tests should be improved.
Every bigmem test should run in separate process. It's effect on memory fragmentation shouldn't affect other tests.
Only one bigmem test should be run at the same time. Otherwise several parallel bigmem tests can exhaust RAM and lead to swapping. In multiprocessing mode the process running a bigmem test should send a signal to other processes, wait until they finished their current tests and paused, run a bigmem test, and send a signal to other processes that they can continue.
The hard limit of addressed memory should be set for every bigmem test, according to the declared requirements. It is better to crash one test and report the failure, than hang on swapping.
All other tests should be run with the limit set to 2 GiB (or 1 GiB?). This will help to find memory consuming tests which should be made bigmem tests.
The maxrss of the process running a bigmem test should be reported in verbose mode after finishing a test.
pitrou commented on Oct 7, 2017
+1 to all this :-)
cmaloney commented on May 9, 2026
Currently bigmemtest in verbose mode only reports memory usage on Linux by way of reading /proc. I do a lot of work on macOS and would like to be able to see quickly the memory usage there. Hoping to add Windows and macOS support to help improve bigmemtest. With that also fix 5 above, printing the maxrss after finishing each test.
To support that I'd like to add a private C extension function that uses platform-specific APIs to get process memory usage and update _MemoryWatchdog / memory_watchdog.py to use that information in its print. That makes it so the platform-specific code lives entirely inside C conditional compilation.
The code currently pre-opens the /proc file and passes that through as stdin to reduce PID recycling issues if the parent exits. Unfortunately there is no cross-platform pidfd type mechanism to make this really simple. Instead the new code relies on a pipe to the parent to determine "has the parent been lost" which lets us detect parent process termination and prevent issues if the PID is recycled. It also means we should exit more quickly when the parent intentionally exits and gives a good point to add the end of test print.
vstinner commented on May 11, 2026
This issue has not been touched for 9 years, I suggest you opening a new issue for your idea. We can add a private function in a test module such as _testcapi. I didn't understand the pidfd issue, I should see the code to understand :-)
StanFromIreland commented on Jun 25, 2026
Currently on our buildbot the test takes over one and a half hours, while some of that is unavoidable, it would be good to decrease it a little if possible.
vstinner commented on Jun 25, 2026
I modified test.libregrtest to log the memory usage with a new get_process_memory_usage() function in test.libregrtest.utils. The function works on Linux, Windows, macOS and FreeBSD.
I also modified @support.bigmemtest to reuse this get_process_memory_usage() function.
Example:
The mem: 14.8 GiB part is new!
gh-75876: Use functools.wraps() in test.support test decorators (GH-1…
[3.13] gh-75876: Use functools.wraps() in test.support test decorators (
gh-75876: Correct the memory use declared by some bigmem tests (GH-15…
[3.13] gh-75876: Correct the memory use declared by some bigmem tests (…
[3.14] gh-75876: Correct the memory use declared by some bigmem tests (…
serhiy-storchaka commented on Aug 7, 2026
#155302 makes the bigmem tests running in a separate subprocess -- point 1 in my plan. Thanks to reusing the runInSubprocess machinery it is transparent to the user -- you can use all TestCase methods. The follow-ups will implement other points (I already have 3 and 5).
#155307 corrects underspecified memory usage. It was needed to be fixed before imposing the hard limit (point 3). There were only 6 incorrect values in 233 bigmem tests -- for most tests the estimate was very accurate, as result of my past work.
2 remaining items
[3.15] gh-75876: Use functools.wraps() in test.support test decorators (
pythongh-75876: Use functools.wraps() in test.support test decorators (…
pythongh-75876: Correct the memory use declared by some bigmem tests (p…
gh-75876: Run bigmem tests in a subprocess (GH-155302)
gh-75876: Report the peak memory use of a bigmem test (GH-156024)
serhiy-storchaka commented on Aug 19, 2026
#156024 implements point 5 -- reports the maxrss and majflt of the process running a bigmem test.
pythongh-75876: Correct the memory use declared by two bigmem tests
pythongh-75876: Kill a bigmem test which uses more memory than it dec…
serhiy-storchaka commented on Aug 22, 2026
#156023 implements point 2 -- limits the address space of a bigmem test. There are a few subtleties. Glibc reserves some amount of address space per thread per CPU core (it is not really used), so there is an option to skip the address space limitation for tests with large amount of threads. ASan build and macOS also reserve huge address space -- it is skipped there too. For these cases (and for Windows which does not have such API), the watchdog kills the test when it's real use exceeds the limit.
[3.13] gh-75876: Run bigmem tests in a subprocess (GH-155302) (#156014)
[3.14] gh-75876: Run bigmem tests in a subprocess (GH-155302) (GH-156013
[3.15] gh-75876: Run bigmem tests in a subprocess (GH-155302) (GH-159087