Repository navigation
gh-75876: Limit the address space of a bigmem test - #156023
Open
serhiy-storchaka wants to merge 5 commits into
Open
serhiy-storchaka wants to merge 5 commits into
serhiy-storchaka wants to merge 5 commits into
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
A test run with -M now runs in a subprocess whose address space is limited to what it declares plus 512 MiB. A test which uses much more memory than it declares fails with a MemoryError instead of making the machine swap. Pass limit_address_space=False for a test which reserves much more address space than it uses. The two threaded tests in test_interpreters do: glibc reserves an arena per thread, up to 8 per CPU, and this alone exceeds the declared size on a machine with many cores.
Copy link
Copy Markdown
Documentation build overview4 files changed ± library/tarfile.html ± library/test.html ± tutorial/stdlib.html ± whatsnew/changelog.html |
Sorry, something went wrong.
AddressSanitizer reserves terabytes of address space for its shadow memory, so any limit stops the interpreter from starting. macOS reserves much more address space than Linux does, so 512 MiB is not enough of a margin there.
Copy link
Copy Markdown
Member
Author
|
!buildbot bigmem |
Sorry, something went wrong.
Copy link
Copy Markdown
|
🤖 New build scheduled with the buildbot fleet by @serhiy-storchaka for commit 8bc2c01 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F156023%2Fmerge The command will test the builders whose names match following regular expression: bigmem The builders matched are:
|
Sorry, something went wrong.
repr() of a list or a tuple of 2**31 False takes 15.3 bytes per item, not 15, and str.split() takes 10.7 bytes per item and more for a larger string, not 10. The difference is small, but it grows with the size of the test: 614 MiB for test_repr_large.
…lares The address space cannot be limited everywhere: not under AddressSanitizer, which reserves terabytes for its shadow memory, and not on macOS, which reserves much more than it uses. The parent process already watches the memory of the subprocess running the test, so let it kill the test which uses more than it declares plus the same margin. This also catches a test which fills the memory without reserving that much address space.
The memory watchdog now both reports the peak (pythonGH-156024) and kills a test which uses more memory than it declares.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A test run with -M now runs with the address space limited to what it declares plus 512 MiB, so that a test which uses much more memory than it declares fails with a MemoryError instead of making the machine swap.
The margin does not grow with the declared size: the interpreter itself reserves about 250 MiB, whatever the test asks for.
limit_address_space=False exempts a test which reserves much more address space than it uses. The two threaded tests in test_interpreters need it: glibc reserves an arena per thread, up to 8 per CPU, and this alone exceeds the declared size on a machine with many cores.