FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-129005: Remove copies from _pyio using take_bytes by cmaloney · Pull Request #141539 · python/cpython · GitHub

/ cpython Public

gh-129005: Remove copies from _pyio using take_bytes - #141539

Merged
vstinner merged 2 commits into
python:mainfrom
cmaloney:gh129005_take_bytes
Nov 18, 2025
ConversationCommits (2)ChecksFiles changed
Merged

gh-129005: Remove copies from _pyio using take_bytes#141539
vstinner merged 2 commits into
python:mainfrom
cmaloney:gh129005_take_bytes

Conversation

cmaloney commented Nov 13, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Memory usage now matches that of _io for large files.

$ ./python -Wd -I -m test test_io.test_largefile -uall,largefile,walltime -M32G -v
== CPython 3.15.0a1+ (heads/gh129005_take_bytes-dirty:d8b88498ebf, Nov 13 2025, 15:43:51) [Clang 21.1.5 ]
== Linux-6.17.7-arch1-1-x86_64-with-glibc2.42 little-endian
== Python build: debug
== cwd: <build_dir>/build/test_python_worker_116910æ
== CPU count: 32
== encodings: locale=UTF-8 FS=utf-8
== resources: all

Using random seed: 365280675
0:00:00 load avg: 9.60 Run 1 test sequentially in a single process
0:00:00 load avg: 9.60 [1/1] test_io.test_largefile
test_large_read (test.test_io.test_largefile.CLargeFileTest.test_large_read) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
ok
test_lseek (test.test_io.test_largefile.CLargeFileTest.test_lseek) ... ok
test_osstat (test.test_io.test_largefile.CLargeFileTest.test_osstat) ... ok
test_seek_read (test.test_io.test_largefile.CLargeFileTest.test_seek_read) ... ok
test_seek_readall (test.test_io.test_largefile.CLargeFileTest.test_seek_readall) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
 ... process data size: 2.3G
ok
test_seekable (test.test_io.test_largefile.CLargeFileTest.test_seekable) ... ok
test_truncate (test.test_io.test_largefile.CLargeFileTest.test_truncate) ... ok
test_large_read (test.test_io.test_largefile.PyLargeFileTest.test_large_read) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
ok
test_lseek (test.test_io.test_largefile.PyLargeFileTest.test_lseek) ... ok
test_osstat (test.test_io.test_largefile.PyLargeFileTest.test_osstat) ... ok
test_seek_read (test.test_io.test_largefile.PyLargeFileTest.test_seek_read) ... ok
test_seek_readall (test.test_io.test_largefile.PyLargeFileTest.test_seek_readall) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
 ... process data size: 2.3G
ok
test_seekable (test.test_io.test_largefile.PyLargeFileTest.test_seekable) ... ok
test_truncate (test.test_io.test_largefile.PyLargeFileTest.test_truncate) ... ok
test_it (test.test_io.test_largefile.TestCopyfile.test_it) ... ok
test_it (test.test_io.test_largefile.TestSocketSendfile.test_it) ... ok

----------------------------------------------------------------------
Ran 16 tests in 6.950s

OK
0:00:06 load avg: 9.72 [1/1] test_io.test_largefile passed

== Tests result: SUCCESS ==

1 test OK.

Total duration: 7.0 sec
Total tests: run=16
Total test files: run=1/1
Result: SUCCESS

For reference; this test goes from 8.5sec overall duration on my AMD 64 bit Arch Linux box to 7 seconds.

pythongh-129005: Remove copy using take_bytes from _pyio
d8b8849 
Memory usage now matches that of _io for large files
bedevere-app Bot mentioned this pull request Nov 13, 2025
Closed

cmaloney commented Nov 14, 2025

Copy link
Copy Markdown
Contributor Author

Optimization enabled by gh-139871 (Trying to keep that one issue referencing all the things which I use it for; planning to expand the What's New with a single "here's all the modules that got faster" list once optimize)

cmaloney changed the title gh-129005: Remove copy from _pyio using take_bytes gh-129005: Remove copies from _pyio using take_bytes Nov 14, 2025
cmaloney mentioned this pull request Nov 15, 2025
Merged
vstinner reviewed Nov 17, 2025
View reviewed changes
Comment thread Lib/_pyio.py
@@ -620,15 +620,15 @@ def read(self, size=-1):
if n < 0 or n > len(b):
raise ValueError(f"readinto returned {n} outside buffer size {len(b)}")
del b[n:]

vstinner Nov 17, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Is there a difference between b.resize(n) and del b[n:]?

cmaloney Nov 18, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

.resize() could grow while the del b[n:] never will. There's some code which does a = bytearray(b'12'); del a[3:] (intentionally del a slice past the end) as part of its buffer management. Generally I prefer resize because there's a lot less code involved than the slice code (and the slice code eventually calls resize anyways)

vstinner approved these changes Nov 17, 2025
View reviewed changes

vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM

vstinner merged commit 58f3fe0 into python:main Nov 18, 2025
48 checks passed
cmaloney deleted the gh129005_take_bytes branch November 18, 2025 22:10
StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
pythongh-129005: Remove copies from _pyio using take_bytes (python#14…
4b4776a 
…1539)

Memory usage now matches that of _io for large files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

vstinner vstinner approved these changes

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

Back | FazBrowse Home | New Git URL