FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-117151: IO performance improvement, increase io.DEFAULT_BUFFER_SIZ… · python/cpython@8dbfe76 · GitHub

/ cpython Public

Commit 8dbfe76

Browse files
rmmancom
committed
gh-117151: IO performance improvement, increase io.DEFAULT_BUFFER_SIZE to 128k, adjust open() to use max(st_blksize, io.DEFAULT_BUFFER_SIZE)
1 parent 42351c3 commit 8dbfe76

4 files changed

Lines changed: 13 additions & 9 deletions

File tree

‎Doc/library/functions.rst‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1381,10 +1381,11 @@ are always available. They are listed here in alphabetical order.
13811381
:func:`io.TextIOWrapper.reconfigure`. When no *buffering* argument is
13821382
given, the default buffering policy works as follows:
13831383

1384-
* Binary files are buffered in fixed-size chunks; the size of the buffer is
1385-
chosen using a heuristic trying to determine the underlying device's "block
1386-
size" and falling back on :const:`io.DEFAULT_BUFFER_SIZE`. On many systems,
1387-
the buffer will typically be 4096 or 8192 bytes long.
1384+
* Binary files are buffered in fixed-size chunks; the size of the buffer
1385+
is set to `max(io.DEFAULT_BUFFER_SIZE, st_blksize)` using a heuristic
1386+
trying to determine the underlying device's "block size" when available
1387+
and falling back on :const:`io.DEFAULT_BUFFER_SIZE`.
1388+
On most systems, the buffer will typically be 131072 bytes long.
13881389

13891390
* "Interactive" text files (files for which :meth:`~io.IOBase.isatty`
13901391
returns ``True``) use line buffering. Other text files use the policy

‎Lib/_pyio.py‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,8 @@
2323
valid_seek_flags.add(os.SEEK_HOLE)
2424
valid_seek_flags.add(os.SEEK_DATA)
2525

26-
# open() uses st_blksize whenever we can
27-
DEFAULT_BUFFER_SIZE = 8 * 1024 # bytes
26+
# open() uses max(st_blksize, io.DEFAULT_BUFFER_SIZE) when st_blksize is available
27+
DEFAULT_BUFFER_SIZE = 128 * 1024 # bytes
2828

2929
# NOTE: Base classes defined here are registered with the "official" ABCs
3030
# defined in io.py. We don't use real inheritance though, because we don't want
@@ -248,8 +248,7 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
248248
except (OSError, AttributeError):
249249
pass
250250
else:
251-
if bs > 1:
252-
buffering = bs
251+
buffering = max(bs, DEFAULT_BUFFER_SIZE)
253252
if buffering < 0:
254253
raise ValueError("invalid buffering size")
255254
if buffering == 0:

‎Misc/NEWS.d/next/Core and Builtins/2024-04-30-14-03-09.gh-issue-117151.yt2H8c.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Increase ``io.DEFAULT_BUFFER_SIZE`` from 8k to 128k. Adjust :func:`open` on
2+
Linux to use ``max(io.DEFAULT_BUFFER_SIZE, device block size)`` rather than
3+
the device block size. Improve I/O write performance by up to 5 times.
4+
Patch by Romain Morotti.

‎Modules/_io/_iomodule.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ extern Py_ssize_t _PyIO_find_line_ending(
7878
*/
7979
extern int _PyIO_trap_eintr(void);
8080

81-
#define DEFAULT_BUFFER_SIZE (8 * 1024) /* bytes */
81+
#define DEFAULT_BUFFER_SIZE (128 * 1024) /* bytes */
8282

8383
/*
8484
* Offset type for positioning.

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL