FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

bpo-43613: Faster implementation of gzip.compress and gzip.decompress by rhpvorderman · Pull Request #27941 · python/cpython · GitHub

/ cpython Public

bpo-43613: Faster implementation of gzip.compress and gzip.decompress - #27941

Merged
ambv merged 22 commits into
python:mainfrom
rhpvorderman:bpo-43613
Sep 2, 2021
ConversationCommits (22)ChecksFiles changed
Merged

bpo-43613: Faster implementation of gzip.compress and gzip.decompress#27941
ambv merged 22 commits into
python:mainfrom
rhpvorderman:bpo-43613

Conversation

rhpvorderman commented Aug 25, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor

This also includes the changes from #25011 for bpo-43612. These make more sense in the context of these changes.

Currently gzip.compress and gzip.decompress are implemented with GzipFile. This is a lot of overhead when a simple in memory compression is needed. As shown in the benchmarks below, the overhead is considerable for datasizes below 4096 bytes (which are probably very common targets for in memory compression and decompression).

This PR changes the implementations to compress and decompress in memory.

I compiled python before and after this change with --enable-optimizations to ensure a fair comparison.
I used this script to benchmark:

import gzip
import pathlib
import statistics
import sys
import timeit

DATA=pathlib.Path(sys.argv[1]).read_bytes()

SIZES = [0, 128, 512, 1024, 4096, 8192, 16384]

def benchmark(bench_string, number=1000, repetitions=10):
    for size in SIZES:
        data = DATA[:size]
        compressed_data = gzip.compress(data)
        timeit_kwargs=dict(globals=dict(**locals(), **globals()),
                           number=number)
        results = [timeit.timeit(bench_string, **timeit_kwargs) for _ in range(repetitions)]
        average = statistics.mean(results)
        print(f"Data size {size}: {round(average * (1_000_000 / number),2)} microseconds average")

if __name__ == "__main__":
    print("gzip compression")
    benchmark("gzip.compress(compressed_data)")
    print()
    print("gzip decompression")
    benchmark("gzip.decompress(compressed_data)")

Before:

gzip compression
Data size 0: 7.92 microseconds average
Data size 128: 12.1 microseconds average
Data size 512: 18.45 microseconds average
Data size 1024: 22.41 microseconds average
Data size 4096: 32.51 microseconds average
Data size 8192: 41.03 microseconds average
Data size 16384: 57.99 microseconds average

gzip decompression
Data size 0: 8.99 microseconds average
Data size 128: 10.26 microseconds average
Data size 512: 12.62 microseconds average
Data size 1024: 13.55 microseconds average
Data size 4096: 21.12 microseconds average
Data size 8192: 30.59 microseconds average
Data size 16384: 61.24 microseconds average

After:

gzip compression
Data size 0: 3.68 microseconds average
Data size 128: 7.64 microseconds average
Data size 512: 14.06 microseconds average
Data size 1024: 17.42 microseconds average
Data size 4096: 27.25 microseconds average
Data size 8192: 37.09 microseconds average
Data size 16384: 53.48 microseconds average

gzip decompression
Data size 0: 1.98 microseconds average
Data size 128: 3.74 microseconds average
Data size 512: 5.36 microseconds average
Data size 1024: 6.72 microseconds average
Data size 4096: 14.1 microseconds average
Data size 8192: 23.57 microseconds average
Data size 16384: 52.72 microseconds average

https://bugs.python.org/issue43613

Add test for zlib.compress wbits
608876c
Add wbits argument to zlib.compress
12fb5a5
Use clinic to generate input
1b34510
Update documentation for zlib
4e90311
Add blurb news entry for zlib.compress wbits parameter
9717252
Fix doc typo
d70390e
Remove unnecessary whitespace, add punctionation and complete sentences.
c819e3e
Break line to comply with PEP-7
1f3481f
Update blurb to include :func: reference
8019932
Remove erroneous double backticks
0ea98cf
Faster gzip.compress implementation
d1c86dc
More efficiently decompress gzip files in memory
19a0358
Ensure correct endianness
5155857
Remove redundant line
fa188a6
Fix typos and test errors
4e76cf5
Revert changing default on compress for backwards compatibility
0280c95
Update documentation with gzip speed improvements
8ddee29
Add a blurb for gzip speed improvements
77f79fd
Use + instead of bytes.join() method
ca3e543
rhpvorderman mentioned this pull request Aug 25, 2021
Closed
ambv reviewed Aug 25, 2021
View reviewed changes
Comment thread Lib/gzip.py Outdated
ambv reviewed Aug 25, 2021
View reviewed changes
Comment thread Lib/test/test_zlib.py Outdated
ambv reviewed Aug 25, 2021
View reviewed changes
Comment thread Lib/gzip.py Outdated
rhpvorderman and others added 3 commits August 30, 2021 14:23
Reword docstring for read_gzip_header
f881f7e 
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Update docstring for gzip.compress
97a8100
Use subtest for zlib.compress/decompress test.
eeb7766 
Co-authored-by: Łukasz Langa <lukasz@langa.pl>

rhpvorderman commented Aug 30, 2021

Copy link
Copy Markdown
Contributor Author

Thanks for your review and suggestions @ambv! I updated the code.

ambv merged commit ea23e78 into python:main Sep 2, 2021
This was referenced Oct 18, 2021
Closed
Merged
rhpvorderman deleted the bpo-43613 branch November 19, 2021 09:30
rhpvorderman mentioned this pull request Oct 17, 2022
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

ambv ambv left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

Back | FazBrowse Home | New Git URL