| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
….request Add urllib.request.HTTPGzipHandler, an opt-in handler that requests gzip-compressed responses and transparently decodes them. The handler is not one of the default build_opener() handlers, so urlopen()'s behaviour is unchanged unless it is added explicitly. When the request has no Accept-Encoding header, the handler adds "Accept-Encoding: gzip"; if the response then carries "Content-Encoding: gzip", the body is decoded incrementally as it is read and the Content-Encoding and Content-Length headers, which describe the compressed body, are removed. A request that already carries an Accept-Encoding header is left untouched, so a caller that means to handle the compressed response itself still receives it. Only gzip is handled, and the zlib module is required.
Documentation build overview3 files changed ± library/urllib.request.html ± whatsnew/3.16.html ± whatsnew/changelog.html |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
This adds urllib.request.HTTPGzipHandler, an opt-in handler that sends Accept-Encoding: gzip and transparently decodes a Content-Encoding: gzip response body. It's the long-standing request in this issue (originally bpo-1508475). Ref #43521.
Design — the thing that kept this open for 20 years wasn't code, it was one unmade design call: an opt-in handler versus default-on decoding down at http.client. This PR is a concrete answer to that call, not a claim on it — the call stays the maintainers'. I went opt-in: the handler is not added to build_opener()'s default handler list, so urlopen() behaves identically for everyone who doesn't ask for it; you get it with build_opener(HTTPGzipHandler). I chose opt-in over on-by-default because the strongest unanswered objection in the thread was Martin Panter's (2014): transparent decoding at this layer would silently decompress a deliberately-gzipped download such as a .tar.gz, and would break partial Range requests. Making the decode the caller's explicit choice sidesteps that. Senthil Kumaran (2012) preferred on-by-default with a flag to switch it off — that's a reasonable alternative; I can flip the default to that if you'd rather — the mechanism is identical either way.
No double-decode: the handler decodes only when it added the Accept-Encoding header itself. If the request already carries a caller-supplied Accept-Encoding, the response is returned untouched — this answers the double-decode objection from Serhiy Storchaka (2012) and Martin Panter (2014), whose own code already decodes gzip from urlopen(). When it does decode, it removes Content-Encoding and Content-Length, which describe the compressed body (Serhiy Storchaka and Senthil Kumaran, 2012, matching RFC guidance).
Streaming: the body is decompressed incrementally with zlib.decompressobj(16 + zlib.MAX_WBITS) as the caller reads, so a read() with no size limit can't bypass the decode (Martin Panter, 2014) and there's no full-buffer decompression bomb — memory is bounded per read. I deliberately did not reuse xmlrpc.client's gzip helper (Martin Panter, 2015): it reads the whole body into a BytesIO and caps it at 20 MiB, which suits XML-RPC payloads but not arbitrary-size urllib downloads. A response made of several concatenated gzip members is decoded in full — not just the first — and trailing bytes that aren't another member are ignored, matching the reader's existing leniency. A truncated gzip stream raises EOFError; a truncated transfer still surfaces http.client.IncompleteRead.
Scope: gzip only. I left deflate out on purpose — its raw-vs-zlib-wrapper ambiguity is a known interop trap, so requesting it is easy but decoding it reliably across servers isn't; it can be a follow-up. zlib is required, and constructing the handler without it raises ImportError, which for an opt-in handler is clearer than a silent no-op.
Tests cover opt-in-ness (the handler is not in build_opener()'s defaults but is accepted explicitly), header injection versus non-override, round-trip decode, chunked/streaming reads, Content-Encoding/Content-Length removal, full pass-through when the caller set their own Accept-Encoding, non-gzip pass-through, concatenated-member decode, trailing-junk tolerance, and the truncated-stream error. test_urllib2, test_urllib and test_urllib2_localnet pass. Docs, a What's New entry and a NEWS blurb are included.