Support Zip64 when compressing iterables and strings - #25
Open
arjan-s wants to merge 18 commits into
Open
Conversation
Copy link
Copy Markdown
Sorry, something went wrong.
I use this to flush partial zips as files are streamed into them from requests, and then at the end add manifest and error files to the end of the archive I've also added a related test and example of use
Copy link
Copy Markdown
|
Would it be possible to merge this? I ran into this issue because I need to stream very large amounts of data. |
Sorry, something went wrong.
Copy link
Copy Markdown
Author
|
Hi @matthewatabet, it seems the original developer of this package is not active anymore. So I've forked the package as zipstream-new: #33 |
Sorry, something went wrong.
Add partial flushing of ZipStreams
Find compression type.
When flushing, stream out the iterators in First in first out order. Python `pop()` with no arguments would take the last path but I think it makes sense to stream the first things first. We ran into this issue where we add a bunch of files which depend on long-running futures to provide the data. The futures hit a server which processes them roughly in order, so we get better streaming performance if we make this change.
Stream data in order it was received
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
May 30, 2021
Changes made after forking v1.1.4: v1.1.5 (2019-03-18) * Support Zip64 when compressing iterables and strings (allanlei/python-zipstream#25) v1.1.6 (2019-06-06) * Add partial flushing of ZipStreams (arjan-s/python-zipstream#1) v1.1.7 (2019-10-22) * Stream data in the order it was received (arjan-s/python-zipstream#4) v1.1.8 (2020-09-14) * New datetime parameter in write_iter (arjan-s/python-zipstream#8)
fix readme example
msk
pushed a commit
to msk/pkgsrc
that referenced
this pull request
May 11, 2026
Changes made after forking v1.1.4: v1.1.5 (2019-03-18) * Support Zip64 when compressing iterables and strings (allanlei/python-zipstream#25) v1.1.6 (2019-06-06) * Add partial flushing of ZipStreams (arjan-s/python-zipstream#1) v1.1.7 (2019-10-22) * Stream data in the order it was received (arjan-s/python-zipstream#4) v1.1.8 (2020-09-14) * New datetime parameter in write_iter (arjan-s/python-zipstream#8)
jperkin
pushed a commit
to TritonDataCenter/pkgsrc
that referenced
this pull request
May 14, 2026
Changes made after forking v1.1.4: v1.1.5 (2019-03-18) * Support Zip64 when compressing iterables and strings (allanlei/python-zipstream#25) v1.1.6 (2019-06-06) * Add partial flushing of ZipStreams (arjan-s/python-zipstream#1) v1.1.7 (2019-10-22) * Stream data in the order it was received (arjan-s/python-zipstream#4) v1.1.8 (2020-09-14) * New datetime parameter in write_iter (arjan-s/python-zipstream#8)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Zip64 support currently doesn't work when the inputs for the generated zip file are iterables and/or strings. The reason for this is that __write() assumes a file size of 0 bytes for those inputs. Because of the file size the module doesn't enable Zip64 support. When it detects during compression that the file size is larger than ZIP64_LIMIT, it (correctly) raises RuntimeError('File size has increased during compressing').
This patch adds the optional (and thus backwards compatible) argument buffer_size to write_iter and writestr. This allows programs using the module to specify the buffer size that will result from the iterable or string, and in turn that allows __write() to enable Zip64 support when necessary.