Repository navigation
Export zlib.{adler,crc}32_combine #134635
Description
Feature or enhancement
Proposal:
This is already provided by zlib, and is useful for computing a crc32 in multiple chunks (not linearly).
What I currently do is use ctypes, but that doesn't work on all environments (like Windows). I don't think we'll need a pure python fallback, just a wrapper of what's in zlib, equivalent of https://github.com/fastzip/fastzip/blob/9019107a6732fab9004e625b075faab96f29265a/fastzip/_crc32_combine.py#L63-L67
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response
Linked PRs
Metadata
Metadata
Assignees
Projects
- Status
Activity
picnixz commented on May 24, 2025
cc @emmatyping (or is zstd not supporting this one? I don't know if you now want to take care of compression issues as well in general)
emmatyping commented on May 24, 2025
@picnixz this is zlib related, not zstandard, but I am hoping to take care of more compression issues going forward!
emmatyping commented on May 24, 2025
I think this request is reasonable, we probably would want to also add adler32_combine for completeness/symmetry.
I don't think we should add crc32_combine_gen/crc32_combine_op without additional motivation.
picnixz commented on May 24, 2025
Yeah, I actually misread the issue first :')
Great to hear!
Yes, sure. Note that we could also think about whether to add binascii.crc32_combine as well (we have a minimal support for CRC32 even without zlib in binascii but I don't know if it would also make sense to expose crc32_combine in this case). I wouldn't recommend it as a first iteration (and we anyway don't have binascii.adler32)
emmatyping commented on May 24, 2025
I should note that if one is merely iterating over chunks of data, using the value argument to zlib.crc32 (i.e. setting an initial checksum) should work to compute a running checksum.
However, looking at the fastzip code, they are concurrently calculating the checksums and assembling them afterwards, so they cannot chain calls to zlib.crc32.
picnixz commented on May 25, 2025
Actually, there was #82868 and #17048 for adler32_combine but it was not tested. I'll close the older issue and will add both as part of this specific issue.
gh-134635: add `zlib.{adler32,crc32}_combine` to combine checksums (#…
pythongh-134635: add `zlib.{adler32,crc32}_combine` to combine checks…
pythongh-134635: add `zlib.{adler32,crc32}_combine` to combine checks…
pythongh-134635: add `zlib.{adler32,crc32}_combine` to combine checks…