FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-132983: Don't allow trailer data in ZstdFile by Rogdham · Pull Request #133736 · python/cpython · GitHub

/ cpython Public

gh-132983: Don't allow trailer data in ZstdFile - #133736

Merged
AA-Turner merged 1 commit into
python:mainfrom
Rogdham:zstdfile-trailer-exception
May 10, 2025
ConversationCommits (1)ChecksFiles changed
Merged

gh-132983: Don't allow trailer data in ZstdFile#133736
AA-Turner merged 1 commit into
python:mainfrom
Rogdham:zstdfile-trailer-exception

Conversation

Rogdham commented May 9, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

We previously made sure that an exception is raised when decompressing trailer data with decompress:

>>> from compression.zstd import compress, decompress
>>> invalid = compress(b'xxx') + b'yyy'
>>> decompress(invalid)
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    decompress(invalid)
    ~~~~~~~~~~^^^^^^^^^
  File "/redacted/Lib/compression/zstd/__init__.py", line 157, in decompress
    results.append(decomp.decompress(data))
                   ~~~~~~~~~~~~~~~~~^^^^^^
_zstd.ZstdError: Unable to decompress zstd data: Unknown frame descriptor

Indeed, the Zstandard specification says “Zstandard compressed data is made of one or more frames”, and it does not say that random data can be added at the end.

However, this is not the case in ZstdFile / zstd.open:

>>> from compression.zstd import ZstdFile
>>> from io import BytesIO
>>> ZstdFile(BytesIO(invalid)).read()
b'xxx'

After this PR, the last call becomes:

>>> ZstdFile(BytesIO(invalid)).read()
Traceback (most recent call last):
  File "<python-input-5>", line 1, in <module>
    ZstdFile(BytesIO(invalid)).read()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/redacted/Lib/compression/zstd/_zstdfile.py", line 176, in read
    return self._buffer.read(size)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/redacted/Lib/compression/_common/_streams.py", line 118, in readall
    while data := self.read(sys.maxsize):
                  ~~~~~~~~~^^^^^^^^^^^^^
  File "/redacted/Lib/compression/_common/_streams.py", line 91, in read
    data = self._decompressor.decompress(rawblock, size)
_zstd.ZstdError: Unable to decompress zstd data: Unknown frame descriptor

bedevere-app Bot mentioned this pull request May 9, 2025
Closed
13 tasks
Rogdham marked this pull request as ready for review May 9, 2025 09:36
AA-Turner added the needs backport to 3.14 bugs and security fixes label May 9, 2025

emmatyping commented May 9, 2025

Copy link
Copy Markdown
Member

The current behavior matches LZMA. I think unlike decompress which is handed what is necessarily a zstd stream of one or more frames, with ZstdFile, a user may be parsing a format which has additional information after a zstd stream.

>>> from lzma import LZMAFile, compress
>>> from io import BytesIO
>>> invalid = compress(b'foo') + b'bar'
>>> LZMAFile(BytesIO(invalid)).read()
b'foo'
>>>

Rogdham commented May 9, 2025

Copy link
Copy Markdown
Contributor Author

You are right this is the case for LZMAFile with format FORMAT_AUTO (which is the default) and also for BZ2File.

However, LZMAFile with format FORMAT_XZ as well as GzipFile raise an exception in that case.

>>> from lzma import LZMAFile, compress, FORMAT_XZ
>>> from io import BytesIO
>>> invalid = compress(b'foo') + b'bar'
>>> LZMAFile(BytesIO(invalid), format=FORMAT_XZ).read()
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    LZMAFile(BytesIO(invalid), format=FORMAT_XZ).read()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/redacted/lzma.py", line 208, in read
    return self._buffer.read(size)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/redacted/_compression.py", line 118, in readall
    while data := self.read(sys.maxsize):
                  ~~~~~~~~~^^^^^^^^^^^^^
  File "/redacted/_compression.py", line 99, in read
    raise EOFError("Compressed file ended before the "
                   "end-of-stream marker was reached")
EOFError: Compressed file ended before the end-of-stream marker was reached

emmatyping approved these changes May 9, 2025
View reviewed changes

emmatyping left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Okay this looks good then!

Rogdham commented May 9, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

In addition, consider decompress(compress(b"xxx") + b"yyy"):

  • returns b"xxx" on: lzma (format FORMAT_AUTO), bz2
  • raises an exception on: lzma (format FORMAT_XZ), gzip

Since for zstd we raise an exception on that, I would say to do the same for ZstdFile to be consistent.

AA-Turner merged commit 50b5370 into python:main May 10, 2025

miss-islington-app Bot commented May 10, 2025

Copy link
Copy Markdown

Thanks @Rogdham for the PR, and @AA-Turner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 10, 2025
pythongh-132983: Don't allow trailer data in ZstdFile (pythonGH-133736)
0c55b02 
(cherry picked from commit 50b5370)

Co-authored-by: Rogdham <3994389+Rogdham@users.noreply.github.com>

bedevere-app Bot commented May 10, 2025

Copy link
Copy Markdown

GH-133799 is a backport of this pull request to the 3.14 branch.

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label May 10, 2025
AA-Turner pushed a commit that referenced this pull request May 10, 2025
[3.14] gh-132983: Don't allow trailer data in ZstdFile (GH-133736) (#…
99ca086 
…133799)

gh-132983: Don't allow trailer data in ZstdFile (GH-133736)
(cherry picked from commit 50b5370)

Co-authored-by: Rogdham <3994389+Rogdham@users.noreply.github.com>
Rogdham deleted the zstdfile-trailer-exception branch May 10, 2025 06:45
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
pythongh-132983: Don't allow trailer data in ZstdFile (python#133736)
3882f69
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
pythongh-132983: Don't allow trailer data in ZstdFile (python#133736)
11bf99d
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

emmatyping emmatyping approved these changes

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

Back | FazBrowse Home | New Git URL