FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Add MySQL protocol compression with zlib and zstd by methane · Pull Request #1279 · PyMySQL/PyMySQL · GitHub

Repository navigation

Add MySQL protocol compression with zlib and zstd - #1279

Open
methane wants to merge 2 commits into
mainfrom
codex/mysql-protocol-compression
Open

methane wants to merge 2 commits into
mainfrom
codex/mysql-protocol-compression

Conversation

methane commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

PyMySQL currently rejects compress=True. This change enables MySQL protocol compression with zlib and zstd, including zstd on Python 3.9–3.13 through the optional PyMySQL[zstd] extra.

compress=True prefers zstd when available and supported by the server, then falls back to zlib or an uncompressed connection. compress="zlib" and compress="zstd" require the selected algorithm. Outgoing payloads below 400 bytes, or those that do not shrink, remain uncompressed within the compressed protocol.

The framing follows the MySQL protocol and the approach used by go-sql-driver/mysql: compression starts after authentication, supports packets split across frames and sequence wraparound, and handles large requests and responses. Decompression is bounded by the declared length and rejects corrupt or incomplete streams. Python 3.14+ uses compression.zstd; earlier versions use backports.zstd>=1.3.0, installed only when python_version < "3.14". CI installs the extra, and the README and changelog document the API.

Compressed-frame lengths use int.to_bytes(3, "little") so overflow is rejected, and the complete header and payload are assembled in one join. Decoder contexts remain independent per frame: the supported stateful decoder APIs stop at EOF and provide no reset operation. Sharing one global decoder would therefore fail on later frames even if calls were serialized. The standard-library one-shot zstd helper also constructs a new decoder internally and does not provide the bounded-output check used here. The code documents this constraint and the concurrent-stream tests verify isolation.

Validation:

  • Compression tests against MySQL 8.4.11: 90 passed on each of Python 3.9, 3.13, and 3.14, including TLS/non-TLS connections, reconnects, malformed frames, and request/response payloads above 16 MiB. Added deterministic 1/2/5/17-byte transport reads, MySQL headers and bodies split across compressed frames, normal continuation packets (including an empty terminator), the actual 24-bit frame limit with compressible and incompressible input, and independent decoding across eight concurrent threads.
  • Full suite with Python 3.13 and MySQL zstd connections: 272 passed, 12 skipped, 1 expected failure, plus 4 passing subtests; converter doctests: 4 passed.
  • Verified extra installation selects backports.zstd 1.3.0 on Python 3.9 and 1.7.0 on Python 3.13; Python 3.14 installs no backport and uses the standard library. The optional-backend tests also cover missing dependencies and zlib fallback.
  • Ruff formatting, Ruff lint, and git diff --check passed.

Summary by CodeRabbit

  • New Features
    • Added optional MySQL protocol compression using zlib or zstd. Automatic selection prefers zstd when available and supported, then zlib; connections remain uncompressed if the server supports neither.
    • Compression skips outgoing payloads smaller than 400 bytes and payloads that would not become smaller.
  • Documentation
    • Updated the installation guide and changelog with compression options, automatic selection behavior, and zstd setup details.

coderabbitai Bot commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info ⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 5d737124-0bf4-4532-a638-17ea84e3a008

📥 Commits

Reviewing files that changed from the base of the PR and between e303e22 and 2300c6b.

📒 Files selected for processing (2)
  • pymysql/_compression.py
  • pymysql/tests/test_compression.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

PyMySQL now supports MySQL protocol compression with zlib and optional zstd. The connection negotiates compression during authentication and uses compressed framing for later traffic. The change adds codec and connection tests, an optional zstd dependency, and installation and changelog documentation.

Changes

MySQL protocol compression

Layer / File(s) Summary
Compressed frame codec and backend support
pymysql/_compression.py, pymysql/tests/test_compression.py, pyproject.toml, .github/workflows/test.yaml
Adds compressed frame encoding and decoding with zlib or optional zstd. Tests cover frame limits, sequence handling, compression thresholds, malformed frames, and backend selection.
Compression options and handshake negotiation
pymysql/connections.py, pymysql/constants/CLIENT.py, pymysql/tests/test_compression.py, README.md, CHANGELOG.md
Adds compression option validation and zlib or zstd capability negotiation. Documents supported modes, fallback behavior, dependencies, and when payloads remain uncompressed.
Compressed connection I/O and lifecycle
pymysql/connections.py, pymysql/tests/test_compression.py
Starts compressed framing after authentication and routes later reads and writes through the compressed stream. Resets compression state for connection and command operations. Tests cover authentication, sequence resets, TLS, reconnects, and large packets.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Connection
  participant MySQLServer
  participant CompressedStream
  Connection->>MySQLServer: Negotiate zlib or zstd capability
  Connection->>MySQLServer: Complete authentication with normal framing
  Connection->>CompressedStream: Create stream after authentication
  Connection->>CompressedStream: Pass subsequent writes for framing
  CompressedStream->>MySQLServer: Send compressed protocol frames
  MySQLServer->>CompressedStream: Return compressed protocol frames
  CompressedStream->>Connection: Provide decoded packet bytes
Loading

Merge Risk: ⚪ Minimal · up to 2300c

No concrete merge-blocking issue was identified. Compression support appears ready to merge subject to normal checks.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 2300c

Compression preserves the inspected authentication and TLS ordering and rejects malformed streams with bounded output. The main remaining concern is connection reuse after an unexpected interruption during compressed decoding. Exposure is limited to connections using compression; no credential bypass or broader privilege gain was established.

Retained concerns

  • Low · reliability · inferred: Unexpected interruption during compressed-frame processing can escape without closing the connection. The codec consumes frame bytes before decoding and committing its buffer, but Connection._read_bytes closes only for InternalError. If a caller catches an interruption or unexpected decoding/allocation exception and reuses that connection, framing or response-consumption state may be incomplete. Expected malformed-stream errors and raw transport failures already close correctly; the concern is the newly added processing interval outside those safeguards.
Security review details

Security Blast Radius

  • inferred — The new decoding surface is reachable by the selected MySQL peer on connections negotiating compression. A peer controlling that stream can supply frame headers and compressed payloads to the client process. The inspected changes introduce no additional database privilege, tenant identity, or infrastructure authority; process-wide resource consequences were not fully characterized.

Security Findings and Attack Paths

  • inferred — Unexpected interruption after frame consumption creates a possible response-state desynchronization path if the application subsequently reuses the open connection. This is a failure-containment concern, not a demonstrated credential compromise or attacker-controlled privilege escalation.

Trust Boundaries and Controls

  • observed — TLS wrapping, when selected, precedes authentication traffic. Built-in authentication exchanges use normal framing, and compression activates after their terminal OK packet. Custom handlers retain the historical contract that a None return can indicate a completed exchange.
  • observed — Frame payload and declared output lengths are 24-bit fields. Decoding limits returned output to the declared length plus one and rejects incorrect lengths, incomplete streams, trailing data, and recognized decoder errors. These are output controls, not proof of a total native-memory or CPU bound.

Resilience and Maintainability Implications

  • observed — Recognized decoding errors, short raw reads, and transport failures close the connection and clear its compression state. This prevents normal malformed-frame rejection from leaving a reusable poisoned stream. Connection-level closure tests cover zlib; standalone malformed-frame tests are parameterized for both available algorithms.

Hardening Proposals

  • proposed — Make interruption or unexpected failure anywhere in a compressed-read transition terminal for that connection, matching the raw reader’s cleanup policy while preserving the original exception.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.35% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 63 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely identifies the main change: adding MySQL protocol compression with zlib and zstd.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1 📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

methane linked an issue Oct 1, 2026 that may be closed by this pull request
Comment thread pymysql/_compression.py Outdated


def _pack_int24(value):
return struct.pack("<I", value)[:3]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

value.to_bytes(3, "little") の方が速いしoverflow errorになって安全だからそっちにして。

Comment thread pymysql/_compression.py Outdated
Comment on lines +44 to +49
header = (
_pack_int24(len(payload))
+ bytes([self.sequence])
+ _pack_int24(original_length)
)
write_bytes(header + payload)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

header の一時オブジェクトが無駄だから一気に header + payload まで全部作って write_bytes して。

Comment thread pymysql/_compression.py
methane added this to the v1.3 milestone Oct 1, 2026

This branch has not been deployed

No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compression

1 participant


Back | FazBrowse Home | New Git URL