| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
…rotocol#3146) Since CPython 3.12 the C json scanner guards recursion by remaining C-stack headroom rather than a fixed depth, so whether a 100k-deep body raises RecursionError (-> PARSE_ERROR) or parses into a giant list that fails request validation (-> INVALID_REQUEST) depends on the thread's stack size. macOS threads run with a smaller default stack, which is why the test failed there while passing on Ubuntu and Windows. Split the test in two: the nested-body test now asserts a 400 with either rejection code (both are correct; neither is a crash), and a new monkeypatch-based test pins the RecursionError -> PARSE_ERROR mapping deterministically on every platform. Fixes modelcontextprotocol#3146
There was a problem hiding this comment.
Not a maintainer, just a contributor who went looking for the mechanism here because the reasoning was interesting. The fix looks right to me, and the monkeypatched test is the part I would keep regardless: relaxing the assertion alone would have quietly dropped all coverage of the RecursionError to PARSE_ERROR mapping, and pinning it separately avoids that. Two corrections to the docstring, both from measurements rather than reading.
I measured json.loads("[" * N + "]" * N) on Linux x86_64, each run in its own subprocess so a hard crash is observable, with thread stack sizes set via threading.stack_size().
| Python | main thread | 1 MB | 8 MB | 64 MB | 256 MB | tracks stack? |
|---|---|---|---|---|---|---|
| 3.11.16 | RecursionError from depth 1,000 | same | same | same | same | no |
| 3.12.3 | parses to 5,000, raises from 10,000 | SIGSEGV | same as main | same as main | same as main | no |
| 3.13.15 | parses to 5,000, raises from 10,000 | SIGSEGV | same as main | same as main | same as main | no |
| 3.14.7 | parses to 50,000, raises from 75,000 | raises from 10,000 | flips ~64,901 | parses past 400,000 | parses past 400,000 | yes |
On 3.12 and 3.13 the flip depth is identical at 8 MB, 64 MB and 256 MB, so the guard there ignores the stack completely. Only 3.14 moves with it, and on 3.14 it is cleanly linear: roughly 8,100 levels per MB, about 129 bytes of C stack per level. Depth 100,000 needs somewhere between 12 MB and 16 MB on this platform.
This matters a little beyond wording, because the CI matrix runs 3.10 through 3.14. On everything below 3.14 the 100,000-deep body always raises and the response is always PARSE_ERROR, deterministically.
The docstring attributes the macOS result to runner threads getting a smaller default stack. Following that through: a smaller stack means less headroom, so json.loads raises sooner, which gives PARSE_ERROR. That is what the original test asserted, so it would have passed.
The failure in #3146 was INVALID_REQUEST, which means the body parsed. That needs more headroom than 100,000 levels requires, not less.
My guess is that the per-level C-stack cost differs by architecture, and arm64 macOS fits 100,000 levels in a stack where x86_64 Linux needs about 16 MB. I cannot confirm that, since I only have Linux to measure on, so please treat it as a hypothesis rather than a correction to yours. Either way the conclusion is the same, and it is the one your fix already encodes: which of the two codes you get is not something the test should pin.
Something like this, if the measurements above hold up on your side:
Which JSON-RPC code it gets is platform-dependent. From CPython 3.14 the C json scanner guards recursion by actual C-stack headroom rather than a fixed limit, so whether a 100k-deep body parses depends on how much stack the parsing thread has and on the per-level cost on that architecture. It either fails to parse (RecursionError, so PARSE_ERROR) or parses into a giant list that then fails request validation (INVALID_REQUEST). Both are correct rejections; the deterministic PARSE_ERROR mapping is covered by the monkeypatch test below.
Since the outcome is deterministic below 3.14, you could keep the stronger assertion where it still holds:
if sys.version_info >= (3, 14):
assert response.json()["error"]["code"] in (PARSE_ERROR, INVALID_REQUEST)
else:
assert response.json()["error"]["code"] == PARSE_ERRORThat keeps four of the five supported versions asserting the exact code. It also adds a version branch to a test, which is its own kind of cost, so I would understand leaving it as it is.
On 3.12 and 3.13, a thread with a 1 MB stack segfaults parsing a deeply nested body rather than raising, because the fixed limit does not account for the actual stack available. That is CPython's, not this PR's, and the server is unlikely to parse on such a thread. Mentioning it only because the test being replaced was named ..._not_a_crash, so it seemed worth someone knowing.
Disclosure: I used AI assistance for the measurements and for writing this up. I ran them myself and can go through any of it.
Sorry, something went wrong.
…RROR below 3.14 Review measurements on modelcontextprotocol#3147 (and re-measured here on arm64 macOS) show the stack-headroom-based recursion guard in the C json scanner starts in CPython 3.14, not 3.12 - on 3.12/3.13 the flip depth is identical across 8/16/64 MB stacks. Docstring updated accordingly. The macOS mechanism was also backwards in the old docstring: it is not a smaller thread stack but a larger one. CPython pins non-main threads to a 16 MiB stack on macOS (THREAD_STACK_SIZE in Python/thread_pthread.h); measured default-thread flip on arm64 macOS 3.14 is ~149,641 levels - identical to an explicit 16 MiB stack - so the 100k-deep body parses and fails validation (INVALID_REQUEST) instead of raising RecursionError. Since the outcome is deterministic below 3.14, the test now asserts the exact PARSE_ERROR code there and accepts either rejection only on 3.14+.
|
Thanks @opensource-joe — this is exactly the kind of review I hoped the writeup would attract, and you're right on both counts. I re-measured on arm64 macOS (each probe in its own subprocess, threading.stack_size() set explicitly, binary-searched flip depths):
1. Confirmed: the headroom guard starts in 3.14, not 3.12. On 3.12/3.13 the flip depth is identical at 8/16/64 MB here too — the guard ignores the stack. On 3.14 it's linear at ~112 bytes/level on arm64 (vs your ~129 on x86_64). I also reproduce your 1 MB SIGSEGV on 3.12/3.13. 2. Your "points the wrong way" catch is correct, and the actual mechanism turned out to be neither of our guesses. A default-stack thread on macOS flips at ~149,641 — exactly the 16 MB row. CPython pins non-main threads to a 16 MiB stack on macOS (THREAD_STACK_SIZE in Python/thread_pthread.h), which is larger than typical elsewhere, not smaller. So on macOS the 100k body fits, parses, and fails validation → INVALID_REQUEST, while a platform whose parsing thread has ≲12 MB raises → PARSE_ERROR. Per-level cost differing by arch (your hypothesis) is real but secondary; the 2× thread-stack difference is the main driver. I've pushed 1956ff6 adopting both suggestions: docstring rewritten with the corrected version boundary and mechanism, and the version-gated assertion — since the outcome is deterministic below 3.14, the test now pins the exact PARSE_ERROR there and accepts either rejection only on 3.14+. Full file 52/52 on 3.13 and 3.14, ruff + pyright clean. |
Sorry, something went wrong.
…h coverage The prior if/else on sys.version_info created a branch where only one side executes per CI job (each job pins a single Python version), so coverage's --cov-branch --cov-fail-under=100 flags the always-taken branch as partial on every job in the matrix (all failed on this PR's last CI run). Collapsing to a single assert against a version-gated tuple keeps the exact same behavioral pinning (PARSE_ERROR only below 3.14, either code on 3.14+) without introducing an untestable branch. Verified locally on Python 3.10 (lowest supported): full suite passes (5444 passed, 9 skipped, 1 xfailed), coverage report shows 100.00% total including this file, ruff check and ruff format --check clean. <sub>AI Disclosure: https://gist.github.com/maxisbey/6123d132484e4c533eab519a2800693d</sub>
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Sorry, something went wrong.
The version-gated ternary at line 1033 left one branch of the sys.version_info >= (3, 14) check uncovered on any single interpreter, tripping the repo's fail_under=100 branch-coverage gate in CI (coverage.py measures conditional-expression branches the same as if/else). Converted to an explicit if/else with the existing '# pragma: lax no cover' convention used elsewhere in the test suite for platform/version-gated lines (e.g. tests/examples/conftest.py:33). Verified locally: pytest -k deeply_nested_body passes, and 'coverage run/report --include=*test_streamable_http_modern.py' now reports 100.00% branch coverage for the file (was failing before this change on this interpreter, since only one arm of the ternary executes per Python version). AI-assisted: change was scoped and verified by an AI coding agent (Claude) under human supervision; local repro and coverage numbers above are from an actual run, not simulated.
| Back | FazBrowse Home | New Git URL |
Fixes #3146.
Root cause
The test assumed a 100k-deep body always makes json.loads raise RecursionError. Since CPython 3.12 the C json scanner guards recursion by remaining C-stack headroom, not a fixed depth — so the outcome depends on the thread's stack size. Measured on Apple Silicon macOS, CPython 3.14:
macOS runner threads get a smaller default stack than the main thread, which is why macos-latest sees -32600 while Ubuntu/Windows see -32700. Both are correct 400 rejections of the same hostile body; the old assertion was pinning a platform accident.
Change (test-only)
Verified locally on macOS arm64: both tests pass on CPython 3.13 and 3.14, full test_streamable_http_modern.py 52/52, ruff + pyright clean.
AI Disclaimer