| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
In NDJSON elements-file mode a split-PDF chunk that returned 200 with an empty body was logged and skipped, so the combined elements_file was short by those pages while the call still succeeded. `split_pdf_allow_failed` does not cover it -- an empty 200 counts as a successful chunk -- and `combine_chunk_files_to_ndjson` returns only a path, so nothing downstream could detect the truncation. Recombination now raises `EmptyChunkResponseError` (a `ValueError`) instead, matching the buffered path, which raises `JSONDecodeError` on the same response. An empty JSON array is still accepted: `[]` and an empty body are different responses on the wire and only the latter is a defect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit raised on every empty HTTP-200 chunk body, which is wrong for NDJSON: zero records is zero lines, so an empty body is a well-formed result and a split whose pages are blank would have failed. Recombination now takes each chunk's Content-Type. An empty body raises only when the chunk is not application/x-ndjson -- JSON has no empty document, so an empty JSON 200 stays a defect, and an unknown media type is read as JSON, which is what the deployed API returns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
No issues found across 6 files
Shadow auto-approve: would auto-approve. Bug fix: empty JSON split-PDF chunks now raise ValueError instead of silent truncation; empty NDJSON chunks remain valid. Parametrized tests pin the corrected behavior; no human-owned tradeoff remains.
Re-trigger cubic
Sorry, something went wrong.
|
LGTM! The splitting code was always so ugly because we had to work around the autogen code. I.e. we could only intercept the request, pull the file out, and hijack the outgoing payload. Now that that's all behind us, we can make this so much nicer. Now that I'm thinking about it let me make sure we have a ticket. |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
The bug
In NDJSON elements-file mode, a split-PDF chunk that returned HTTP 200 with an empty body was logged and skipped, so the combined elements_file was silently short by that chunk's pages while the call still returned 200.
Nothing downstream could detect it: combine_chunk_files_to_ndjson hands back only the combined path, and split_pdf_allow_failed=False does not cover the case because an empty 200 counts as a successful chunk. The buffered path fails outright on the same response (res.json() raises JSONDecodeError), so enabling NDJSON mode converted a hard failure into silent truncation.
The fix
Recombination now raises EmptyChunkResponseError (a ValueError, matching where the buffered path's JSONDecodeError lands) instead of skipping.
Emptiness is judged against the chunk's own Content-Type, because the two formats disagree about what an empty body means:
combine_chunk_files_to_ndjson takes an optional media_types list, positionally matched to chunk_paths; omitting it keeps the strict reading. _elements_from_task_responses collects each chunk's Content-Type before the cached branch overwrites the body with a temp-file path — the header survives both cache branches, so the cached case carries a real media type too.
Tests
_test_unstructured_client/unit/test_ndjson_elements_file.py:
Mutating _is_ndjson_media_type to return False fails 12 of these, including both hook-level cache-mode tests, so the coverage is load-bearing.
271 unit + 64 contract tests pass; pylint 10/10 on both changed modules.
Release
Bumped to 0.46.1 with CHANGELOG and RELEASES entries. Consumers that pin unstructured-client >=0.46.0 for NDJSON elements-file mode should raise the floor to >=0.46.1.
🤖 Generated with Claude Code