fix(commit): parse the encoding header when deserializing - #2286
Merged
Byron merged 2 commits intoOct 15, 2026
Merged
Byron merged 2 commits into
Byron merged 2 commits into
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
`Commit._deserialize` compared a 10-byte slice of each stripped header line against the 9-byte literal `b"encoding "`, so the comparison never matched and the branch reading a commit's declared encoding was unreachable. Every commit parsed from the object database was decoded with the default UTF-8: `Commit.encoding` reported `UTF-8` regardless of the header, and the author, committer, and message of a commit written under `i18n.commitencoding` came back with U+FFFD replacement characters while `git log` shows them correctly. Re-serializing such a commit also dropped its `encoding` header and stored the garbled text. Compare the right slice length so the header is read. Adopt the declared encoding only when Python has a text codec for it: the identity lines are decoded with `errors="replace"`, which does not guard against an unknown codec name, so a commit declaring one would otherwise raise `LookupError` on attribute access. An unknown name keeps the previous behavior (the default encoding) and logs the error. Validated against a repository committed under `i18n.commitencoding = ISO-8859-1`: `commit.encoding`, `commit.author.name`, and `commit.message` now match what `git log` shows. Adds regression tests in `test/test_commit.py` that fail before the change.
I didn't try to fix all the issues with this, let's just make it a little bit better hoping the code won't be around much longer. <!-- agent --> # Known limitations Commit.encoding still conflates the declared encoding with the codec used for best-effort decoding. Unsupported or malformed declarations leave the default encoding—or an earlier accepted declaration—in place, so the resulting attribute does not faithfully represent the commit’s original metadata. The rejected header bytes appear in the diagnostic but are not retained on the object. Fallback decoding with "replace" can irreversibly alter identity and message content. Reserializing such a commit can therefore change both its encoding declaration and its content; this change does not provide lossless round-tripping. Codec validation also checks only that Python accepts the codec, not that it can decode the actual commit content. Assisted-by: GPT 6.1 Sol Co-authored-by: GPT 6.1 Sol <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Commit._deserialize tests each header line with buf[0:10] == b"encoding ". The slice is 10 bytes, the literal is 9, and buf is stripped, so the comparison never matches and the branch that reads a commit's declared encoding is unreachable: every commit read from the object database is decoded with the default UTF-8. For a repository committed under i18n.commitencoding = ISO-8859-1:
so the identity and message GitPython reports for a commit differ from what git shows for the same object, and re-serializing such a commit drops its encoding header and stores the garbled text. Commit headers come from whoever authored the commit, including commits fetched from untrusted repositories.
Fixing the slice alone would introduce a crash: the identity lines decode with errors="replace", which does not guard against an unknown codec name, so a commit declaring encoding pony-speak would raise LookupError from any attribute access. The declared encoding is adopted only when Python has a text codec for it, probed with a one-byte decode because decoding b"" skips the codec lookup; an unknown name keeps the previous default-encoding behavior and logs the error. Adds regression tests for both cases in test/test_commit.py; the first fails before the change.