Fix HTTP 451 DMCA and 403 TOS handling regression (#487) - #488
Merged
josegonzalez merged 1 commit intoFeb 16, 2026
Merged
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
The DMCA handling added in PR josegonzalez#454 had a bug: make_request_with_retry() raises HTTPError before retrieve_data() could check the status code via getcode(), making the case 451 handler dead code. This also affected HTTP 403 TOS violations (e.g. jumoog/MagiskOnWSA). Fix by catching HTTPError in retrieve_data() and converting 451 and blocked 403 responses (identified by "block" key in response body) to RepositoryUnavailableError. Non-block 403s (permissions, scopes) still propagate as HTTPError. Also handle RepositoryUnavailableError in retrieve_repositories() for the --repository case. Rewrote tests to mock urlopen (not make_request_with_retry) to exercise the real code path that was previously untested. Closes josegonzalez#487
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes issues raised in #487
Found a bug that made the HTTP 451 DMCA handling from #454 dead code. make_request_with_retry() raises HTTPError before retrieve_data() ever checks the status code via getcode(), so the case 451 handler was never reached. The same issue affected HTTP 403 TOS-blocked repositories.
Fixed by catching HTTPError around the make_request_with_retry() call in retrieve_data() and converting HTTP 451 and TOS-block 403s (distinguished by the "block" key in the response body) into RepositoryUnavailableError. Also added a catch in retrieve_repositories() for the single-repo fetch path (--repository flag), which returns [] with a warning instead of crashing.
Improved by narrowing the 403 handler to only convert responses with a "block" key (TOS violations), leaving permission-denied and rate-limit 403s unchanged. Renamed dmca_url to legal_url throughout since the attribute now covers both DMCA and TOS cases. Removed the dead match/case statement and replaced it with a guard assertion.
Confirmed by updating tests in test_http_451.py to mock urlopen (not make_request_with_retry) so they exercise the real error path, and adding tests for 403 TOS blocks, non-block 403 pass-through, rate-limit 403 retry behavior, and retrieve_repositories graceful handling. Also ran live tests against joskwanten/msx-picoverse (DMCA 451), jumoog/MagiskOnWSA (TOS 403), iamrodos/Emu (normal private repo), full joskwanten user backup, and a non-existent repo (404) to confirm all scenarios work correctly.