FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-79638: Treat an unreachable robots.txt as "disallow all" by serhiy-storchaka · Pull Request #138555 · python/cpython · GitHub

  1. python
  2. cpython

Repository navigation

gh-79638: Treat an unreachable robots.txt as "disallow all" - #138555

Merged
serhiy-storchaka merged 5 commits into
python:mainfrom
serhiy-storchaka:robotparser-unreachable
May 7, 2026
ConversationCommits (5)ChecksFiles changed
Merged

gh-79638: Treat an unreachable robots.txt as "disallow all"#138555
serhiy-storchaka merged 5 commits into
python:mainfrom
serhiy-storchaka:robotparser-unreachable

Conversation

serhiy-storchaka commented Sep 5, 2025 •
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Member

Disallow all access in urllib.robotparser if the robots.txt file is unreachable due to server or network errors.

serhiy-storchaka added needs backport to 3.13 only security fixes needs backport to 3.14 bugs and security fixes labels Sep 5, 2025
bedevere-app Bot mentioned this pull request Sep 5, 2025
Closed
pythongh-79638: Treat an unreachable robots.txt as "disallow all"
317c116 
Disallow all access in urllib.robotparser if the robots.txt file is
unreachable due to server or network errors.

github-actions Bot commented Apr 29, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 29, 2026
serhiy-storchaka enabled auto-merge (squash) May 4, 2026 19:45
serhiy-storchaka disabled auto-merge May 4, 2026 19:49
serhiy-storchaka enabled auto-merge (squash) May 5, 2026 09:09
serhiy-storchaka merged commit 310fe88 into python:main May 7, 2026
49 of 52 checks passed

miss-islington-app Bot commented May 7, 2026

Copy link
Copy Markdown

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

bedevere-app Bot commented May 7, 2026

Copy link
Copy Markdown

GH-149512 is a backport of this pull request to the 3.14 branch.

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label May 7, 2026

bedevere-app Bot commented May 7, 2026

Copy link
Copy Markdown

GH-149513 is a backport of this pull request to the 3.13 branch.

bedevere-app Bot removed the needs backport to 3.13 only security fixes label May 7, 2026
hugovk added a commit to hugovk/cpython that referenced this pull request May 7, 2026
Revert "pythongh-79638: Treat an unreachable robots.txt as "disallow …
fdf75b3 
…all" (pythonGH-138555)"

This reverts commit 310fe88.

hugovk commented May 7, 2026

Copy link
Copy Markdown
Member

This had been set to automerge but was failing CI, so wasn't meant to merge right now.

I'd temporarily removed branch protection to be able to merge #149500 which was needed to fix configure, which meant this PR also merged.

I've reverted it in #149514 to get main back to green. Sorry about this.

@serhiy-storchaka Would you like me to re-open a fresh version of this PR once that is sorted?

serhiy-storchaka commented May 7, 2026

Copy link
Copy Markdown
Member Author

I'll recreate it tomorrow.

serhiy-storchaka deleted the robotparser-unreachable branch May 8, 2026 08:24

serhiy-storchaka commented May 8, 2026

Copy link
Copy Markdown
Member Author

Restored as #149541.

serhiy-storchaka mentioned this pull request Jun 28, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

berkerpeksag Awaiting requested review from berkerpeksag

Assignees

No one assigned

Labels

stale Stale PR or inactive for long period of time.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

Back | FazBrowse Home | New Git URL