Repository navigation
Commit 122541b
committed
Issue 21469: Mitigate risk of false positives with robotparser.
* Repair the broken link to norobots-rfc.txt.
* HTTP response codes >= 500 treated as a failed read rather than as a not
found. Not found means that we can assume the entire site is allowed. A 5xx
server error tells us nothing.
* A successful read() or parse() updates the mtime (which is defined to be "the
time the robots.txt file was last fetched").
* The can_fetch() method returns False unless we've had a read() with a 2xx or
4xx response. This avoids false positives in the case where a user calls
can_fetch() before calling read().
* I don't see any easy way to test this patch without hitting internet
resources that might change or without use of mock objects that wouldn't
provide must reassurance.1 parent 73308d6 commit 122541b
1 file changed
Lines changed: 9 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -7,7 +7,7 @@ | |||
| 7 | 7 | 2) PSF license for Python 2.2 | |
| 8 | 8 | ||
| 9 | 9 | The robots.txt Exclusion Protocol is implemented as specified in | |
| 10 | - http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html | ||
| 10 | + http://www.robotstxt.org/norobots-rfc.txt | ||
| 11 | 11 | """ | |
| 12 | 12 | ||
| 13 | 13 | import urllib.parse, urllib.request | |
@@ -57,7 +57,7 @@ def read(self): | |||
| 57 | 57 | except urllib.error.HTTPError as err: | |
| 58 | 58 | if err.code in (401, 403): | |
| 59 | 59 | self.disallow_all = True | |
| 60 | - elif err.code >= 400: | ||
| 60 | + elif err.code >= 400 and err.code < 500: | ||
| 61 | 61 | self.allow_all = True | |
| 62 | 62 | else: | |
| 63 | 63 | raw = f.read() | |
@@ -85,6 +85,7 @@ def parse(self, lines): | |||
| 85 | 85 | state = 0 | |
| 86 | 86 | entry = Entry() | |
| 87 | 87 | ||
| 88 | + self.modified() | ||
| 88 | 89 | for line in lines: | |
| 89 | 90 | if not line: | |
| 90 | 91 | if state == 1: | |
@@ -129,6 +130,12 @@ def can_fetch(self, useragent, url): | |||
| 129 | 130 | return False | |
| 130 | 131 | if self.allow_all: | |
| 131 | 132 | return True | |
| 133 | + # Until the robots.txt file has been read or found not | ||
| 134 | + # to exist, we must assume that no url is allowable. | ||
| 135 | + # This prevents false positives when a user erronenously | ||
| 136 | + # calls can_fetch() before calling read(). | ||
| 137 | + if not self.last_checked: | ||
| 138 | + return False | ||
| 132 | 139 | # search for given user agent matches | |
| 133 | 140 | # the first match counts | |
| 134 | 141 | parsed_url = urllib.parse.urlparse(urllib.parse.unquote(url)) | |
0 commit comments