gh-153404: Silently ignore non-decimal digits in robots.txt Crawl-delay and Request-rate#153405
Conversation
…wl-delay and Request-rate str.isdigit() returns True for non-decimal Unicode digits such as U+00B2 SUPERSCRIPT TWO, which int() then rejects with ValueError. In RobotFileParser.parse() this raised a raw ValueError that aborted parsing of the entire robots.txt file, unlike every other malformed directive which is silently ignored. Guard the Crawl-delay and Request-rate conversions with str.isdecimal() so a value written with non-decimal digits is ignored instead of raising.
| Crawl-delay: ² | ||
| """ | ||
| good = ['/foo.html'] | ||
| bad = [] |
There was a problem hiding this comment.
I think you can consolidate test classes.
Add crawl_delay and request_rate.
There was a problem hiding this comment.
Done. Consolidated into one class with both directives.
| robots_txt = """\ | ||
| User-agent: * | ||
| Disallow: /tmp/ | ||
| Request-rate: ²/5 |
There was a problem hiding this comment.
Add also line with decimal numerator and non-decimal denominator.
There was a problem hiding this comment.
Added Request-rate: 5/² case.
serhiy-storchaka
left a comment
serhiy-storchaka
left a comment
There was a problem hiding this comment.
Add explicit crawl_delay = None and request_rate = None.
Please modify test_request_rate so that it asserts request_rate even if it is None.
|
Done. Added explicit |
serhiy-storchaka
added
needs backport to 3.13
|
Thanks @tonghuaroot for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15. |
|
GH-153830 is a backport of this pull request to the 3.15 branch. |
bedevere-app
Bot
removed
the
needs backport to 3.15
|
GH-153831 is a backport of this pull request to the 3.14 branch. |
|
GH-153832 is a backport of this pull request to the 3.13 branch. |
2 participants
str.isdigit()returns True for non-decimal Unicode digits such asU+00B2 SUPERSCRIPT TWO, whichint()then rejects withValueError. InRobotFileParser.parse()this raised a rawValueErrorthat aborted parsing of the entirerobots.txtfile, even though every other malformed directive is silently ignored (AllowandDisallowalready dotry/except ValueError: pass, and invalidCrawl-delay: pears/Request-rate: 9/bananavalues are already dropped, with existing tests).The
Crawl-delayandRequest-rateguards now usestr.isdecimal(), the predicate for the decimal digitsint()accepts, so a value written with non-decimal digits is ignored instead of raising.Scope note: a control character that
str.strip()removes butint()rejects (for example a byte inU+001C..U+001Finside aRequest-ratetoken) still raises, because theRequest-ratebranch converts the unstripped token. That is a separate, pre-existing divergence, left out of this single-purpose change.