Repository navigation
Possible slowdown of regex searching in 3.11 #91404
Description
| BPO | 47248 |
|---|---|
| Nosy | @markshannon, @serhiy-storchaka, @animalize, @sweeneyde |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2022-04-07.11:30:33.473>
labels = ['3.11', 'performance']
title = 'Possible slowdown of regex searching in 3.11'
updated_at = <Date 2022-04-08.07:22:53.130>
user = 'https://github.com/markshannon'bugs.python.org fields:
activity = <Date 2022-04-08.07:22:53.130>
actor = 'malin'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2022-04-07.11:30:33.473>
creator = 'Mark.Shannon'
dependencies = []
files = []
hgrepos = []
issue_num = 47248
keywords = ['3.11regression']
message_count = 4.0
messages = ['416923', '416928', '416959', '416961']
nosy_count = 4.0
nosy_names = ['Mark.Shannon', 'serhiy.storchaka', 'malin', 'Dennis Sweeney']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue47248'
versions = ['Python 3.11']
Activity
markshannon commented on Apr 7, 2022
The 3 regular expression benchmarks in the pyperformance suite, regex_v8, regex_effbot and regex_dna show slowdowns between 3% and 10%.
Looking at the stats, nothing seems wrong with specialization or the memory optimizations.
Which strongly suggests a regression in the sre module itself, but I can't say so for certain.
animalize commented on Apr 7, 2022
Could you give the two versions? I will do a git bisect.
I tested 356997c~1 and 356997c [1], msvc2022 non-pgo release build:
# regex_dna ###
an +- std dev: 151 ms +- 1 ms -> 152 ms +- 1 ms: 1.01x slower
t significant
# regex_effbot ###
an +- std dev: 2.47 ms +- 0.01 ms -> 2.46 ms +- 0.02 ms: 1.00x faster
t significant
# regex_v8 ###
an +- std dev: 21.7 ms +- 0.1 ms -> 22.4 ms +- 0.1 ms: 1.03x slower
gnificant (t=-30.82)
356997c
sweeneyde commented on Apr 8, 2022
Possibly related to the new atomic grouping support from #76163?
animalize commented on Apr 8, 2022
It seems not likely.
I will do some benchmarks for this issue, more information (version/platform) is welcome.
brandtbucher commented on Apr 13, 2022
Not sure how we're supposed to be linking issues to PRs yet, but check out #91495.
gh-91404: Use computed gotos and reduce indirection in re (#91495)
brandtbucher commented on Apr 15, 2022
Leaving this open, since it may still be worth exploring where the prior 3.10-to-3.11 slowdown came from.
Hello71 commented on Jun 6, 2022
Chromium build effectively hangs with Python 3.11 due to re slowdown. generate_about_tracing_contents takes 6 seconds on Python 3.10, and I killed it after 1 hour on Python 3.11.
Reduced example:
re.findall(r'(?:(?:/[^\n]*)?\n)*/', '../utils/common":3}],15:[function(t,e,a){"use strict";e.exports=function(){this.input=null,this.next_in=0,this.avail_in=0,this.total_in=0,this.output=null,this.next_out=0,this.avail_out=0,this.total_out=0,this.msg="",this.state=null,this.data_type=2,this.adler=0}},{}],"/":[function(t,e,a){"use strict";var i={};(0,t("./lib/utils/common").assign)(i,t("./lib/deflate"),t("./lib/inflate"),t("./lib/zlib/constants")),e.exports=i},{"./lib/deflate":1,"./lib/inflate":2,"./lib/utils/common":3,"./lib/zlib/constants":6}]},{},[])("/")});\n')takes about 0.02 seconds on Python 3.10, and 8.6 seconds on Python 3.11.
Full example: slow.tar.gz
sweeneyde commented on Jun 6, 2022
I git-bisected @Hello71's "Reduced example" slowdown to commit 6e3eee5
Author: Ma Lin <animalize@users.noreply.github.com> Date: Mon Apr 4 00:16:20 2022 +0800 bpo-23689: re module, fix memory leak when a match is terminated by a signal or memory allocation failure (GH-32283)gpshead commented on Jun 15, 2022
#93882 proposed to rollback
6e3eee5
there were a few other changes to re after that landed so the rollback required some hand edits.
gpshead commented on Jun 15, 2022
I can confirm that the PR fixes the performance regression in @Hello71 's reproducer - thanks @sweeneyde for tracking that down.
gh-91404: Revert "bpo-23689: re module, fix memory leak when a match …
pythongh-91404: Revert "bpo-23689: re module, fix memory leak when a …
gh-91404: Revert "bpo-23689: re module, fix memory leak when a match …
Revert 30935 defer bpo45162 to 312 (#94285)
Add Brandt's pythongh-91404 regex speedup from Optimizations to Misc …
Remove pythongh-91404 re entry from Optimizations to add to Faster CP…