gh-130167: Minor textwrap.dedent() optimization - #131925
Closed
Marius-Juston wants to merge 2 commits into
Closed
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
Copy link
Copy Markdown
|
All commit authors signed the Contributor License Agreement. |
Sorry, something went wrong.
Copy link
Copy Markdown
Member
|
It seems for the benchmarks handling large text, there's a consistent slowdown. Could you plot a chart of running the function with inputs of different sizes? Perhaps take the unicodeobject file and profile the first 500 lines, then the first 1000, 1500, etc, for this PR and the version using min/max. A |
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
|
It stays a consistently slower by 5-8% except for long lines test
|
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
|
From additional testing, this way is slightly faster in the ranges where the lines are between 1-18.
after analyzing all the function documentation in the Python codebase the average documentation length seems to be the following https://gist.github.com/Marius-Juston/08c574901317ee40837ab87ce0855685:
( excluding all 1 line comments ) so technically, this implementation is slightly faster for the average documentation length ( now is this performance increase really worth it, no probably not lol ) |
Sorry, something went wrong.
eendebakpt
reviewed
Mar 31, 2025
Comment thread
Lib/textwrap.py
Outdated
| l1 = min(non_blank_lines, default='') | ||
| l2 = max(non_blank_lines, default='') | ||
| margin = 0 | ||
| l1 = None |
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Suggested change
| l1 = None | |
| l1 = '' |
You can then skip the is None check in the loop. Similar for l2 (but you have to pick the default with care)
Sorry, something went wrong.
eendebakpt
reviewed
Mar 31, 2025
Comment thread
Lib/textwrap.py
Outdated
| if line and not line.isspace(): | ||
| if l1 is None or line < l1: | ||
| l1 = line | ||
| if l2 is None or line > l2: |
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Once you skip the if None check, this if can be an elseif
Sorry, something went wrong.
Copy link
Copy Markdown
Member
|
I'm against this change not only because it's harder to read but also because it only offers an overall <10% speed-up. In general, we accept optimizations exceeding that threshold. |
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
|
@eendebakpt with the new changes:
Small values
Large file
So on the large files we get 0 improvement, a 13% slight improvement on the synthetic test benches and on smaller values |
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
Agreed that this optimization is most likely not necessary, though this is very slightly more efficient memory wise since it does not have a non_blank_lines variable so it technically only duplicates the lines variable only twice rather than 3 times like in the current implementation ( in real world this is most definitively not really a problem just a thought ) |
Sorry, something went wrong.
eendebakpt
reviewed
Mar 31, 2025
Comment thread
Lib/textwrap.py
Comment on lines
+435
to
+453
| val = False | ||
| for i, line in enumerate(lines): | ||
| # Compute min + max concurrently + normalize others | ||
| if line and not line.isspace(): | ||
| if val: | ||
| if line < l1: | ||
| l1 = line | ||
| elif line > l2: | ||
| l2 = line | ||
| else: | ||
| val = True | ||
| l1 = l2 = line | ||
|
|
||
| else: | ||
| lines[i] = '' | ||
|
|
||
| if not val or not l1: | ||
| return '\n'.join(lines) | ||
|
|
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Suggested change
| val = False | |
| for i, line in enumerate(lines): | |
| # Compute min + max concurrently + normalize others | |
| if line and not line.isspace(): | |
| if val: | |
| if line < l1: | |
| l1 = line | |
| elif line > l2: | |
| l2 = line | |
| else: | |
| val = True | |
| l1 = l2 = line | |
| else: | |
| lines[i] = '' | |
| if not val or not l1: | |
| return '\n'.join(lines) | |
| l1 = 'z' | |
| l2 = '' | |
| for i, line in enumerate(lines): | |
| # Compute min + max concurrently + normalize others | |
| if line and not line.isspace(): | |
| if line < l1: | |
| l1 = line | |
| if line > l2: | |
| l2 = line | |
| else: | |
| lines[i] = '' | |
| if not l1: | |
| return '\n'.join(lines) | |
| margin = 0 | |
| for margin, c in enumerate(l2): | |
| if c != l1[margin] or c not in ' \t': | |
| break | |
This saves chechking the val. Performance gain is only a bit though.
I agree with the other reviewers: unless we find out a way to make this much faster, the small gains are not worth the change.
@Marius-Juston Thanks for putting in the work to get all the performance numbers!
Sorry, something went wrong.
ZeroIntensity
requested changes
Mar 31, 2025
ZeroIntensity
left a comment
Copy link
Copy Markdown
Member
There was a problem hiding this comment.
Thanks for getting the benchmarks, sometimes its hard to get contributors to quote numbers, and you've gone above and beyond with benchmarking :).
But unfortunately, the numbers make me -1 on this.
- min/max are builtins that are written in C; they're pretty fast, especially compared to multiple lines of Python code. If we're concerned about those being slow, let's try to optimize those on the C level instead.
- There is a significant amount of added complexity here, for seemingly not much benefit.
- Optimizing Python code by shifting things around is generally not a good idea, because when the interpreter changes, those optimizations become obsolete. We should definitely not be quoting memory usage, number of (pointer) copies, or number of lookups; those are all things that change on an interpreter-wide level.
I've learned that it can be hard to know when to stop painting. I think we should put down the brush, and enjoy the optimization that's already landed 🙂.
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
|
Makes sense. I am just having so much fun doing this it is definitely hard to stop lol! |
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
|
Closing this down then! ( I will be putting a feature request for the min_max implementation) |
Sorry, something went wrong.
Copy link
Copy Markdown
Member
I think it's already been proposed but rejected so search through issues and https://discuss.python.org/c/ideas/6 first. |
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
I think I found it, the https://discuss.python.org/t/minmax-function-alongside-min-and-max/2834/73 issue got closed down. It seems like it got stale and just ignored? A shame. |
Sorry, something went wrong.
Copy link
Copy Markdown
Member
|
Feel free to open a new thread referencing the one you found. Personally I'd suggest statistics.min_max. It might be useful to identify a few motivating examples in the stdlib / popular packages -- this will be one of the first questions you're asked on Discourse. A |
Sorry, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Minor optimization to @AA-Turner #131792 where you compute the min and max of the non_blank_lines simultaneously, removes additional use of line.isspace() as well.