FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-126220: Adapt `_lsprof` to Argument Clinic by sobolevn · Pull Request #126233 · python/cpython · GitHub

/ cpython Public

gh-126220: Adapt _lsprof to Argument Clinic - #126233

Merged
sobolevn merged 10 commits into
python:mainfrom
sobolevn:issue-126220
Nov 4, 2024
ConversationCommits (10)ChecksFiles changed
Merged

gh-126220: Adapt _lsprof to Argument Clinic#126233
sobolevn merged 10 commits into
python:mainfrom
sobolevn:issue-126220

Conversation

sobolevn commented Oct 31, 2024 •
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Member

I decided to go with the AC, because:

  1. It is supported in this module
  2. It solves the problem
  3. It fixes multiple signatures

But, I believe it will be slower. If we really want to preserve speed in this case, I can add manual size checks for args.

Refs #103534

bedevere-app Bot mentioned this pull request Oct 31, 2024
Closed
ZeroIntensity approved these changes Oct 31, 2024
View reviewed changes

ZeroIntensity left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM. I'm always happy with fixes via AC, because they're much more scalable.

gaogaotiantian commented Oct 31, 2024

Copy link
Copy Markdown
Member

What's the perf impact here?

sobolevn commented Oct 31, 2024

Copy link
Copy Markdown
Member Author

@gaogaotiantian I've never had experience with profiling cProfile. Do you have any links / suggestions on how to do that?

gaogaotiantian commented Oct 31, 2024

Copy link
Copy Markdown
Member

Just do a quick fib() with cprofile and compare it with the previous implementation would do. Maybe also with one without cprofile.

sobolevn commented Oct 31, 2024

Copy link
Copy Markdown
Member Author

@gaogaotiantian maybe I did something wrong, but looks like we also got a speedup from my numbers 😅 (I have little idea about how cProfile works).

# Test code: ex.py
def test():
    def fib():
        start, nextn = 0, 1
        while True:
            yield start
            start, nextn = nextn, start + nextn

    gen = fib()
    for _ in range(100000):
        next(gen)

import cProfile
cProfile.run(test.__code__)

Running it on main:

» ./python.exe ex.py
         200003 function calls in 0.188 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.017    0.017    0.188    0.188 ex.py:1(test)
   100000    0.147    0.000    0.147    0.000 ex.py:2(fib)
        1    0.000    0.000    0.188    0.188 {built-in method builtins.exec}
   100000    0.024    0.000    0.171    0.000 {built-in method builtins.next}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Running on this PR:

» ./python.exe ex.py                                                     
         200003 function calls in 0.185 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.016    0.016    0.185    0.185 ex.py:1(test)
   100000    0.145    0.000    0.145    0.000 ex.py:2(fib)
        1    0.000    0.000    0.185    0.185 {built-in method builtins.exec}
   100000    0.024    0.000    0.168    0.000 {built-in method builtins.next}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Am I missing something?

gaogaotiantian commented Oct 31, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

Sorry I meant a recursive fib().

import time
def fib(n):
    if n <= 1:
        return 1
    return fib(n - 1) + fib(n - 2)
start = time.time()
# enable profiler
fib(24)
# disable profiler
print(time.time() - start)

We need to consider the worst case for cprofile, which is when the code has a lot of function calls. We need to know the overhead of the profiler, and how that is compared with before.

You can also refer to #103533, there's a profiling code listed. That might give some extra information.

Moving to sys.monitoring gave us a 20%+ improvement on overhead, just want to check what's the regression with clinic and whether it's acceptable. Profiler is a bit sensitive to performance because the more overhead there is, the less useful it is.

sobolevn commented Oct 31, 2024

Copy link
Copy Markdown
Member Author

It still gives me a perf boost for some reason 🤔

import time
import cProfile

profiler = cProfile.Profile()

def fib(n):
    if n <= 1:
        return 1
    return fib(n - 1) + fib(n - 2)
start = time.perf_counter()
profiler.enable()
fib(30)  # 100th
profiler.disable()
print(time.perf_counter() - start)
# Old: 0.9349823750017094
# New: 0.9253051670020795
# 1% faster

gaogaotiantian commented Oct 31, 2024

Copy link
Copy Markdown
Member

Is the boost stable(reproducible)?

sobolevn commented Oct 31, 2024 •
edited
Loading

Copy link
Copy Markdown
Member Author

Yes, it is pretty much the same on m2 macos:

(.venv) ~/Desktop/cpython2  main ✗                                                        
» ./python.exe ex.py                                                     
0.9349823750017094
                                                                                           
(.venv) ~/Desktop/cpython2  main ✗                                                        
» ./python.exe ex.py
0.9357958749969839
                                                                                           
(.venv) ~/Desktop/cpython2  main ✗                                                        
» ./python.exe ex.py
0.9352227920026053
                                                                                           
(.venv) ~/Desktop/cpython2  main ✗                                                        
» ./python.exe ex.py
0.924946416002058
                                                                                           
(.venv) ~/Desktop/cpython2  main ✗                                                        
» ./python.exe ex.py
0.926337541997782
                                                                                           
(.venv) ~/Desktop/cpython2  main ✗                                                        
» ./python.exe ex.py
0.9347994170000311

(.venv) ~/Desktop/cpython2  issue-126220 ✗  
» ./python.exe ex.py
0.9370929580036318
                                                                                           
(.venv) ~/Desktop/cpython2  issue-126220 ✗                                                
» ./python.exe ex.py
0.9262957079990883
                                                                                           
(.venv) ~/Desktop/cpython2  issue-126220 ✗                                                
» ./python.exe ex.py
0.9328266249940498
                                                                                           
(.venv) ~/Desktop/cpython2  issue-126220 ✗                                                
» ./python.exe ex.py
0.9382220000011148
                                                                                           
(.venv) ~/Desktop/cpython2  issue-126220 ✗                                                
» ./python.exe ex.py
0.9285237919984502
                                                                                           
(.venv) ~/Desktop/cpython2  issue-126220 ✗                                                
» ./python.exe ex.py
0.9296539580027456

gaogaotiantian commented Oct 31, 2024

Copy link
Copy Markdown
Member

Yeah from the result I don't think there's an observable boost. On the other hand, that's good, because no observable regression either.

gaogaotiantian added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Oct 31, 2024
gaogaotiantian approved these changes Oct 31, 2024
View reviewed changes

gaogaotiantian left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Just a side note - this needs to be backported to 3.12 as well :) I added the tags so you should be able to just merge it.

sobolevn commented Oct 31, 2024

Copy link
Copy Markdown
Member Author

@gaogaotiantian sorry, I forgot to highlight it initially. Please, double check params names that I introduced. Are they correct?

gaogaotiantian commented Oct 31, 2024

Copy link
Copy Markdown
Member

Actually, I have some doubts about the changes for functions that are not crashing. This is a bug fix, which would be backported. I don't think we should mix in the code polish to the PR. Even though I think changing this in main would be fine, I would prefer having only the crash fix (could be with AC) in one PR, and the rest in the other. One thing that bothers me immediately is that I can't quickly convince myself the changes to __init__ is purely equivalent.

As for the argument name, the obj in the callbacks should be either unused or instruction_offset(which is what it is).

gaogaotiantian commented Oct 31, 2024

Copy link
Copy Markdown
Member

Can we split this PR into 2? One with the changes to only the 4 callbacks, and the other with all the other methods?

sobolevn commented Oct 31, 2024

Copy link
Copy Markdown
Member Author

Fair enough, I would split this PR later! 👍
Thank you!

gaogaotiantian approved these changes Oct 31, 2024
View reviewed changes

gaogaotiantian left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM!

sobolevn commented Oct 31, 2024

Copy link
Copy Markdown
Member Author

@erlend-aasland maybe you would be interested in double checking the AC part? :)

sobolevn changed the title gh-126220: Fix crash on calls to _lsprof.Profiler methods with 0 args gh-126220: Convert _lsprof to AC Nov 1, 2024
erlend-aasland reviewed Nov 1, 2024
View reviewed changes
Comment thread Modules/_lsprof.c Outdated
Comment thread Modules/_lsprof.c Outdated
Comment thread Modules/_lsprof.c Outdated
Comment thread Modules/_lsprof.c Outdated
erlend-aasland changed the title gh-126220: Convert _lsprof to AC gh-126220: Adapt _lsprof to Argument Clinic Nov 1, 2024
sobolevn and others added 3 commits November 2, 2024 08:30
Apply suggestions from code review
3cea189 
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Address review
b84dde3
Typo
bf3e120
gaogaotiantian reviewed Nov 3, 2024
View reviewed changes
Comment thread Modules/_lsprof.c Outdated
erlend-aasland reviewed Nov 3, 2024
View reviewed changes
Comment thread Modules/_lsprof.c Outdated

sobolevn commented Nov 4, 2024

Copy link
Copy Markdown
Member Author

Done!

sobolevn merged commit c806cd5 into python:main Nov 4, 2024

miss-islington-app Bot commented Nov 4, 2024

Copy link
Copy Markdown

Thanks @sobolevn for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

sobolevn commented Nov 4, 2024

Copy link
Copy Markdown
Member Author

Thanks everyone!

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 4, 2024
pythongh-126220: Adapt _lsprof to Argument Clinic (pythonGH-126233)
9eaf3e6 
(cherry picked from commit c806cd5)

Co-authored-by: sobolevn <mail@sobolevn.me>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

miss-islington-app Bot commented Nov 4, 2024

Copy link
Copy Markdown

Sorry, @sobolevn, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker c806cd5af677c385470001efc68da38a32919196 3.12

bedevere-app Bot commented Nov 4, 2024

Copy link
Copy Markdown

GH-126402 is a backport of this pull request to the 3.13 branch.

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Nov 4, 2024

sobolevn commented Nov 4, 2024

Copy link
Copy Markdown
Member Author

Oups, backports are not needed anymore. Closing them.

sobolevn removed the needs backport to 3.12 only security fixes label Nov 4, 2024
picnixz pushed a commit to picnixz/cpython that referenced this pull request Dec 8, 2024
pythongh-126220: Adapt _lsprof to Argument Clinic (python#126233)
c86f49b 
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
ebonnal pushed a commit to ebonnal/cpython that referenced this pull request Jan 12, 2025
pythongh-126220: Adapt _lsprof to Argument Clinic (python#126233)
2081eee 
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

gaogaotiantian gaogaotiantian approved these changes
erlend-aasland erlend-aasland approved these changes
ZeroIntensity ZeroIntensity approved these changes
markshannon Awaiting requested review from markshannon

Assignees

sobolevn

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

Back | FazBrowse Home | New Git URL