gh-89263: Add typing.get_overloads by JelleZijlstra · Pull Request #31716 · python/cpython · GitHub

Credit to Spencer Brown for the idea to use overloads' firstlineno to decide whether to clear the registry.

https://bugs.python.org/issue45100

Thanks! I have a few thoughts.

Overall, I guess I'm a little concerned about "dumping" so many new functions in the global functools namespace. functools is already kind of a hodgepodge of unrelated things, and I worry that this will make the problem worse.

Normally I'd say that having a class with 0 instance methods would be a code smell, but what about something like this?

class VariantRegistry:
    _registry = defaultdict(list)

    @staticmethod
    def _get_key_for_callable(func):
        func = getattr(func, "__func__", func)
        try:
            return f"{func.__module__}.{func.__qualname__}"
        except AttributeError:
            return None

    @classmethod
    def register_variant(func, variant):
        key = cls._get_key_for_callable(func)
        if key is not None:
            cls._variant_registry[key].append(variant)

    @classmethod
    def get_variants(cls, func):
        return cls._registry[cls._get_key_for_callable(func)]

    @classmethod
    def clear_variants(cls, func=None):
        if func is None:
           cls._variant_registry.clear()
        else:
            cls._variant_registry.pop(cls._get_key_for_callable(func), None)

Thanks for the review!

I see the point about dumping, but I don't think using a class just as a bag of methods is a good solution. This group of functions serves a similar purpose to update_wrapper and wraps (enabling introspection for advanced usage of functions) and functools now feels like a good home for them.

What happens if a user does something weird (and probably misguided?) like this?

@overload
def complex_func(arg: str) -> int: ...

@overload
def complex_func(arg: int) -> str: ...

@singledispatch
def complex_func(arg: object):
    raise NotImplementedError

@complex_func.register
def _(arg: str) -> int: 
    return int(arg)

@complex_func.register
def _(arg: int) -> str:
    return str(arg)

What's the runtime behaviour in this case? Do we care? Do we warn users not to do this in the docs? Do we add a test case to make sure the behaviour is "correct"?

I think we need to account for this possible (mis)usage somehow, even if type checkers would probably balk at it.

What's the runtime behaviour in this case? Do we care? Do we warn users not to do this in the docs? Do we add a test case to make sure the behaviour is "correct"?

I think we need to account for this possible (mis)usage somehow, even if type checkers would probably balk at it.

We simply get variants both from overload and singledispatch. I added a test case based on this example.

Implemented this approach. I now see a 5x performance ratio:

In [9]: %timeit overload(f)
223 ns ± 1.85 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [10]: %timeit old_overload(f)
44.7 ns ± 0.74 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [11]: 223 / 44.7
Out[11]: 4.988814317673378

Can you add a comment explaining why this sequence is repeated?

Done. Also added a test case to make sure that clearing one function's overloads doesn't clear another.

Thinking out loud, clear_overloads() is becoming the most complicated part of this. Perhaps we don't need to support clearing overloads per function, which would simplify it greatly. Use cases I can think of are:

You want to save some memory after importing all your code -> you call clear_overloads() and get rid of them all.
You are constantly reloading a module with overloads in it and changing the line numbers. Then maybe you want to clear_overloads() for just that function. But that seems really exotic.

So maybe we should only support clearing the whole overload registry at once. Can you think of other realistic reasons people may want to use clear_overloads()?

Hm. Given the reload(module) use case, maybe the only options should be clear all or clear a specific module? The new data structure makes that easy. The question would then be whether the argument should be the (full) module name or the module object? (Or either?) The sequence of events would be something like

typing.clear_overload(mod)
importlib.reload(mod)

I can't really think of a use case for clearing a specific function. Did you have one in mind originally? Or was it just convenient for testing?

I think for now we should allow only clearing the whole registry. I see the use case for wanting to repeatedly reload modules while wanting to introspect overloads, but it seems unlikely to be common. We can always add an argument later to clear overloads by module if someone asks for it, but once we add it we're stuck with it.

@gvanrossum @AlexWaygood any further feedback? The last thing to resolve is whether clear_overloads should take arguments (#31716 (comment)).

This looks pretty good now! Just a few thoughts:

This sentence sounds quite strange to me:

*func* is the function object for the implementation of the overloaded function.

But I'm not sure I have a better suggestion off the top of my head ://

Yes, unfortunately the overload docs don't have a clear term for the implementation function either.

I'll wait until @AlexWaygood approves, then I'll do one more pass if you want me to.

Suggested change

      
               ``get_overloads`` returns an empty sequence.
          
               return an empty sequence.

Tiny nit -- this paragraph starts off using the imperative mood

But the previous sentence is in the indicative, so I think this is fine?

Well, it's not really a big deal either way :)

Hooray!

@gvanrossum would you like to take another look or should I merge this?

Go for it!

JelleZijlstra added 2 commits March 5, 2022 21:24

initial

2ee377d

Implementation, tests, and docs

831b565

JelleZijlstra requested review from Fidget-Spinner, gvanrossum and rhettinger as code owners March 7, 2022 02:16

bedevere-bot added the awaiting core review label Mar 7, 2022

the-knights-who-say-ni added the CLA signed label Mar 7, 2022

JelleZijlstra commented Mar 7, 2022

View reviewed changes

Comment thread Doc/library/functools.rst Outdated

fix versionadded

f03f8a9

AlexWaygood reviewed Mar 7, 2022

View reviewed changes

Comment thread Lib/functools.py Outdated

Comment thread Lib/functools.py Outdated

Comment thread Lib/functools.py Outdated

Merge branch 'main' into funcregistry

404668a

make get_key_for_callable private

7a5b0d1

AlexWaygood reviewed Mar 8, 2022

View reviewed changes

Comment thread Doc/library/functools.rst Outdated

AlexWaygood reviewed Mar 8, 2022

View reviewed changes

Comment thread Lib/test/test_typing.py Outdated

AlexWaygood reviewed Mar 8, 2022

View reviewed changes

Comment thread Doc/library/functools.rst Outdated

AlexWaygood reviewed Mar 8, 2022

View reviewed changes

doc updates; remove unnecessary try-except

6998255

JelleZijlstra marked this pull request as draft March 9, 2022 04:56

JelleZijlstra added 6 commits March 27, 2022 15:42

Merge remote-tracking branch 'upstream/main' into funcregistry

26bb908

rename method

f52b757

Don't store singledispatch in the registry

fc6a925

more tests

b524244

and another

e95558e

fix line length in new tests

31fd72d

JelleZijlstra commented Mar 27, 2022

View reviewed changes

Comment thread Doc/library/functools.rst Outdated

Update Doc/library/functools.rst

7041ad3

JelleZijlstra commented Mar 27, 2022

View reviewed changes

Comment thread Doc/library/typing.rst Outdated

Update Doc/library/typing.rst

e26b0db

Simpler implementation (thanks Guido)

103bfd4

JelleZijlstra removed the request for review from rhettinger April 9, 2022 01:55

gvanrossum reviewed Apr 9, 2022

View reviewed changes

AlexWaygood self-requested a review April 9, 2022 04:34

More comments and tests

d453f7f

AlexWaygood added topic-typing type-feature A feature request or enhancement labels Apr 13, 2022

JelleZijlstra changed the title bpo-45100: Add typing.get_overloads gh-89263: Add typing.get_overloads Apr 14, 2022

JelleZijlstra added 2 commits April 13, 2022 19:00

Merge remote-tracking branch 'upstream/main' into funcregistry

450afeb

simplify clear_overloads

ea62287

AlexWaygood reviewed Apr 14, 2022

View reviewed changes

JelleZijlstra added 4 commits April 14, 2022 09:41

use partial

905253c

add test

debbf8a

docs changes (thanks Alex)

754c134

Merge remote-tracking branch 'upstream/main' into funcregistry

1ad8224

AlexWaygood reviewed Apr 14, 2022

View reviewed changes

AlexWaygood approved these changes Apr 14, 2022

View reviewed changes

AlexWaygood added the stdlib Standard Library Python modules in the Lib/ directory label Apr 14, 2022

AlexWaygood mentioned this pull request Apr 16, 2022

Improve help() by making typing.overload() information accessible at runtime #89263

Open

JelleZijlstra merged commit 055760e into python:main Apr 16, 2022

bedevere-bot removed the awaiting core review label Apr 16, 2022

JelleZijlstra deleted the funcregistry branch April 16, 2022 16:01

JelleZijlstra mentioned this pull request Apr 16, 2022

Add get_overloads() python/typing#1140

Merged

TeamSpen210 mentioned this pull request Apr 16, 2022

[Feature Request] Support typing.get_overloads() under Python ≥ 3.11 beartype/beartype#54

Open

AlexWaygood mentioned this pull request Sep 21, 2023

gh-109653: typing.py: improve import time by creating soft-deprecated members on demand #109651

Merged

+                      impl, overloads = self.set_up_overloads()
+                      self.assertNotEqual(typing._overload_registry, {})
+                      self.assertEqual(list(get_overloads(impl)), overloads)
+                      clear_overloads()
+                      self.assertEqual(typing._overload_registry, {})
+                      self.assertEqual(get_overloads(impl), [])

+                 the documentation for :func:`@overload <overload>`,
+                 ``get_overloads(process)`` will return a sequence of three function objects
+                 for the three defined overloads. If called on a function with no overloads,
+                 ``get_overloads`` returns an empty sequence.

		func. func is the function object for the implementation of the
		overloaded function. For example, given the definition of ``process`` in

	``get_overloads`` returns an empty sequence.
	return an empty sequence.

gh-89263: Add typing.get_overloads by JelleZijlstra · Pull Request #31716 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-89263: Add typing.get_overloads - #31716

gh-89263: Add typing.get_overloads#31716JelleZijlstra merged 37 commits intopython:mainpython/cpython:mainfrom JelleZijlstra:funcregistryJelleZijlstra/cpython:funcregistryCopy head branch name to clipboard

Conversation

JelleZijlstra commented Mar 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

AlexWaygood left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra commented Mar 8, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra commented Mar 27, 2022

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra commented Apr 9, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra commented Apr 14, 2022

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gvanrossum commented Apr 14, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra commented Apr 16, 2022

Uh oh!

gvanrossum commented Apr 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

gh-89263: Add typing.get_overloads#31716
JelleZijlstra merged 37 commits into
python:mainfrom
JelleZijlstra:funcregistry

JelleZijlstra commented Mar 7, 2022 •
edited
Loading

AlexWaygood left a comment •
edited
Loading