Repository navigation
functools.partial placeholders #119127
Description
Feature or enhancement
Proposal:
I know that this has been brought up in the past, but I would appreciate if this was reconsidered. I would be happy to bring this up to a nice level.
from functools import partial
from partial2 import partial as partial2, Placeholder
_ = VOID = Placeholder()
%timeit partial(opr.sub, 1, 2) # 130 ns
%timeit partial2(opr.sub, 1, 2) # 155 ns
p1 = partial(opr.sub, 1)
p2 = partial2(opr.sub, 1)
p3 = partial2(opr.sub, _, 1)
p4 = partial2(opr.sub, VOID, 1)
print(p1(2)) # -1
print(p2(2)) # -1
print(p3(2)) # 1
print(p4(2)) # 1
%timeit p1(2) # 48 ns
%timeit p2(2) # 52 ns
%timeit p3(2) # 54 ns
%timeit p4(2) # 54 nsThe logic is as follows:
def partial(func, *p_args, **p_kwds):
p_args = list(p_args)
nvoid = sum(a is VOID for a in p_args)
arng = range(len(p_args))
def wrapper(*args, **kwds):
if len(args) < nvoid:
raise ValueError('Not all VOIDs are filled')
t_args = p_args.copy()
j = 0
for i in arng:
if p_args[i] is VOID:
t_args[i] = args[j]
j += 1
t_args.extend(args[j:])
return func(*t_args, **{**p_kwds, **kwds})
return wrappervectorcall change looks like:
if (np) {
nargs_new = pto_nargs + nargs - np;
Py_ssize_t ip = 0;
for (Py_ssize_t i=0; i < pto_nargs; i++) {
if (PyObject_TypeCheck(pto_args[i], &placeholderType)){
memcpy(stack + i, args + ip, 1 * sizeof(PyObject*));
ip += 1;
} else {
memcpy(stack + i, pto_args + i, 1 * sizeof(PyObject*));
}
}
if (nargs_total > np){
memcpy(stack + pto_nargs, args + np, (nargs_total - np) * sizeof(PyObject*));
}
// EO Placeholders ------
} else {Has this already been discussed elsewhere?
I have already discussed this feature proposal on Discourse
Links to previous discussion of this feature:
https://discuss.python.org/t/functools-partial-placeholder-arguments/53487
Activity
rhettinger commented on May 17, 2024
Note, this has been discussed and rejected multiple times in the past. Please specifically address these comments:
Also see:
Leaving this open for further comment and reconsideration by other developers.
dg-pb commented on May 17, 2024
[quote="Raymond Hettinger, post:20, topic:19163, username:rhettinger"]
I’m 100% opposed to including better-partial or anything like it in the standard library. At best, it would be an attractive nuisance.
[/quote]
In https://discuss.python.org/t/functools-partial-placeholder-arguments/53487 I have laid out several examples and use cases for this. But the main gain here is speed improvement by being able to efficiently utilise functions from operator module to use as predicates to functionals, where argument order is not convenient. I have been lately working on iterator recipes and from my experience the partial wrap is 20 ns vs lambda call being ~50ns can result in considerable speed-ups. Especially if it is used more than once.
[quote="Raymond Hettinger, post:9, topic:19163, username:rhettinger"]
Every variant I’ve seen is more complex, less speedy, and harder to debug than a simple def or lambda. Every variant required something extra to learn and remember, unlike def or lambda which are Python basics.
[/quote]
Placeholder sentinel still needs work, it would have much nicer representation. E.g: (functools.PPHoldder, 1)
Personally, I would make use of this functionality in algorithm recipes, iterator recipes, utility functions and similar places of lower level code where performance is important. And I don't see this as lambda or def replacement, but rather functionality that is well suited for a clear purpose.
Also, partial already exists with many of the problems that are being used against this functionality. But given the fact that partial exists already why not make it better and make use of it in the area where it excels (performance)?
Why I think this deserves reconsideration is that this hasn't been implemented and benchmarked before. All the previous attempts were in python, where instead of performance gain which has led me to this point, the result was a significant performance loss - up to 20x slower than current standard library implementation. So naturally, those are not used, because lambda/def is a simpler, much faster than better_partial or any other 3rd party implementation and has all the flexibility.
While before supporting reasoning was convenience, Linters, MyPy issues and similar, my main argument is performance.
rhettinger commented on Jun 6, 2024
I've left this open for comment and no one else has stepped in to opine, so I'm going to go ahead and approve it.
The API is a minimal addition that gives more flexibility without venturing into creating a mini-DSL. It is unclear whether it will present any typing challenges beyond what partial() already does, but I expect that will only be a minor obstacle.
The PR needs a lot of work but the concept is sound. I've been trying it out for a while and it looks reasonable in real code. It supports multiple placeholders but in the examples I've found only one is needed.
I'm dubious about the motivation as an optimization (PyPy doesn't need this and CPython may yet optimize a plain lambda version). However, it does make partial more broadly useful and that is a sufficient reason.
rhettinger commented on Jun 6, 2024
FWIW, the example I most enjoyed is bin8 = partial(format, Placeholder, '08b').
picnixz commented on Jun 8, 2024
Personally, most of the time I need a placeholder for is when I want right partialization, not left partialization. While there are times where partialization in arbitrary order makes sense, an alternative would to provide rpartial interface without introducing new singletons or immortal objects. The specs on how to use rpartial would still remain an open question (i.e., is rpartial(f, a, b)(x) for f(x, a, b) or f(x, b, a)?)
serhiy-storchaka commented on Jun 26, 2024
rpartial() could be simpler, and it would cover most of the use cases, but the placeholder provides more flexibility. If Raymond is fine with this, we should choose this option.
Although, there is a case which is supported by rpartial(), but not be the proposed feature with a placeholder -- when we want to add positional arguments after the variable number of positional arguments for the partial function. I think that we could support *Placeholder as a placeholder for variable number of arguments. But this can be implemented in a separate issue, for the first step it is enough to support Placeholder for individual arguments.
sobolevn commented on Jun 26, 2024
I like functional programming a lot (proof: https://github.com/dry-python/returns), but I don't think that this API is nice. Here are my thoughts:
It is nice as a separate package, but I don't feel like I would like to use it in the stdlib. Or teach how to use it in the stdlib. Sorry :(
dg-pb commented on Jun 26, 2024
This is either not correct or I am failing to understand what you mean.
serhiy-storchaka commented on Aug 12, 2024
There is a question: how interpret Placeholder passed as a keyword argument? It could be expected that partial(func, a=Placeholder) is a callable that passes its first positional argument as the "a" keyword argument to func(). But this is difficult to implement, and we currently have no good use cases for this (although it may be changed in future). So I suggest to reject passing Placeholder as a keyword argument.
dg-pb commented on Aug 12, 2024
Yeah, I don't see any need for this. And it would indeed be difficult, I think there is enough of complexity for now to settle.
I think there is no harm in literal treatment of such. At least I can not see any potential issues here. It just gets treated as any other object.
I tried to come up with potential benefits. E.g. passing Placeholder via partial keyword which constructs another partial, where Placeholder (or not) is put into positional argument. But can not come up with anything realistic. Although I potentially see myself trying to exploit this fact in the future.
So I am generally neutral, but slightly positive leaving it as it is due to performance.
serhiy-storchaka commented on Aug 12, 2024
It is easier to make valid something that was previously invalid than to change semantic of already valid expression.
dg-pb commented on Aug 12, 2024
So I am still unsure about this.
If it was free, it would be natural.
However as I suspected it is quite costly. Effect on instantiation for both versions given the number of keyword arguments:
For someone who mostly cares about __call__ performance this is not a big impact, but for applications that rely on creating many partial objects this is fairly significant.
Is the chance to make use of Placeholder in keywords in the future actually realistic? It is unlikely that adding complexity to keywords will be worth it given the effort required and performance decrease - such extension would have much higher performance penalty (compared to this extension) as manipulating dicts is much more costly than tuples.
Also, is there any significant risk that someone is going to pass Placeholder to partial via keywords? Even if by any chance it is desired to extend functionality to keywords (in some way) in the future, I don't think it would break a lot of code as it is very unlikely anyone will be passing Placeholder to keywords.
4 remaining items
dg-pb commented on Sep 5, 2024
There is 1 point which is in favour of b).
There is less chance that the user will depend on type(Placeholder)() raising an error than returning a singleton object. Thus, it is easier to revert from b) to a) than from a) to b).
However, I don't think this is the main consideration as I think choosing more appropriate option from the very beginning would be better so that the probability for future need of reversion is minimised.
After all, the cost function for reverting is not simply a) vs b), but rather:
Where P(A to B) is probability that in the future this will need to be changed from A to B.
So I would rather pick more appropriate option from the beginning.
Maybe someone who knows the rationale for making None & friends singletons could help. I.e. why was decision made for type(None)() to return None instead of raising an error? Are there any specific reasons or is it more along the lines of "smoother user experience" for constant which is highly exposed to the user?
To me Placeholder still seems closer to None (& friends) or typing.NoReturn than for example, to constants in inspect module, such as _ParameterKind enums or any of the results of '(?m)^[^\W\d]+\w* \= object\(\)', where none of sentinels seem to be user facing.
So currently my position is that A is more appropriate.
@serhiy-storchaka, given this comment and the one above, are you still convinced that B is better?
@rhettinger, do you have any preferences here?
serhiy-storchaka commented on Sep 5, 2024
I still prefer option b. We should not add feature without a clear use case. Actually, we should discourage any use of Placeholder besides passing it as a positional argument to partial() or partialmethod(). Even partial(func, arg1, arg2) can be unsafe if arg1 is dynamically generated and can be Placeholder. So using a fabric function that returns Placeholder is very unlikely.
Waiting on @rhettinger.
gh-119127: functools.partial placeholders (gh-119827)
pythongh-119127: functools: Improve docs for partial and Placeholder
pythongh-119127: Fix _functools.Placeholder
pythongh-119127: Fix _functools.Placeholder singleton
pythongh-119127: Fix _functools.Placeholder singleton
gh-119127: functools: Improve docs for partial and Placeholder (#124575)
gh-119127: Fix _functools.Placeholder singleton (#124601)
dg-pb commented on Oct 17, 2024
3 follow-up PRs.
Largely independent from each other.
All ready for review.
functools.partial will be a method descriptor
functools.partial will be a method descriptor
functools.partial will be a method descriptor
functools.partial will be a method descriptor