Update list.{count, index, remove} to accept `object` type by randolf-scholz · Pull Request #15472 · python/typeshed · GitHub

Partially fixes #15271

Hm, not exactly sure what's going on with numpy, my best guess is that since _NestedSequence is defined with (https://github.com/numpy/numpy/blob/main/numpy/_typing/_nested_sequence.py)

    def count(self, value: Any, /) -> int:
        """Return the number of occurrences of `value`."""
        raise NotImplementedError

    def index(self, value: Any, /) -> int:
        """Return the first index of `value`."""
        raise NotImplementedError

then after we make list use object rather than Any, this is still a subtype but possibly no longer a "proper_subtype". This is a mypy interal distinction and likely needs to be addressed on mypys side.

EDIT: I am starting to think that this may be related to mypy interal solver semantics. we removed some constrains, so now it may find solutions for the T of a list-constructor inside nested function calls, where previously there was no solution and it defaulted to Never which then propagate throught the call stack and may change inference in some cases.

The primer output is not promising. The fixed # type: ignores are mostly in test code. On the other hand there seems to be some churn, and even some true positives (in colour) that aren't flagged anymore.

The main issue seems to be [var-annotated] issue originating from numpy:

import numpy as np

reveal_type(np.array([]))

# on main
tmp_a.py:3: note: Revealed type is "numpy.ndarray[tuple[Any, ...], numpy.dtype[Any]]"
# PR:
tmp_a.py:3: note: Revealed type is "numpy.ndarray[tuple[Any, ...], numpy.dtype[Never]]"

@srittau I wouldn't call the colour results true-positives. For instance https://github.com/colour-science/colour/blob/c9cfa1f333452cc05e42f09eebd2f3dd935a5981/colour/utilities/network.py#L1756-L1773 is exactly one of the cases of a valid list.remove call that is currently a false positive.

OK, I think I have a repro of the numpy issue that causes the [var-annotated] errors. It has to do with the NestedSequence Protocol. Here is a minimal reproduction:

from typing import Any, Protocol, reveal_type

class NestedSeq[_T_co](Protocol):

    def __getitem__(self, index: int, /) -> "_T_co | NestedSeq[_T_co]":
        raise NotImplementedError

    def count(self, value: Any, /) -> int:
        raise NotImplementedError

def as_nested_sequence[T](seq: NestedSeq[T], /) -> NestedSeq[T]:
    return seq

reveal_type(as_nested_sequence([]))
# main: NestedSeq[Any], PR: NestedSeq[Never]

And this happens due to how mypy solves type-vars. on main, count is [T] -> int, so mypy solves T=Any, whereas on PR, count is [object] -> int, so mypy has no constraints for T and picks T=Never.

Imo, this is more of a mypy problem than a typeshed problem. But it could be worked around if for example numpy used default=Any for the type-variable of _NestedSequence. cc @jorenham

Imo, this is more of a mypy problem than a typeshed problem. But it could be worked around if for example numpy used default=Any for the type-variable of _NestedSequence. cc @jorenham

Sure, we could do that.

Imo, this is more of a mypy problem than a typeshed problem. But it could be worked around if for example numpy used default=Any for the type-variable of _NestedSequence. cc @jorenham

Sure, we could do that.

The workaround will be present in the upcoming 2.4.5 and 2.5.0 releases.

numpy 2.4.5 was just released (https://github.com/numpy/numpy/releases/tag/v2.4.5)

@jorenham Ugh, terribly sorry but that wasn't sufficient. mypy still resolves _ScalarT to Never in this overload:

@overload
def array(
    object: _ArrayLike[_ScalarT],
    dtype: None = None,
    *,
    copy: bool | _CopyMode | None = True,
    order: _OrderKACF = "K",
    subok: bool = False,
    ndmin: int = 0,
    ndmax: int = 0,
    like: _SupportsArrayFunc | None = None,
) -> NDArray[_ScalarT]: ...

because it doesn't find any constraints on _ScalarT when doing np.array([]), so mypy emits var-annotated1.

So actually it is this typevar that needs a default (either Any or np.generic I guess).

Again, sorry for not testing this more thoroughly locally, doing the default on _NestedSequence itself in hindisght obviously only helps when we use _NestedSequence directly, but not with def [T](_NestedSequence[T]) -> Array[T] type constructors. Then we need def [T=Any](_NestedSequence[T]) -> Array[T] instead.

Which, let's be honest, probably shouldn't even be an error enabled by default. pyrights / pyrefly / ty all use list[Unknown] when doing x = [] and don't emit any errors with standard settings. ↩

Ah no worries. At least we know what to put in the mypy bug report now ;)

@srittau I inspected the mypy-primer in more detail. Looking at the issues in detail, I believe the mypy-primer changes are positive in almost all cases.

optuna

visualization_tests/test_rank.py#L699: This error should be expected. If mypy complains on x2=[], then one should expect it to also complain on x2=np.array([])

colour

colour/utilities/network.py:1772 This is a false positive. Here, node is an instance of PortNode and _children is annotated as list[Self], i.e. list[PortGraph], and PortGraph subclasses PortNode. Since list.remove is equality based, this does the right thing at runtime, and shows that the current annotation is too strict.
colour/notation/munsell/centore2014.py:2053 the flagged code here is correct. The underlying issue is an incorrect annoation in _munsell_maximum_chromas_from_renotation, is annotated as returning tuple[tuple[Pair, float]] , but actually returns tuple[tuple[Triple, float]] (with Triple=tuple[float, float, float], Pair=tuple[float, float]). mypy doesn't catch this incorrect annotation because chromas = {} is not annotated inside the function.

pandas

the var-annotated errors are expected, as with the optuna case.
pandas/plotting/_matplotlib/core.py:1207 Not enitrely sure what happens here, didn't debug this one.

xarray

xarray/tests/test_concat.py:112 Not exactly sure what's happening here, if required I could look into it
xarray/tests/test_backends.py:2090 I am not entirely sure what happens here, but it seems that even on main, np.ma.array(y, mask=[True, True, False, False, False]) with y: list[float] produces a var-annotated. However, with an inlined list constructor, on main mypy thinks that the first list expression in np.ma.array([-1, -1, 10, 10.1, 10.2], mask=[True, True, False, False, False]) is a list[Any], whereas with this PR it thinks it is a list[float], which is more accurate.

scipy-stubs

tests/linalg/test_blas.pyi:27 With the new annotations, list[str] should satisfy the _SequenceNotStr[str] Protocol, which it previously didn't. This overload and result seems more precise then bevor.
3 unused-ignores, nice
tests/special/test_ufuncs.pyi:75 This is already flagged with pyright: ignore and pyrefly: ignore, so I'd assume its a mypy false-negative.

The pyright failures seem unrelated

tests/special/test_ufuncs.pyi:75 This is already flagged with pyright: ignore and pyrefly: ignore, so I'd assume its a mypy false-negative.

Those are indeed tests for true positives. The trick is that in scipy-stubs all type-checkers are configured to complain in case of unused ignore comments. So it's indeed a good thing that mypy now also reports an error.

I think list[int].index(12.34) should be a type checker error. If the thing can fail at runtime with some float values, we shouldn't say that float is supported.

More generally, I don't think it makes sense to allow an overlapping type in these methods (let alone object). If the runtime returns a fallback or is otherwise silent, like dict.get or set.discard, that's different.

@Akuli There are legitimate use cases, for instance calling .index with a union type (like T | None) or supertype of the generic type.

class RGB: ...
class RGBA(RGB): ...

class Demo:
    colors: list[RGBA]

    def check_value(self, value: RGB) -> None:
        if value in self.colors:
            position = self.colors.index(value)
            ...

I understand why you want list[int].index(12.34) to be a typing error, this is a code smell and likely incorrect code. But again, if we set this in stone in the stubs you deny any legitimate use case. What you want should imho be some opt-in linter rule that type checkers like mypy or ty can implement if they want. But typeshed is the wrong place for this imo.

In your example, ideally the type checker would narrow the type of value into RGBA because it is in a List[RGBA]. Then the .index() would succeed as is.

You could also make the same argument with any other operation that needs an RGBA object, and I don't see how list.index() is special here. For example:

class RGB:
    red: float
    green: float
    blue: float
    ...

class RGBA(RGB):
    alpha: float
    ...

colors: list[RGBA] = [...]
value = get_some_rgb_color()
if value in colors:
    print(f"opacity is {value.alpha}")

More generally, the correct fix is to help type checkers know the correct type, not to silence errors when they have the wrong type.

To be concrete, I would be very surprised if mistakes like list[int].index(12.34) (or list[int].index("hello")) were less frequently occurring, and hence less important, than the problems that this PR solves. And as I said in your other PR, mypy_primer is not a good way to measure this because most of the code it checks already passes the type check.

In your example, ideally the type checker would narrow the type of value into RGBA because it is in a List[RGBA]. Then the .index() would succeed as is.

This would not be ideal, but actually incorrect as list containment is equality based. Your own example will error on the line print(f"opacity is {value.alpha}") if RGBA overwrites __eq__ in such a way that an RGB can equal an RGBA:

from dataclasses import dataclass

@dataclass(frozen=True)
class RGB:
    red: int
    green: int
    blue: int

@dataclass(frozen=True)
class RGBA(RGB):
    alpha: int = 255

    def __eq__(self, other):
        if isinstance(other, RGBA):
            return hash(self) == hash(other)
        if isinstance(other, RGB):
            return RGB.__hash__(self) == hash(other) and self.alpha == 255
        return NotImplemented

black = RGB(0, 0, 0)
colors = [RGBA(0, 0, 0)]
if black in colors:
    print(black.alpha)  # ERROR at runtime

another example with pytorch:

import torch

# Note: torch.Size subclasses tuple[int, ...]
sizes: list[torch.Size] = [torch.Size([1, 2, 3])]
print(sizes[0].numel())  # 6 (product of values)
value = (1, 2, 3)
if value in sizes:
    print(value.numel())  # ERROR

There is actually a regression in mypy type checking here, introduced in 1.20.0 python/mypy#21512

I see. There might not be a way to express this so that overrided __eq__ methods and the error for list[int].index(float) both work. But even then, falling back to object is just not the way to go.

There are probably many other places too where overrided __eq__ methods behave weirdly.

Again I think what you want is a fine strict linter rule to have, but you seem very eager to force this linter rule on everyone else, even when it caues false positives in their code.

Diff from mypy_primer, showing the effect of this PR on open source code:

pwndbg (https://github.com/pwndbg/pwndbg)
+ pwndbg/aglib/elf.py:445: error: Unused "type: ignore" comment  [unused-ignore]

optuna (https://github.com/optuna/optuna)
+ tests/visualization_tests/test_rank.py:699: error: Need type annotation for "x2"  [var-annotated]

colour (https://github.com/colour-science/colour)
- colour/utilities/network.py:2083: error: Argument 1 to "remove" of "list" has incompatible type "PortNode"; expected "Self"  [arg-type]
+ colour/notation/munsell/centore2014.py:2081: error: Incompatible types in assignment (expression has type "floating[_16Bit] | floating[_32Bit] | float64", variable has type "float64")  [assignment]
- colour/notation/munsell/centore2014.py:2055: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]"  [arg-type]
- colour/notation/munsell/centore2014.py:2060: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]"  [arg-type]
- colour/notation/munsell/centore2014.py:2067: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]"  [arg-type]
- colour/notation/munsell/centore2014.py:2072: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]"  [arg-type]
- colour/plotting/models.py:1457: error: Incompatible types in assignment (expression has type "Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | str | _NestedSequence[complex | bytes | str]", variable has type "ndarray[tuple[Any, ...], dtype[Any]]")  [assignment]
+ colour/plotting/models.py:1457: error: Incompatible types in assignment (expression has type "Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | str | _NestedSequence[complex | bytes | str]", variable has type "ndarray[Any, Any]")  [assignment]

dedupe (https://github.com/dedupeio/dedupe)
+ dedupe/labeler.py:410: error: List item 1 has incompatible type "Iterable[Literal[0, 1]]"; expected "_SupportsArray[dtype[signedinteger[_64Bit]]] | _NestedSequence[_SupportsArray[dtype[signedinteger[_64Bit]]]]"  [list-item]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/dtypes/dtypes.py:984: error: List comprehension has incompatible type List[dtype[Any] | ExtensionDtype]; expected List[_NestedSequence[Never]]  [misc]
+ pandas/core/arrays/sparse/array.py:1580: error: Need type annotation for "sparse_values"  [var-annotated]
+ pandas/core/internals/managers.py:2530: error: List comprehension has incompatible type List[ndarray[tuple[Any, ...], dtype[Any]] | ExtensionArray]; expected List[_SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]]]  [misc]
+ pandas/plotting/_matplotlib/misc.py:170: error: Need type annotation for "s"  [var-annotated]
+ pandas/plotting/_matplotlib/core.py:1216: error: List item 0 has incompatible type "Number | number[Any, Any]"; expected "_SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]]"  [list-item]

bokeh (https://github.com/bokeh/bokeh)
+ src/bokeh/core/property/wrappers.py: note: In class "PropertyValueList":
+ src/bokeh/core/property/wrappers.py:265:5: error: Argument 1 of "remove" is incompatible with supertype "builtins.list"; supertype defines the argument type as "object"  [override]
+ src/bokeh/core/property/wrappers.py:265:5: note: This violates the Liskov substitution principle
+ src/bokeh/core/property/wrappers.py:265:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

xarray (https://github.com/pydata/xarray)
+ xarray/tests/test_concat.py: note: In function "create_typed_datasets":
+ xarray/tests/test_concat.py:112: error: Argument 1 to "reshape" has incompatible type "list[Timedelta]"; expected "_SupportsArray[dtype[Never]] | _NestedSequence[_SupportsArray[dtype[Never]]]"  [arg-type]
+ xarray/tests/test_concat.py: note: At top level:

scipy-stubs (https://github.com/scipy/scipy-stubs)
+ tests/linalg/test_blas.pyi:27: error: Expression is of type "list[_FortranFunction]", not "list[_FortranFunction] | _FortranFunction"  [assert-type]
+ tests/linalg/test__decomp_svd.pyi:72: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/linalg/test__decomp_svd.pyi:73: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/linalg/test__decomp_svd.pyi:74: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/special/test_ufuncs.pyi:87: error: Argument 1 to "__call__" of "_UFunc11f" has incompatible type "list[complex]"; expected "CanArrayND[floating[_16Bit] | integer[_16Bit] | integer[_8Bit] | numpy.bool[builtins.bool], tuple[Any, ...]] | SequenceND[CanArrayND[floating[_16Bit] | integer[_16Bit] | integer[_8Bit] | numpy.bool[builtins.bool], tuple[Any, ...]]] | SequenceND[builtins.bool]"  [arg-type]
+ tests/special/test_ufuncs.pyi:87: note: "list" is missing following "CanArrayND" protocol member:
+ tests/special/test_ufuncs.pyi:87: note:     __array__

don't worry about the scipy-stubs mypy_primer diff

Update list.{count, index, remove} to accept object type

0f25cae

randolf-scholz mentioned this pull request Feb 28, 2026

Allow arbitrary types in dict.pop (3 overloads) #15297

Closed