Environment
- Pythonnet version: 3.0.5
- Python version: 3.13
- Operating System: MacOS Sequoia 15.1.1
- .NET Runtime: 8.0.403
Details
After #2530, I took the time to test from Python some of the most common C# containers, testing in particular the methods from the standard Python abc's (Sequence, MutableSequence, Mapping, MutableMapping, ...).
Here are my results, along with a few proposed fixes. Please let me know if there are any mistakes.
Shared setup
import pythonnet
pythonnet.load('coreclr')
import clr
from System.Collections.Generic import Dictionary, List, KeyValuePair
from System.Collections.Immutable import ImmutableArray, ImmutableList, ImmutableDictionary
from System import Int32, String, Object, Nullable, Array
from System.Collections.Generic import CollectionExtensions
Array-like containers
ArrT = Array[Int32]
ImmArrT = ImmutableArray[Int32]
ListT = List[Int32]
arr = ArrT([0, 1, 2, 3, 4])
immarr = ImmutableArray.Create[Int32](0, 1, 2, 3, 4) # is this supposed to work, instead of using Create()?
lst = ListT(ArrT([0, 1, 2, 3, 4]))
immlst = ImmutableList.ToImmutableList[Int32](ArrT([0, 1, 2, 3, 4]))
What worked everywhere
- __iter__
- __reversed__
- __contains__
- __len__
- reverse()
- __getitem__ when the index is within bounds
- index() when the element is in the container
- __setitem__ when the index is within bounds
What didn't work
- __getitem__ when the index is negative or out of bounds
arr[-1] # OK
arr[999] # OK, raises IndexError
immarr[-1] # System.IndexOutOfRangeException: Index was outside the bounds of the array.
lst[-1] # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
immlst[-1] # System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'index')
Python expects a negative index to be interpreted as the position from the end of the container. Otherwise this breaks the implementation of pop() from abc (on top of a common assumption). This translation is only done for Array (here) for some reason.
Also, the exception when the index is out of bounds should be IndexError, otherwise this breaks the implementation of Sequence.index() among other things.
Possible fix
Add the following method to class SequenceMixin:
def __getitem__(self, key):
length = len(self)
key = key if key >= 0 else length + key
if key not in range(length):
raise IndexError('index out of range')
return self.get_Item(key)
... and make sure List doesn't override it?
- index() when the element is NOT in the container
arr.index(999) # OK, raises ValueError
immarr.index(999) # System.IndexOutOfRangeException: Index was outside the bounds of the array.
lst.index(999) # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
immlst.index(999) # System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'index')
This is caused by the previous point, the exceptions are thrown in Sequence.index().
- __getitem__, __setitem__ and __delitem__ with slices
arr[0:3] # TypeError: array index has type slice, expected an integer
immarr[0:3] # TypeError: No method matches given arguments for ImmutableArray`1.get_Item: (<class 'slice'>)
lst[0:3] # TypeError: No method matches given arguments for List`1.get_Item: (<class 'slice'>)
immlst[0:3] # TypeError: No method matches given arguments for ImmutableList`1.get_Item: (<class 'slice'>)
This is ok, clearly not supported which is fine. Exceptions self-explanatory.
- __setitem__ when the index is negative or out of bounds
arr[-1] = 9 # OK
lst[-1] = 9 # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
arr[999] = 9 # OK, raises IndexError
lst[999] = 9 # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
OK for Array; for List, same as __getitem__: a negative index is not translated, and an out of bounds index raises the wrong exception.
Possible fix
Like for __getitem__, add the following method to class MutableSequenceMixin:
def __setitem__(self, key, value):
length = len(self)
key = key if key >= 0 else length + key
if key not in range(length):
raise IndexError('index out of range')
self.set_Item(key, value)
- methods that would result in a different Array length
del arr[4] # SystemError: Objects/longobject.c:583: bad argument to internal function
arr.insert(0, 5) # IndexError; calls abc implementation, which is a stub!
arr.append(5) # IndexError; uses abc implementation, which falls back to insert
arr.pop() # SystemError (same as del); uses abc implementation, which calls del
arr.clear() # SystemError (same as del); uses abc implementation, which calls del
arr.extend([10, 11, 12]) # IndexError; uses abc implementation, which calls append
arr.remove(9) # SystemError (same as del); uses abc implementation, which calls del
arr += [10, 11, 12] # IndexError; uses abc implementation, which falls back to extend
This could be fine, to disallow changing an array's length, even though Array.Resize and Array.Copy could in theory allow us to have all of these working and be fairly efficient.
But the exceptions have me think that this is another kind of problem:
- the SystemError comes from the bowels of CPython, precisely here, so I assume this is unintended behavior;
- the IndexError is caused by MutableSequenceMixin not implementing insert(); if not supporting this is intended, it could be made more understandable by implementing it (in Array, not MutableSequenceMixin) and throwing TypeError, System.NotSupportedException or something more explanatory.
Crash output
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Python.Runtime.BorrowedReference.DangerousGetAddress() in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/BorrowedReference.cs:line 18
at Python.Runtime.NewReference..ctor(BorrowedReference reference, Boolean canBeNull) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/NewReference.cs:line 20
at Python.Runtime.Runtime.PyTuple_SetItem(BorrowedReference pointer, IntPtr index, BorrowedReference value) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Runtime.cs:line 1482
at Python.Runtime.ClassBase.mp_ass_subscript_impl(BorrowedReference ob, BorrowedReference idx, BorrowedReference v) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Types/ClassBase.cs:line 498
[1] 37819 abort python3 delme_test_pythonnet.py
Like for Dictionary, __delitem__ on a List causes a hard crash; possibly related to #2530.
- other methods mutating List length
lst.insert(0, 5) # IndexError; calls abc implementation, which is a stub!
lst.append(5) # IndexError; uses abc implementation, which calls insert
lst.pop() # IndexError; uses abc implementation, which calls __getitem__(-1) and del
lst.clear() # IndexError; uses abc implementation, which calls pop
lst.extend([10, 11, 12]) # IndexError; uses abc implementation, which calls append
lst.remove(3) # CRASH; uses abc implementation, which calls del
lst += [10, 11, 12] # IndexError; uses abc implementation, which falls back to extend
These should definitely work; it mostly boils down to the previous point, the missing support for negative indexes and insert() not being implemented in MutableMappingMixin.
Possible fix
Add the following method to class MutableSequenceMixin:
def insert(self, index, value):
self.Insert(index, value)
And also apply some of the previous proposed fixes, on top of fixing del.
Dictionary-like containers
DictT = Dictionary[Int32, String]
DictT2 = Dictionary[String, Int32]
DictT3 = Dictionary[String, Nullable[Int32]]
d = DictT()
d[10] = "10"
d[20] = "20"
d[30] = "30"
d2 = DictT2()
d3 = DictT3()
rod = CollectionExtensions.AsReadOnly[Int32, String](DictT(d)) # ReadOnlyDictionary<int, string>
immd = ImmutableDictionary.ToImmutableDictionary[Int32, String](d) # ImmutableDictionary<int, string>
What worked everywhere
- __contains__
- __len__
- __getitem__ when the key is in the dictionary
- __setitem__
- keys
- values
- items, although maybe it could be achieved without copying
- get
- clear
- update
What didn't work
This crashes with the following output; already tracked in #2530.
Crash output
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Python.Runtime.BorrowedReference.DangerousGetAddress() in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/BorrowedReference.cs:line 18
at Python.Runtime.NewReference..ctor(BorrowedReference reference, Boolean canBeNull) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/NewReference.cs:line 20
at Python.Runtime.Runtime.PyTuple_SetItem(BorrowedReference pointer, IntPtr index, BorrowedReference value) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Runtime.cs:line 1482
at Python.Runtime.ClassBase.mp_ass_subscript_impl(BorrowedReference ob, BorrowedReference idx, BorrowedReference v) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Types/ClassBase.cs:line 498
[1] 44289 abort python3 delme_test_pythonnet.py
- __iter__ returns KeyValuePairs instead of tuples
list(d) # gives a list of KeyValuePair
This contradicts the Mapping protocol which mandates that the mapping is an iterable of its keys, and breaks the popitem() implementation from collections.abc. At least the latter should be fixed.
d.popitem() # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[Int32,String]'>)
d2.popitem() # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[String,Int32]'>)
d3.popitem() # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[String,Nullable[Int32]]'>)
This is caused by the previous point, the exceptions are thrown in MutableMapping.popitem() which expects the dictionary to be an iterable of its keys.
Possible fix
Add the following to MutableMappingMixin:
def popitem(self):
try:
key = next(iter(self.Keys))
except StopIteration:
raise KeyError from None
value = self[key]
del self[key]
return key, value
... on top of fixing del.
- __getitem__ when the key is not in the dictionary
d[40] # KeyNotFoundException: The given key '40' was not present in the dictionary.
rod[40] # KeyNotFoundException: The given key '40' was not present in the dictionary.
immd[40] # KeyNotFoundException: The given key '40' was not present in the dictionary.
The exception is not translated to a ValueError, which might theoretically break some functions trying to catch it, but it doesn't seem as important here.
- pop and setdefault for some types
d.pop(10) # OK
d2.pop("10") # TypeError: No method matches given arguments for Dictionary`2.TryGetValue: (<class 'str'>, <class 'NoneType'>)
d3.pop("10") # OK
d.setdefault(10, "10") # OK
d2.setdefault("10", 10) # TypeError: No method matches given arguments for Dictionary`2.TryGetValue: (<class 'str'>, <class 'NoneType'>)
d3.setdefault("10", 10) # OK
d3.setdefault("10") # Also OK
I believe this breaks only with non-nullable value types: the exception is called from the MutableMappingMixin implementation of pop(), which calls self.TryGetValue(key, None) here.
Possible fix: call self.TryGetValue(key) instead? I believe this is supported since commit f69753c.
Environment
Details
After #2530, I took the time to test from Python some of the most common C# containers, testing in particular the methods from the standard Python abc's (Sequence, MutableSequence, Mapping, MutableMapping, ...).
Here are my results, along with a few proposed fixes. Please let me know if there are any mistakes.
Shared setup
Array-like containers
What worked everywhere
What didn't work
Python expects a negative index to be interpreted as the position from the end of the container. Otherwise this breaks the implementation of pop() from abc (on top of a common assumption). This translation is only done for Array (here) for some reason.
Possible fixAlso, the exception when the index is out of bounds should be IndexError, otherwise this breaks the implementation of Sequence.index() among other things.
Add the following method to class SequenceMixin:
... and make sure List doesn't override it?
This is caused by the previous point, the exceptions are thrown in Sequence.index().
This is ok, clearly not supported which is fine. Exceptions self-explanatory.
OK for Array; for List, same as __getitem__: a negative index is not translated, and an out of bounds index raises the wrong exception.
Possible fixLike for __getitem__, add the following method to class MutableSequenceMixin:
This could be fine, to disallow changing an array's length, even though Array.Resize and Array.Copy could in theory allow us to have all of these working and be fairly efficient.
But the exceptions have me think that this is another kind of problem:
Like for Dictionary, __delitem__ on a List causes a hard crash; possibly related to #2530.
These should definitely work; it mostly boils down to the previous point, the missing support for negative indexes and insert() not being implemented in MutableMappingMixin.
Possible fixAdd the following method to class MutableSequenceMixin:
And also apply some of the previous proposed fixes, on top of fixing del.
Dictionary-like containers
What worked everywhere
What didn't work
This crashes with the following output; already tracked in #2530.
Crash outputThis contradicts the Mapping protocol which mandates that the mapping is an iterable of its keys, and breaks the popitem() implementation from collections.abc. At least the latter should be fixed.
This is caused by the previous point, the exceptions are thrown in MutableMapping.popitem() which expects the dictionary to be an iterable of its keys.
Possible fixAdd the following to MutableMappingMixin:
... on top of fixing del.
The exception is not translated to a ValueError, which might theoretically break some functions trying to catch it, but it doesn't seem as important here.
I believe this breaks only with non-nullable value types: the exception is called from the MutableMappingMixin implementation of pop(), which calls self.TryGetValue(key, None) here.
Possible fix: call self.TryGetValue(key) instead? I believe this is supported since commit f69753c.