Using `ctypes.CField` for annotations of fields within `ctypes.Structure`-like · Issue #10567 · python/typeshed · GitHub

Currently, attributes of ctypes.Structure, such as ctypes.wintypes.POINT, are annotated with names and types that specified in _fields_.

typeshed/stdlib/ctypes/wintypes.pyi

Lines 126 to 128 in df08fce

    
           class POINT(Structure): 
        
               x: LONG 
        
               y: LONG

The problem with this approach is that at runtime, the attributes return types like int or bytes, while type checkers interpret them as returning c_int or c_char.

Below is the simplest example of a structure created for explanation purposes.
At runtime, the types of these fields are ctypes.CField. In the example below, x is a data descriptor that returns int as the getter and accepts both c_int and int as the setter.

>>> import ctypes
>>>      
>>> class Foo(ctypes.Structure):
...     pass
... 
>>> Foo._fields_ = [('x', ctypes.c_int)]
>>> Foo.x
<Field type=c_long, ofs=0, size=4>
>>> type(Foo.x) 
<class '_ctypes.CField'>
>>> foo = Foo()
>>> foo
<__main__.Foo object at 0x0000023CDB80B8C0>
>>> foo.x
0
>>> foo.x = 3
>>> foo.x
3
>>> foo.x = ctypes.c_int(2)
>>> foo.x
2

Having stubs with types that deviate from the runtime is not an ideal situation.
Therefore, I propose modifying CField as shown below for use in annotating fields.

_T = TypeVar("_T")
_CT = TypeVar("_CT", bound=_CData)


class CField(Generic[_T, _CT]):
    offset: int
    size: int
    @overload
    def __get__(self, instance: None, owner: type[Any]) -> Self: ...
    @overload
    def __get__(self, instance: Any, owner: type[Any] | None) -> _T: ...
    def __set__(self, instance: Any, value: _T | _CT) -> None: ...


class Foo(Structure):
    x: ClassVar[CField[int, c_int]]


# Foo._fields_ = [('x', c_int)]  # required in runtime


a = Foo.x  # CField[int, c_int]
foo = Foo()
b = foo.x  # int
foo.x = 3  # OK
foo.x = c_int(2)  # OK
foo.x = 3.14  # NG
foo.x = c_double(3.14)  # NG

another idea

I thought it elegant to specify only subclasses of _SimpleCData[_T] as type parameters, given the potential for inferring _T, as shown below.

class Bar(Structure):
    x: ClassVar[CField[c_int]]  # returns `int`, can take `int` or `c_int`

However, current static type system cannot that.

Moreover, considering that there exist not only subclasses of _SimpleCData defined within ctypes, but also third-party developers who define _SimpleCData subclasses within their own projects.

There is the redundancy, but I believe that utilizing the two type parameters would enhance flexibility.

I know python/cpython#104533. It might be necessary to communicate with stakeholders in https://discuss.python.org/t/annotation-based-sugar-for-ctypes/26579/ .

I've only looked briefly at your proposal, but using the descriptor protocol looks indeed to be the right solution in this case. An exploratory PR to judge the effects of such a change would be welcome!

While working, I noticed something.
In fields like Array[c_wchar], we cannot assign an instance of the Array[c_wchar] type; we can only assign types like str.

>>> import ctypes
>>> class Foo(ctypes.Structure):
...     pass
... 
>>> Foo._fields_ = [('x', (ctypes.c_wchar * 14))] 
>>> Foo.x
<Field type=c_wchar_Array_14, ofs=0, size=28>
>>> foo = Foo()
>>> foo.x
''
>>> foo.x = 'spam'
>>> foo.x
'spam'
>>> y = (ctypes.c_wchar * 14)()      
>>> y.value = 'ham' 
>>> foo.x = y
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unicode string expected instead of c_wchar_Array_14 instance

Therefore, I believe it would be more appropriate to have a descriptor protocol with below three type parameters;

The type that configured to the field (the type passed to _fields_)
The type that the field returns
The type that the field accepts

They are such as CField[_CT, _RetT, _SetT].

In the initial PR, I intend to make changes only to _ctypes, ctypes, and wintypes.
For other third-party libraries, as I am not well-versed in their specifications, I would prefer to leave that topic to other maintainers.

I PRed #10595.

#10595 is merged.

Sorry to comment on a closed issue. I posted the following issue on pyright: microsoft/pyright#5932 (comment) which redirected me back here.

In the following example, both using from_buffer_copy() inside something() and calling it directly on the MyStruct object gives a pyright error

import ctypes
from typing import ClassVar


class MyStruct(ctypes.Structure):
    _fields_ = [("a", ctypes.c_uint32)]


class MyClass:
    _struct: ClassVar = MyStruct

    @classmethod
    def something(cls, encoded: bytes):
        return cls._struct.from_buffer_copy(encoded)


MyStruct.from_buffer_copy(b"1234")

I have not used the ctypes library a lot, am i doing something wrong by calling a classmethod (e.g from_buffer_copy() on a Structure object directly?

The curious thing is that Structure is a subclass of _ctypes._CData, which has a from_buffer_copy class method, yet it ended up in this situation.

_StructUnionMeta has __getattr__, and when we access attributes not statically defined in _StructUnionBase, it returns _CField.

However, _StructUnionBase is a subclass of _CData, so from_buffer_copy is already defined.

Before I changed the type stubs, _CField has not been Callable.

Is there some interaction between the descriptor protocol and metaclasses that I'm not aware of?

I have the understanding that the issue has been resolved, as mentioned below.

#10777
#10795
microsoft/pyright#5932 (comment), microsoft/pyright#5932 (comment)
https://github.com/microsoft/pyright/releases/tag/1.1.330

Using `ctypes.CField` for annotations of fields within `ctypes.Structure`-like · Issue #10567 · python/typeshed · GitHub

Repository navigation

Using ctypes.CField for annotations of fields within ctypes.Structure-like #10567

another idea

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Using `ctypes.CField` for annotations of fields within `ctypes.Structure`-like · Issue #10567 · python/typeshed · GitHub

Repository navigation

Using ctypes.CField for annotations of fields within ctypes.Structure-like #10567

Description

another idea

Activity

junkmd commented on Aug 13, 2023

srittau commented on Aug 14, 2023

junkmd commented on Aug 16, 2023

junkmd commented on Aug 16, 2023

junkmd commented on Aug 17, 2023

junkmd commented on Sep 1, 2023

einarwar commented on Sep 13, 2023

junkmd commented on Sep 13, 2023

junkmd commented on Oct 7, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions