FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-112433: Add optional _align_ attribute to ctypes.Structure by monkeyman192 · Pull Request #113790 · python/cpython · GitHub

/ cpython Public

gh-112433: Add optional _align_ attribute to ctypes.Structure #113790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
serhiy-storchaka merged 9 commits into from
Feb 15, 2024
Conversation 41 Commits 9 Checks 0 Files changed
Merged

gh-112433: Add optional _align_ attribute to ctypes.Structure #113790

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a6ef3ac
Add optional _align_ attribute to ctypes.Structure
monkeyman192 Jan 6, 2024
e2cceec
📜🤖 Added by blurb_it.
blurb-it[bot] Jan 28, 2024
2329fa3
Fix issue with aligned subclasses
monkeyman192 Jan 28, 2024
ebaed7e
Improve tests to pass on big endian machines
monkeyman192 Jan 28, 2024
6b183b5
Merge branch 'main' into add_structure_alignment
serhiy-storchaka Jan 28, 2024
c8b37d4
Further improvements to tests
monkeyman192 Jan 29, 2024
68e2d61
Test cleanup and improvements
monkeyman192 Jan 29, 2024
f6a93dd
More test improvements
monkeyman192 Jan 31, 2024
a7bc0fe
Add extra test and fix documentation typo
monkeyman192 Feb 2, 2024
File filter

Filter by extension

Filter by extension .c  (1) .h  (4) .py  (1) .rst  (2) All 4 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
10 changes: 10 additions & 0 deletions Doc/library/ctypes.rst
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -670,6 +670,10 @@ compiler does it. It is possible to override this behavior by specifying a
:attr:`~Structure._pack_` class attribute in the subclass definition.
This must be set to a positive integer and specifies the maximum alignment for the fields.
This is what ``#pragma pack(n)`` also does in MSVC.
It is also possible to set a minimum alignment for how the subclass itself is packed in the
same way ``#pragma align(n)`` works in MSVC.
This can be achieved by specifying a ::attr:`~Structure._align_` class attribute
in the subclass definition.

:mod:`ctypes` uses the native byte order for Structures and Unions. To build
structures with non-native byte order, you can use one of the
Expand Down Expand Up @@ -2534,6 +2538,12 @@ fields, or any other data types containing pointer type fields.
Setting this attribute to 0 is the same as not setting it at all.


.. attribute:: _align_

An optional small integer that allows overriding the alignment of
the structure when being packed or unpacked to/from memory.
Setting this attribute to 0 is the same as not setting it at all.

.. attribute:: _anonymous_

An optional sequence that lists the names of unnamed (anonymous) fields.
Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_global_objects_fini_generated.h
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Include/internal/pycore_global_strings.h
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(_abc_impl)
STRUCT_FOR_ID(_abstract_)
STRUCT_FOR_ID(_active)
STRUCT_FOR_ID(_align_)
STRUCT_FOR_ID(_annotation)
STRUCT_FOR_ID(_anonymous_)
STRUCT_FOR_ID(_argtypes_)
Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_runtime_init_generated.h
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Include/internal/pycore_unicodeobject_generated.h
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

286 changes: 286 additions & 0 deletions Lib/test/test_ctypes/test_aligned_structures.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,286 @@
from ctypes import (
c_char, c_uint32, c_uint16, c_ubyte, c_byte, alignment, sizeof,
BigEndianStructure, LittleEndianStructure,
BigEndianUnion, LittleEndianUnion,
)
import struct
import unittest


class TestAlignedStructures(unittest.TestCase):
def test_aligned_string(self):
for base, e in (
(LittleEndianStructure, "<"),
(BigEndianStructure, ">"),
):
data = bytearray(struct.pack(f"{e}i12x16s", 7, b"hello world!"))
class Aligned(base):
_align_ = 16
_fields_ = [
('value', c_char * 12)
]

class Main(base):
_fields_ = [
('first', c_uint32),
('string', Aligned),
]

main = Main.from_buffer(data)
self.assertEqual(main.first, 7)
self.assertEqual(main.string.value, b'hello world!')
self.assertEqual(bytes(main.string), b'hello world!\0\0\0\0')
self.assertEqual(Main.string.offset, 16)
Comment thread
serhiy-storchaka marked this conversation as resolved.
self.assertEqual(Main.string.size, 16)

serhiy-storchaka Jan 30, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

It is not affected by the _align_ setting, because its size was already 16. Maybe make the value field type c_char * 12? Then the _align_ setting will affect also the size of Aligned.
self.assertEqual(alignment(main.string), 16)

serhiy-storchaka Jan 30, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Does alignment(Main.string) work?

monkeyman192 Jan 30, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

It does not since Main.string is a ctypes.CField which has no alignment info.
self.assertEqual(alignment(main), 16)

def test_aligned_structures(self):
Comment thread
serhiy-storchaka marked this conversation as resolved.
for base, data in (
(LittleEndianStructure, bytearray(b"\1\0\0\0\1\0\0\0\7\0\0\0")),
(BigEndianStructure, bytearray(b"\1\0\0\0\1\0\0\0\7\0\0\0")),
):
class SomeBools(base):
_align_ = 4
_fields_ = [
("bool1", c_ubyte),
("bool2", c_ubyte),
]
class Main(base):
_fields_ = [
("x", c_ubyte),
("y", SomeBools),
("z", c_ubyte),
]

main = Main.from_buffer(data)
self.assertEqual(alignment(SomeBools), 4)
self.assertEqual(alignment(main), 4)
self.assertEqual(alignment(main.y), 4)
self.assertEqual(Main.x.size, 1)
self.assertEqual(Main.y.offset, 4)
self.assertEqual(Main.y.size, 4)
self.assertEqual(main.y.bool1, True)
self.assertEqual(main.y.bool2, False)
self.assertEqual(Main.z.offset, 8)
self.assertEqual(main.z, 7)

def test_oversized_structure(self):
data = bytearray(b"\0" * 8)
for base in (LittleEndianStructure, BigEndianStructure):
class SomeBoolsTooBig(base):
_align_ = 8
_fields_ = [
("bool1", c_ubyte),
("bool2", c_ubyte),
("bool3", c_ubyte),
]
class Main(base):
_fields_ = [
("y", SomeBoolsTooBig),
("z", c_uint32),
]
with self.assertRaises(ValueError) as ctx:

serhiy-storchaka Jan 30, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

It is still not very informative test. If you want to test that the _align_ setting affects the size of Main, you can do this more directly by checking sizeof(Main). If you want to check also content, use the data of size 12.

monkeyman192 Jan 31, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I guess with this test I was just wanting to ensure that the appropriate error is raised when you have a Structure/Union with an alignment which will cause the size it expects to be bigger than what it is given. Rather than checking anything about the content itself.
Main.from_buffer(data)
self.assertEqual(
ctx.exception.args[0],
'Buffer size too small (4 instead of at least 8 bytes)'
)

def test_aligned_subclasses(self):
for base, e in (
(LittleEndianStructure, "<"),
(BigEndianStructure, ">"),
):
data = bytearray(struct.pack(f"{e}4i", 1, 2, 3, 4))
class UnalignedSub(base):
x: c_uint32
_fields_ = [
("x", c_uint32),
]

class AlignedStruct(UnalignedSub):
_align_ = 8
_fields_ = [
("y", c_uint32),
]

class Main(base):
_fields_ = [
("a", c_uint32),
("b", AlignedStruct)
]

main = Main.from_buffer(data)
self.assertEqual(alignment(main.b), 8)
self.assertEqual(alignment(main), 8)
self.assertEqual(sizeof(main.b), 8)
self.assertEqual(sizeof(main), 16)
self.assertEqual(main.a, 1)
self.assertEqual(main.b.x, 3)
self.assertEqual(main.b.y, 4)
self.assertEqual(Main.b.offset, 8)
self.assertEqual(Main.b.size, 8)

def test_aligned_union(self):
for sbase, ubase, e in (
(LittleEndianStructure, LittleEndianUnion, "<"),
(BigEndianStructure, BigEndianUnion, ">"),
):
data = bytearray(struct.pack(f"{e}4i", 1, 2, 3, 4))
class AlignedUnion(ubase):
_align_ = 8
_fields_ = [
("a", c_uint32),
("b", c_ubyte * 7),
]

class Main(sbase):
_fields_ = [
("first", c_uint32),
("union", AlignedUnion),
]

main = Main.from_buffer(data)
self.assertEqual(main.first, 1)
self.assertEqual(main.union.a, 3)
self.assertEqual(bytes(main.union.b), data[8:-1])
self.assertEqual(Main.union.offset, 8)
self.assertEqual(Main.union.size, 8)
self.assertEqual(alignment(main.union), 8)
self.assertEqual(alignment(main), 8)

def test_aligned_struct_in_union(self):
for sbase, ubase, e in (
(LittleEndianStructure, LittleEndianUnion, "<"),
(BigEndianStructure, BigEndianUnion, ">"),
):
data = bytearray(struct.pack(f"{e}4i", 1, 2, 3, 4))
class Sub(sbase):
_align_ = 8
_fields_ = [
("x", c_uint32),
("y", c_uint32),
]

class MainUnion(ubase):
_fields_ = [
("a", c_uint32),
("b", Sub),
]

class Main(sbase):
_fields_ = [
("first", c_uint32),
("union", MainUnion),
]

main = Main.from_buffer(data)
self.assertEqual(Main.first.size, 4)
self.assertEqual(alignment(main.union), 8)
self.assertEqual(alignment(main), 8)
self.assertEqual(Main.union.offset, 8)
self.assertEqual(Main.union.size, 8)
self.assertEqual(main.first, 1)
self.assertEqual(main.union.a, 3)
self.assertEqual(main.union.b.x, 3)
self.assertEqual(main.union.b.y, 4)

def test_smaller_aligned_subclassed_union(self):
for sbase, ubase, e in (
(LittleEndianStructure, LittleEndianUnion, "<"),
(BigEndianStructure, BigEndianUnion, ">"),
):
data = bytearray(struct.pack(f"{e}H2xI", 1, 0xD60102D7))
class SubUnion(ubase):
_align_ = 2
_fields_ = [
("unsigned", c_ubyte),
("signed", c_byte),
]

class MainUnion(SubUnion):
_fields_ = [
("num", c_uint32)
]

class Main(sbase):
_fields_ = [
("first", c_uint16),
("union", MainUnion),
]

main = Main.from_buffer(data)
self.assertEqual(main.union.num, 0xD60102D7)
self.assertEqual(main.union.unsigned, data[4])
self.assertEqual(main.union.signed, data[4] - 256)
self.assertEqual(alignment(main), 4)
self.assertEqual(alignment(main.union), 4)
self.assertEqual(Main.union.offset, 4)
self.assertEqual(Main.union.size, 4)
self.assertEqual(Main.first.size, 2)

def test_larger_aligned_subclassed_union(self):
for ubase, e in (
(LittleEndianUnion, "<"),
(BigEndianUnion, ">"),
):
data = bytearray(struct.pack(f"{e}I4x", 0xD60102D6))
class SubUnion(ubase):
_align_ = 8
_fields_ = [
("unsigned", c_ubyte),
("signed", c_byte),
]

class Main(SubUnion):
_fields_ = [
("num", c_uint32)
]

main = Main.from_buffer(data)
self.assertEqual(alignment(main), 8)
self.assertEqual(sizeof(main), 8)
self.assertEqual(main.num, 0xD60102D6)
self.assertEqual(main.unsigned, 0xD6)
self.assertEqual(main.signed, -42)

def test_aligned_packed_structures(self):
for sbase, e in (
(LittleEndianStructure, "<"),
(BigEndianStructure, ">"),
):
data = bytearray(struct.pack(f"{e}B2H4xB", 1, 2, 3, 4))

class Inner(sbase):
_align_ = 8
_fields_ = [
("x", c_uint16),
("y", c_uint16),
]

class Main(sbase):
_pack_ = 1
_fields_ = [
("a", c_ubyte),
("b", Inner),
("c", c_ubyte),
]

main = Main.from_buffer(data)
self.assertEqual(sizeof(main), 10)
self.assertEqual(Main.b.offset, 1)
# Alignment == 8 because _pack_ wins out.
self.assertEqual(alignment(main.b), 8)
# Size is still 8 though since inside this Structure, it will have
# effect.
self.assertEqual(sizeof(main.b), 8)
self.assertEqual(Main.c.offset, 9)
self.assertEqual(main.a, 1)
self.assertEqual(main.b.x, 2)
self.assertEqual(main.b.y, 3)
self.assertEqual(main.c, 4)


if __name__ == '__main__':
unittest.main()
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Core and Builtins/2024-01-28-02-46-12.gh-issue-112433.FUX-nT.rst
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add ability to force alignment of :mod:`ctypes.Structure` by way of the new ``_align_`` attribute on the class.
Loading
Back | FazBrowse Home | New Git URL