C API: Detect invalid UCS4 characters in debug mode · Issue #158445 · python/cpython · GitHub

Currently, PyUnicode_FromKindAndData() and PyUnicodeWriter_WriteUCS4() create an invalid string if a character is not in the [U+0000; U+10ffff] range. Instead, I propose to raise an exception in this case.

Example creating an invalid string:

$ python3.15
>>> import _testcapi, sys
>>> invalid_ucs4 = (0x11_0000).to_bytes(4, byteorder=sys.byteorder)
>>> s = _testcapi.unicode_fromkindanddata(4, invalid_ucs4*5)

>>> list(s)
SystemError: invalid maximum character passed to PyUnicode_New

>>> s[0]
SystemError: invalid maximum character passed to PyUnicode_New

Linked PRs

With 32-bit wchar_t (ex: Unix), PyUnicode_FromWideChar() and PyUnicodeWriter_WriteWideChar() already raise a ValueError if a character is outside the [U+0000; U+10ffff] range.

pythongh-158445: Reject invalid UCS4 in PyUnicode_FromKindAndData()

Other Python and C functions raising an exception if the code pointer is negative or is greater than 0x10ffff:

Python chr() raises ValueError.
PyUnicode_FromOrdinal(), PyUnicode_Fill() and PyUnicodeWriter_WriteChar() raise ValueError.
PyUnicode_FromFormat("%c") raises OverflowError.
PyUnicode_New() raises SystemError.

PyUnicode_WriteChar() raises ValueError if the character is greater than the string maximum character.

pythongh-158445: Detect invalid UCS4 characters in debug mode

gh-158445: Detect invalid UCS4 characters in debug mode (#158502)

Since checking for invalid characters at runtime was rejected, I repurpose the issue to "Detect invalid UCS4 characters in debug mode".

This broke several buildbots.

gh-158445: Add C API tests on build information functions (#158603)

This broke several buildbots.

Oh, I didn't get buildbot failures report on my PR, so I didn't notice. Thanks for the notification.

So test_capi fails on 2 buildbots: Python is built in release mode with assertion using ./configure (...) CFLAGS=-UNDEBUG

ARM64 Raspbian 3.x: https://buildbot.python.org/#/builders/1678/builds/3888
x86 Debian Non-Debug with X 3.x: https://buildbot.python.org/#/builders/1245/builds/9676

I wrote a first PR #158603 to fix support.built_with_c_assertions() (merged). Previously, -UNDEBUG compiler flag was not handled properly.

I wrote a second PR #158604 to fix test_capi.test_unicode.

gh-158445: Skip invalid UCS-4 tests if Python is built with assertions (

My latest change should fix the 2 buildbots.

Thank you!

gh-158445: Allocate memory in the heap in Py_GetVersion() (#158608)

ARM64 Raspbian 3.x: https://buildbot.python.org/#/builders/1678/builds/3888
x86 Debian Non-Debug with X 3.x: https://buildbot.python.org/#/builders/1245/builds/9676

Both workers are back to green.

C API: Detect invalid UCS4 characters in debug mode · Issue #158445 · python/cpython · GitHub

Repository navigation

C API: Detect invalid UCS4 characters in debug mode #158445

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

C API: Detect invalid UCS4 characters in debug mode · Issue #158445 · python/cpython · GitHub

Repository navigation

C API: Detect invalid UCS4 characters in debug mode #158445

Description

Linked PRs

Activity

vstinner commented on Sep 29, 2026

vstinner commented on Sep 29, 2026

vstinner commented on Sep 30, 2026

encukou commented on Oct 2, 2026

vstinner commented on Oct 2, 2026

vstinner commented on Oct 2, 2026

encukou commented on Oct 2, 2026

vstinner commented on Oct 3, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions