FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-131510: Deprecate PyUnicodeObject structs, revert memory layout by encukou · Pull Request #130697 · python/cpython · GitHub

Repository navigation

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension .c  (2) .h  (3) .py  (3) .rst  (4) All 4 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
139 changes: 128 additions & 11 deletions Doc/c-api/unicode.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -48,17 +48,6 @@ Python:
.. versionadded:: 3.3


.. c:type:: PyASCIIObject
PyCompactUnicodeObject
PyUnicodeObject

These subtypes of :c:type:`PyObject` represent a Python Unicode object. In
almost all cases, they shouldn't be used directly, since all API functions
that deal with Unicode objects take and return :c:type:`PyObject` pointers.

.. versionadded:: 3.3


The following APIs are C macros and static inlined functions for fast checks and
access to internal read-only data of Unicode objects:

Expand Down Expand Up @@ -1832,6 +1821,97 @@ Deprecated API

The following API is deprecated.

.. c:type:: PyASCIIObject
PyCompactUnicodeObject
PyUnicodeObject

These subtypes of :c:type:`PyObject` represent a Python Unicode object. In
almost all cases, they shouldn't be used directly, since all API functions
that deal with Unicode objects take and return :c:type:`PyObject` pointers.

.. versionadded:: 3.3

.. deprecated-removed:: next 3.16

These structures will be removed from the public API to allow updating the
internal layout.

There is no direct replacement for writing to the deprecated ``struct``
fields, since strings are conceptually immutable.
See :c:func:`PyUnicode_New` for creating new strings.

For reading the fields, the following table gives (up to) two options.

The **macro** replacements are the closest public API for reading a field
directly. The macros are fast, but their performance is not guaranteed for
future versions of CPython.

The **function** replacements are part of the
:ref:`limited API <limited-c-api>`, and available as exported functions.
Unlike with direct field access, these functions will compute requested
information is not already available internally. Also, the functions
may signal an error (return ``-1`` or a ``NULL`` pointer with an exception
set); callers must check for this case.

The replacements are one-argument functions and macros, unless noted
otherwise.

.. list-table::
:widths: auto
:header-rows: 1

* * Member
* Macro replacement
* Function replacement
* * ``length``
* :c:func:`PyUnicode_GET_LENGTH`
* :c:func:`PyObject_Length`
* * ``hash``
* :c:func:`PyUnstable_Unicode_GET_CACHED_HASH`
* :c:func:`PyObject_Hash`
* * ``interned``
* :c:func:`PyUnicode_CHECK_INTERNED`
* N/A \(1)
* * ``kind``
* :c:func:`PyUnicode_KIND`
* N/A
* * ``compact``
* :c:func:`PyUnicode_IS_COMPACT` \(2)
* N/A
* * ``ascii``
* :c:func:`PyUnicode_IS_ASCII`
* :py:func:`str.isascii` in Python
* * ``statically_allocated``
* N/A
* N/A
* * ``latin1``, ``ucs2``, ``ucs4``
* :c:func:`PyUnicode_DATA`
* N/A
* * ``utf8``, ``utf8_length``
* N/A
* :c:func:`PyUnicode_AsUTF8AndSize`
(A second argument allows retrieving ``utf8_length``.)

\(1) Note that it is not necessary to check the *interned* field
before a :c:func:`PyUnicode_InternInPlace` call; the function does nothing
if the string is already interned.

\(2) :c:func:`PyUnicode_IS_COMPACT` is also deprecated, see its docs below.

.. c:function:: unsigned int PyUnicode_IS_COMPACT(PyObject *unicode)

Return true if the string object uses a compact internal representation
(the :c:type:`PyCompactUnicodeObject` or :c:type:`PyASCIIObject` struct,
as opposed to :c:type:`PyUnicodeObject`).

These structs themselves are deprecated. Replacements suggested in their
documentation will work with *any* internal representation.

.. versionadded:: 3.3

.. deprecated-removed:: next 3.16


.. c:type:: Py_UNICODE

This is a typedef of :c:type:`wchar_t`, which is a 16-bit type or 32-bit type
Expand Down Expand Up @@ -1872,3 +1952,40 @@ The following API is deprecated.
This API does nothing since Python 3.12.
Previously, this could be called to check if
:c:func:`PyUnicode_READY` is necessary.


Unstable API
^^^^^^^^^^^^

.. c:type:: PyUnstable_ASCIIObject
PyUnstable_CompactUnicodeObject
PyUnstable_UnicodeObject

Aliases of the deprecated :c:type:`PyASCIIObject`,
:c:type:`PyCompactUnicodeObject` and :c:type:`PyUnicodeObject`.

Use these names to avoid deprecation warnings at build time,
when necessary for advanced use cases such as subclassing :py:type:`str`.

Like all unstable API, the structures may be removed in a feature release
of Python (such as 3.16.0).
Users are encouraged to test with CPython pre-releases and use CPython
development channels to discuss replacement APIs.

.. versionadded:: next


.. c:function:: Py_hash_t PyUnstable_Unicode_GET_CACHED_HASH(PyObject *str)

Return the hash of *str*, as with :c:func:`PyObject_Hash`, if the hash
has been cached and is immediately available.
Otherwise, return ``-1`` *without* setting an exception.

The *str* argument must be a string (:c:expr:`PyUnicode_Check(str)`
must be true). This is not checked.

This function never fails with an exception.

Note that there are no guarantees on when a Unicode object's hash is cached.

.. versionadded:: next
12 changes: 12 additions & 0 deletions Doc/deprecations/c-api-pending-removal-in-3.16.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Pending removal in Python 3.16
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* C structures for :py:type:`str` instances, and a macro for querying a
string's memory layout:

* :c:type:`PyASCIIObject`
* :c:type:`PyCompactUnicodeObject`
* :c:type:`PyUnicodeObject`
* :c:type:`PyUnicode_IS_COMPACT`

See their documentation for possible replacements.
27 changes: 27 additions & 0 deletions Doc/whatsnew/3.14.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -1647,6 +1647,17 @@ New features
take a C integer and produce a Python :class:`bool` object. (Contributed by
Pablo Galindo in :issue:`45325`.)

* Add unstable API for ``PyUnicode`` objects, for advanced use cases that
require complex migration from deprecated structures:

* :c:type:`PyUnstable_ASCIIObject`
* :c:type:`PyUnstable_CompactUnicodeObject`
* :c:type:`PyUnstable_UnicodeObject`
* :c:type:`PyUnstable_Unicode_GET_CACHED_HASH`

See their documentation for caveats.
(Contributed by Petr Viktorin in :gh:`131510`)


Limited C API changes
---------------------
Expand Down Expand Up @@ -1773,10 +1784,26 @@ Deprecated

The `pythoncapi-compat project`_ can be used to get these new public
functions on Python 3.13 and older.

(Contributed by Victor Stinner in :gh:`128863`.)


* C structures for :py:type:`str` instances, and a macro for querying a
string's memory layout, are deprecated and planned for
removal in Python 3.16:

* :c:type:`PyASCIIObject`
* :c:type:`PyCompactUnicodeObject`
* :c:type:`PyUnicodeObject`
* :c:type:`PyUnicode_IS_COMPACT`

See their documentation for possible replacements.


.. include:: ../deprecations/c-api-pending-removal-in-3.15.rst

.. include:: ../deprecations/c-api-pending-removal-in-3.16.rst

.. include:: ../deprecations/c-api-pending-removal-in-3.18.rst

.. include:: ../deprecations/c-api-pending-removal-in-future.rst
Expand Down
76 changes: 52 additions & 24 deletions Include/cpython/unicodeobject.h
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,11 @@ typedef struct {
PyObject_HEAD
Py_ssize_t length; /* Number of code points in the string */
Py_hash_t hash; /* Hash value; -1 if not set */
#ifdef Py_GIL_DISABLED
/* Ensure 4 byte alignment for PyUnicode_DATA(), see gh-63736 on m68k.
In the non-free-threaded build, we'll use explicit padding instead */
_Py_ALIGN_AS(4)
#endif
struct {
/* If interned is non-zero, the two references from the
dictionary to this object are *not* counted in ob_refcnt.
Expand All @@ -109,7 +114,12 @@ typedef struct {
3: Interned, Immortal, and Static
This categorization allows the runtime to determine the right
cleanup mechanism at runtime shutdown. */
uint16_t interned;
#ifdef Py_GIL_DISABLED
// Needs to be accessed atomically, so can't be a bit field.
unsigned char interned;
#else
unsigned int interned:2;
#endif
/* Character size:

- PyUnicode_1BYTE_KIND (1):
Expand All @@ -132,62 +142,69 @@ typedef struct {
* all characters are in the range U+0000-U+10FFFF
* at least one character is in the range U+10000-U+10FFFF
*/
unsigned short kind:3;
unsigned int kind:3;
/* Compact is with respect to the allocation scheme. Compact unicode
objects only require one memory block while non-compact objects use
one block for the PyUnicodeObject struct and another for its data
buffer. */
unsigned short compact:1;
unsigned int compact:1;
/* The string only contains characters in the range U+0000-U+007F (ASCII)
and the kind is PyUnicode_1BYTE_KIND. If ascii is set and compact is
set, use the PyASCIIObject structure. */
unsigned short ascii:1;
unsigned int ascii:1;
/* The object is statically allocated. */
unsigned short statically_allocated:1;
unsigned int statically_allocated:1;
#ifndef Py_GIL_DISABLED
/* Padding to ensure that PyUnicode_DATA() is always aligned to
4 bytes (see issue #19537 on m68k) and we use unsigned short to avoid
the extra four bytes on 32-bit Windows. This is restricted features
for specific compilers including GCC, MSVC, Clang and IBM's XL compiler. */
unsigned short :10;
4 bytes (see issue gh-63736 on m68k) */
unsigned int :24;
#endif
} state;
} PyASCIIObject;
} PyUnstable_ASCIIObject;

/* Non-ASCII strings allocated through PyUnicode_New use the
PyCompactUnicodeObject structure. state.compact is set, and the data
immediately follow the structure. */
typedef struct {
PyASCIIObject _base;
PyUnstable_ASCIIObject _base;
Py_ssize_t utf8_length; /* Number of bytes in utf8, excluding the
* terminating \0. */
char *utf8; /* UTF-8 representation (null-terminated) */
} PyCompactUnicodeObject;
} PyUnstable_CompactUnicodeObject;

/* Object format for Unicode subclasses. */
typedef struct {
PyCompactUnicodeObject _base;
PyUnstable_CompactUnicodeObject _base;
union {
void *any;
Py_UCS1 *latin1;
Py_UCS2 *ucs2;
Py_UCS4 *ucs4;
} data; /* Canonical, smallest-form Unicode buffer */
} PyUnicodeObject;
} PyUnstable_UnicodeObject;


_Py_DEPRECATED_EXTERNALLY(3.14) typedef PyUnstable_ASCIIObject
PyASCIIObject;
_Py_DEPRECATED_EXTERNALLY(3.14) typedef PyUnstable_CompactUnicodeObject
PyCompactUnicodeObject;
_Py_DEPRECATED_EXTERNALLY(3.14) typedef PyUnstable_UnicodeObject
PyUnicodeObject;

#define _PyASCIIObject_CAST(op) \
(assert(PyUnicode_Check(op)), \
_Py_CAST(PyASCIIObject*, (op)))
_Py_CAST(PyUnstable_ASCIIObject*, (op)))
#define _PyCompactUnicodeObject_CAST(op) \
(assert(PyUnicode_Check(op)), \
_Py_CAST(PyCompactUnicodeObject*, (op)))
_Py_CAST(PyUnstable_CompactUnicodeObject*, (op)))
#define _PyUnicodeObject_CAST(op) \
(assert(PyUnicode_Check(op)), \
_Py_CAST(PyUnicodeObject*, (op)))
_Py_CAST(PyUnstable_UnicodeObject*, (op)))


/* --- Flexible String Representation Helper Macros (PEP 393) -------------- */

/* Values for PyASCIIObject.state: */
/* Values for _PyASCIIObject.state: */

/* Interning state. */
#define SSTATE_NOT_INTERNED 0
Expand All @@ -198,7 +215,7 @@ typedef struct {
/* Use only if you know it's a string */
static inline unsigned int PyUnicode_CHECK_INTERNED(PyObject *op) {
#ifdef Py_GIL_DISABLED
return _Py_atomic_load_uint16_relaxed(&_PyASCIIObject_CAST(op)->state.interned);
return _Py_atomic_load_uint8_relaxed(&_PyASCIIObject_CAST(op)->state.interned);
#else
return _PyASCIIObject_CAST(op)->state.interned;
#endif
Expand All @@ -220,15 +237,20 @@ static inline unsigned int PyUnicode_IS_ASCII(PyObject *op) {

/* Return true if the string is compact or 0 if not.
No type checks are performed. */
static inline unsigned int PyUnicode_IS_COMPACT(PyObject *op) {
static inline unsigned int _PyUnicode_IS_COMPACT(PyObject *op) {
return _PyASCIIObject_CAST(op)->state.compact;
}
_Py_DEPRECATED_EXTERNALLY(3.14)
static inline unsigned int PyUnicode_IS_COMPACT(PyObject *op) {
return _PyUnicode_IS_COMPACT(op);
}
#define PyUnicode_IS_COMPACT(op) PyUnicode_IS_COMPACT(_PyObject_CAST(op))

/* Return true if the string is a compact ASCII string (use PyASCIIObject
/* Return true if the string is a compact ASCII string (use _PyASCIIObject
structure), or 0 if not. No type checks are performed. */
_Py_DEPRECATED_EXTERNALLY(3.14)
static inline int PyUnicode_IS_COMPACT_ASCII(PyObject *op) {
return (_PyASCIIObject_CAST(op)->state.ascii && PyUnicode_IS_COMPACT(op));
return (_PyASCIIObject_CAST(op)->state.ascii && _PyUnicode_IS_COMPACT(op));
}
#define PyUnicode_IS_COMPACT_ASCII(op) PyUnicode_IS_COMPACT_ASCII(_PyObject_CAST(op))

Expand Down Expand Up @@ -259,7 +281,7 @@ static inline void* _PyUnicode_COMPACT_DATA(PyObject *op) {

static inline void* _PyUnicode_NONCOMPACT_DATA(PyObject *op) {
void *data;
assert(!PyUnicode_IS_COMPACT(op));
assert(!_PyUnicode_IS_COMPACT(op));
data = _PyUnicodeObject_CAST(op)->data.any;
assert(data != NULL);
return data;
Expand All @@ -268,7 +290,7 @@ static inline void* _PyUnicode_NONCOMPACT_DATA(PyObject *op) {
PyAPI_FUNC(void*) PyUnicode_DATA(PyObject *op);

static inline void* _PyUnicode_DATA(PyObject *op) {
if (PyUnicode_IS_COMPACT(op)) {
if (_PyUnicode_IS_COMPACT(op)) {
return _PyUnicode_COMPACT_DATA(op);
}
return _PyUnicode_NONCOMPACT_DATA(op);
Expand All @@ -290,6 +312,12 @@ static inline Py_ssize_t PyUnicode_GET_LENGTH(PyObject *op) {
}
#define PyUnicode_GET_LENGTH(op) PyUnicode_GET_LENGTH(_PyObject_CAST(op))

/* Returns the cached hash, or -1 if not cached yet. */
static inline Py_hash_t
PyUnstable_Unicode_GET_CACHED_HASH(PyObject *op) {
return _PyASCIIObject_CAST(op)->hash;
}

/* Write into the canonical representation, this function does not do any sanity
checks and is intended for usage in loops. The caller should cache the
kind and data pointers obtained from other function calls.
Expand Down
Loading

Back | FazBrowse Home | New Git URL