.. highlight:: c
.. _unicodeobjects:
Unicode Objects and Codecs
--------------------------
Unicode Objects
^^^^^^^^^^^^^^^
Since the implementation of :pep:`393` in Python 3.3, Unicode objects internally
use a variety of representations, in order to allow handling the complete range
of Unicode characters while staying memory efficient. There are special cases
for strings where all code points are below 128, 256, or 65536; otherwise, code
points must be below 1114112 (which is the full Unicode range).
UTF-8 representation is created on demand and cached in the Unicode object.
.. note::
The :c:type:`Py_UNICODE` representation has been removed since Python 3.12
with deprecated APIs.
See :pep:`623` for more information.
Unicode Type
""""""""""""
These are the basic Unicode object types used for the Unicode implementation in
Python:
.. c:var:: PyTypeObject PyUnicode_Type
This instance of :c:type:`PyTypeObject` represents the Python Unicode type.
It is exposed to Python code as :py:class:`str`.
.. c:var:: PyTypeObject PyUnicodeIter_Type
This instance of :c:type:`PyTypeObject` represents the Python Unicode
iterator type. It is used to iterate over Unicode string objects.
.. c:type:: Py_UCS4
Py_UCS2
Py_UCS1
These types are typedefs for unsigned integer types wide enough to contain
characters of 32 bits, 16 bits and 8 bits, respectively. When dealing with
single Unicode characters, use :c:type:`Py_UCS4`.
.. versionadded:: 3.3
.. c:type:: PyASCIIObject
PyCompactUnicodeObject
PyUnicodeObject
These subtypes of :c:type:`PyObject` represent a Python Unicode object. In
almost all cases, they shouldn't be used directly, since all API functions
that deal with Unicode objects take and return :c:type:`PyObject` pointers.
.. versionadded:: 3.3
The structure of a particular object can be determined using the following
macros.
The macros cannot fail; their behavior is undefined if their argument
is not a Python Unicode object.
.. c:namespace:: NULL
.. c:macro:: PyUnicode_IS_COMPACT(o)
True if *o* uses the :c:struct:`PyCompactUnicodeObject` structure.
.. versionadded:: 3.3
.. c:macro:: PyUnicode_IS_COMPACT_ASCII(o)
True if *o* uses the :c:struct:`PyASCIIObject` structure.
.. versionadded:: 3.3
The following APIs are C macros and static inlined functions for fast checks and
access to internal read-only data of Unicode objects:
.. c:function:: int PyUnicode_Check(PyObject *obj)
Return true if the object *obj* is a Unicode object or an instance of a Unicode
subtype. This function always succeeds.
.. c:function:: int PyUnicode_CheckExact(PyObject *obj)
Return true if the object *obj* is a Unicode object, but not an instance of a
subtype. This function always succeeds.
.. c:function:: Py_ssize_t PyUnicode_GET_LENGTH(PyObject *unicode)
Return the length of the Unicode string, in code points. *unicode* has to be a
Unicode object in the "canonical" representation (not checked).
.. versionadded:: 3.3
.. c:function:: Py_UCS1* PyUnicode_1BYTE_DATA(PyObject *unicode)
Py_UCS2* PyUnicode_2BYTE_DATA(PyObject *unicode)
Py_UCS4* PyUnicode_4BYTE_DATA(PyObject *unicode)
Return a pointer to the canonical representation cast to UCS1, UCS2 or UCS4
integer types for direct character access. No checks are performed if the
canonical representation has the correct character size; use
:c:func:`PyUnicode_KIND` to select the right function.
.. versionadded:: 3.3
.. c:macro:: PyUnicode_1BYTE_KIND
PyUnicode_2BYTE_KIND
PyUnicode_4BYTE_KIND
Return values of the :c:func:`PyUnicode_KIND` macro.
.. versionadded:: 3.3
.. versionchanged:: 3.12
``PyUnicode_WCHAR_KIND`` has been removed.
.. c:function:: int PyUnicode_KIND(PyObject *unicode)
Return one of the PyUnicode kind constants (see above) that indicate how many
bytes per character this Unicode object uses to store its data. *unicode* has to
be a Unicode object in the "canonical" representation (not checked).
.. versionadded:: 3.3
.. c:function:: void* PyUnicode_DATA(PyObject *unicode)
Return a void pointer to the raw Unicode buffer. *unicode* has to be a Unicode
object in the "canonical" representation (not checked).
.. versionadded:: 3.3
.. c:function:: void PyUnicode_WRITE(int kind, void *data, \
Py_ssize_t index, Py_UCS4 value)
Write the code point *value* to the given zero-based *index* in a string.
The *kind* value and *data* pointer must have been obtained from a
string using :c:func:`PyUnicode_KIND` and :c:func:`PyUnicode_DATA`
respectively. You must hold a reference to that string while calling
:c:func:`!PyUnicode_WRITE`. All requirements of
:c:func:`PyUnicode_WriteChar` also apply.
The function performs no checks for any of its requirements,
and is intended for usage in loops.
.. versionadded:: 3.3
.. c:function:: Py_UCS4 PyUnicode_READ(int kind, void *data, \
Py_ssize_t index)
Read a code point from a canonical representation *data* (as obtained with
:c:func:`PyUnicode_DATA`). No checks or ready calls are performed.
.. versionadded:: 3.3
.. c:function:: Py_UCS4 PyUnicode_READ_CHAR(PyObject *unicode, Py_ssize_t index)
Read a character from a Unicode object *unicode*, which must be in the "canonical"
representation. This is less efficient than :c:func:`PyUnicode_READ` if you
do multiple consecutive reads.
.. versionadded:: 3.3
.. c:function:: Py_UCS4 PyUnicode_MAX_CHAR_VALUE(PyObject *unicode)
Return the maximum code point that is suitable for creating another string
based on *unicode*, which must be in the "canonical" representation. This is
always an approximation but more efficient than iterating over the string.
.. versionadded:: 3.3
.. c:function:: int PyUnicode_IsIdentifier(PyObject *unicode)
Return ``1`` if the string is a valid identifier according to the language
definition, section :ref:`identifiers`. Return ``0`` otherwise.
.. versionchanged:: 3.9
The function does not call :c:func:`Py_FatalError` anymore if the string
is not ready.
.. c:function:: unsigned int PyUnicode_IS_ASCII(PyObject *unicode)
Return true if the string only contains ASCII characters.
Equivalent to :py:meth:`str.isascii`.
.. versionadded:: 3.2
.. c:function:: Py_hash_t PyUnstable_Unicode_GET_CACHED_HASH(PyObject *str)
If the hash of *str*, as returned by :c:func:`PyObject_Hash`, has been
cached and is immediately available, return it.
Otherwise, return ``-1`` *without* setting an exception.
If *str* is not a string (that is, if ``PyUnicode_Check(obj)``
is false), the behavior is undefined.
This function never fails with an exception.
Note that there are no guarantees on when an object's hash is cached,
and the (non-)existence of a cached hash does not imply that the string has
any other properties.
Unicode Character Properties
""""""""""""""""""""""""""""
Unicode provides many different character properties. The most often needed ones
are available through these macros which are mapped to C functions depending on
the Python configuration.
.. c:function:: int Py_UNICODE_ISSPACE(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is a whitespace character.
.. c:function:: int Py_UNICODE_ISLOWER(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is a lowercase character.
.. c:function:: int Py_UNICODE_ISUPPER(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is an uppercase character.
.. c:function:: int Py_UNICODE_ISTITLE(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is a titlecase character.
.. c:function:: int Py_UNICODE_ISLINEBREAK(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is a linebreak character.
.. c:function:: int Py_UNICODE_ISDECIMAL(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is a decimal character.
.. c:function:: int Py_UNICODE_ISDIGIT(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is a digit character.
.. c:function:: int Py_UNICODE_ISNUMERIC(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is a numeric character.
.. c:function:: int Py_UNICODE_ISALPHA(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is an alphabetic character.
.. c:function:: int Py_UNICODE_ISALNUM(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is an alphanumeric character.
.. c:function:: int Py_UNICODE_ISPRINTABLE(Py_UCS4 ch)
Return ``1`` or ``0`` depending on whether *ch* is a printable character,
in the sense of :meth:`str.isprintable`.
These APIs can be used for fast direct character conversions:
.. c:function:: Py_UCS4 Py_UNICODE_TOLOWER(Py_UCS4 ch)
Return the character *ch* converted to lower case.
.. c:function:: Py_UCS4 Py_UNICODE_TOUPPER(Py_UCS4 ch)
Return the character *ch* converted to upper case.
.. c:function:: Py_UCS4 Py_UNICODE_TOTITLE(Py_UCS4 ch)
Return the character *ch* converted to title case.
.. c:function:: int Py_UNICODE_TODECIMAL(Py_UCS4 ch)
Return the character *ch* converted to a decimal positive integer. Return
``-1`` if this is not possible. This function does not raise exceptions.
.. c:function:: int Py_UNICODE_TODIGIT(Py_UCS4 ch)
Return the character *ch* converted to a single digit integer. Return ``-1`` if
this is not possible. This function does not raise exceptions.
.. c:function:: double Py_UNICODE_TONUMERIC(Py_UCS4 ch)
Return the character *ch* converted to a double. Return ``-1.0`` if this is not
possible. This function does not raise exceptions.
These APIs can be used to work with surrogates:
.. c:function:: int Py_UNICODE_IS_SURROGATE(Py_UCS4 ch)
Check if *ch* is a surrogate (``0xD800