FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

The C-API for Python to C integer conversion is, to be frank, a mess. · Issue #102471 · python/cpython · GitHub

  1. python
  2. cpython

Repository navigation

The C-API for Python to C integer conversion is, to be frank, a mess. #102471

Open
Open
The C-API for Python to C integer conversion is, to be frank, a mess.#102471
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)type-featureA feature request or enhancement

Description

markshannon
opened on Mar 6, 2023

The C-API has built up over 30 years, in a haphazard way. So, it is no surprise that it is a bit of a mess.
What makes it worse is that it is based around the C long type, which is varies in size between architectures and operating systems in odd ways.
C longs are 32 bit on (almost?) all 32 bit machines, 64 bit on most 64 bit machines, except Windows when C longs are 32 bits on 64 bit machines. In other words, it is not a useful fixed size, like int32_t, nor does match the machine word size, like intptr_t.

We need a more consistent API for converting from Python integers to C integers and back again.
We should support both 32 bit and word size C integers. 32 bit, because we often want to store 32 bit values to save space on 64 bit machines, or for portability. We also want to support word size integers for performance and ease of coding.

This means we want 4 functions (2 sizes, 2 directions) to convert between C and Python integers.

Currently we have:

Width Py -> C C -> Py
32 bit Missing* Missing
Machine word Missing* PyLong_FromSsize_t

The C API has a function to convert Python ints to intptr_t, but it is missing efficient overflow handling.
It also has a function with efficient overflow handling, PyLong_AsLongAndOverflow, but that returns a long.

Here's what we want:

Width Py -> C C -> Py
32 bit PyInt_AsInt32 PyInt_FromInt32
Machine word PyInt_AsSsize_t PyInt_FromSsize_t

I'm using PyInt prefix, now that Python 2 is history. It makes it clearer what is the new API.

Note that I'm not handling unsigned values. I think the extra bit of precision is not worth the complexity of a larger API.
And if we decide that they are, we can always add them later.

Linked PRs

Activity

  1. markshannon
    added
    type-featureA feature request or enhancement
    on Mar 6, 2023

  2. markshannon commented on Jul 4, 2023

    markshannon
    on Jul 4, 2023
    MemberAuthor

    We also need a few functions for querying and extracting the value of a Python int.

    We want to query its sign:

    int PyInt_IsNegative();
int PyInt_IsPositive();
int PyInt_IsZero();
int PyInt_Sign();

    We want to import and export the digits of an integer, and to know how many digits there are.
    GNU's MP library has mpz_import and mpz_export, which have quite a complex API, but might be a good model to use.
    In addition we should provide a constant describing the "native" number of bits per digit, so that C extensions can extract the data efficiently.

  3. markshannon commented on Jul 4, 2023

    markshannon
    on Jul 4, 2023
    MemberAuthor

    mpz_import and mpz_export take 6 parameters each, and four of those are small numbers describing the layout. Having many int parameters is hard to read and error-prone. We should combine the layout parameters into a single struct (of 32 bits or less).

    E.g.

    typedef struct _PyIntExportLayout {
     uint8_t bits_per_digit,
     int8_t word_endian,
     int8_t array_endian,
     uint8_t digit_size,
} PyIntExportLayout;

PyLongObject *PyInt_Import(PyIntExportLayout layout, size_t count, const void *data);
int PyInt_Import(PyLongObject *op, PyIntExportLayout layout, size_t count, void *data);
size_t PyInt_DigitCount(PyLongObject *op, uint8_t bits_per_digit);
const PyIntExportLayout PY_INT_NATIVE_LAYOUT; /* Use this when possible, for speed */

  4. casevh commented on Oct 28, 2023

    casevh
    on Oct 28, 2023

    Hi. I'm the primary maintainer of gmpy2. I'd like to provide some comments with my experiences using the C-API.

    I use PyLong_AsLongAndOverflow when I want a long value or immediately proceed with the full conversion of PyLong to mpz as quickly as possible. Avoiding the exception is a significant performance improvement. PyLong_AsUnsignedLongAndOverflow is used occasionally when GMP is expects an unsigned long.

    PyLong_As[Unsigned]LongLongAndOverflow were used with MPIR to get 64-bit values on Windows. (MPIR extended GMP to support 64-bit native integer sizes.) gmpy2 doesn't currently use them but it would be nice if they could be kept.

    I like your PyIntExportLayout idea for specifying the . I have a question about the usage of PyInt_Import - which side owns the conversion?

    Is PyInt_Import intended to access external data (i.e. the mpz data) and create a PyLong? Does PyIntExportLayout then specify the format of the mpz data?

    Would there be a corresponding PyInt_Export that exports the value of a PyLong into an external buffer with the format of the external buffer controlled by PyIntExportLayout? If so, who owns (CPython versus gmpy2) the memory allocated to the external buffer? (Note: GMP, MPFR, and MPC can use a different memory manager than CPython....)

    This is reversed from the current conversion direction. For mpz to PyLong, gmpy2 asks CPython to create a new PyLong with sufficient space to store the output of mpz_export. And for PyLong to mpz, gmpy2 creates a new mpz with sufficient space to store the output of mpz_import.

    I'll add another comment to the thread about the compact format.

    Thanks for all the effort in improving CPython.

    casevh

  5. iritkatriel
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Nov 27, 2023
  6. vstinner
    added 2 commits that reference this issue on Jul 3, 2024

    pythongh-102471: Add PyLong import and export API

    21dd5b9

    pythongh-102471: Add PyLong import and export API

    a087b0a

  7. vstinner commented on Jul 4, 2024

    vstinner
    on Jul 4, 2024
    Member

    32 bit PyInt_AsInt32 PyInt_FromInt32

    I created #120390 for that.

  8. vstinner
    added a commit that references this issue on Jul 4, 2024

    pythongh-102471: Add PyLong import and export API

    d3532a4

  9. serhiy-storchaka commented on Jul 5, 2024

    serhiy-storchaka
    on Jul 5, 2024
    Member

    I had plans to add PyLong_Import() and PyLong_Export() with GMP/libtommath inspired signatures. This is too general interface which allows to support many different representations.

  10. skirpichev commented on Jul 5, 2024

    skirpichev
    on Jul 5, 2024
    Member

    This is too general interface which allows to support many different representations.

    This is relatively complex task, which is better suited to dedicated libraries. I would be rather surprised if some arbitrary precision math library lacks mpz_import/export-like functions.

    If on CPython side we will have a "view" of integers as an array of digits - the rest of work could do any math library.

  11. serhiy-storchaka commented on Jul 5, 2024

    serhiy-storchaka
    on Jul 5, 2024
    Member

    Then please used different names than PyLong_Import()/PyLong_Export().

  12. vstinner
    added a commit that references this issue on Jul 8, 2024

    pythongh-102471: Add PyLong import and export API

    51f48fe
  13. vstinner
    added a commit that references this issue on Aug 6, 2024

    pythongh-102471: Add PyLong import and export API

    f4fdbf2

  14. vstinner commented on Sep 13, 2024

    vstinner
    on Sep 13, 2024
    Member

    We need a more consistent API for converting from Python integers to C integers and back again.
    We should support both 32 bit and word size C integers. 32 bit, because we often want to store 32 bit values to save space on 64 bit machines, or for portability. We also want to support word size integers for performance and ease of coding.

    I added APIs for that with 4c6dca8:

    • Signed:
      • PyLong_FromInt32(), PyLong_FromInt64()
      • PyLong_AsInt32(), PyLong_AsInt64()
    • Unsigned:
      • PyLong_FromUInt32(), PyLong_FromUInt64()
      • PyLong_AsUInt32(), PyLong_AsUInt64()

  15. vstinner commented on Sep 13, 2024

    vstinner
    on Sep 13, 2024
    Member

    We want to query its sign:
    int PyInt_Sign();

    PyLong_GetSign() was added to Python 3.14: https://docs.python.org/dev/c-api/long.html#c.PyLong_GetSign

    int PyInt_IsNegative();
    int PyInt_IsPositive();
    int PyInt_IsZero();

    There is an open discussion for these functions: capi-workgroup/decisions#29

  16. skirpichev commented on Nov 14, 2024

    skirpichev
    on Nov 14, 2024
    Member

    PyLong_IsPositive(), PyLong_IsNegative() and PyLong_IsZero() were added to Python 3.14: https://docs.python.org/dev/c-api/long.html#c.PyLong_IsPositive

  17. vstinner
    added a commit that references this issue on Dec 13, 2024

    gh-102471, PEP 757: Add PyLong import and export API (#121339)

    6446408
  18. skirpichev
    added a commit that references this issue on Dec 13, 2024

    pythongh-102471: convert decimal module to use PyLongWriter API (PEP …

    80f1a04
  19. skirpichev
    added a commit that references this issue on Dec 26, 2024

    pythongh-102471: convert decimal module to use PyLong_Export API (PEP…

    f7f031f
  20. vstinner
    added a commit that references this issue on Jan 6, 2025

    gh-102471: convert decimal module to use PyLong_Export API (PEP 757) (#…

    879d287
  21. srinivasreddy
    added 2 commits that reference this issue on Jan 8, 2025

    pythongh-102471, PEP 757: Add PyLong import and export API (python#12…

    61c48fa

    pythongh-102471: convert decimal module to use PyLong_Export API (PEP…

    5d8a701
  22. vstinner
    added 5 commits that reference this issue on Jan 27, 2025

    pythongh-102471: Credit Sergey B Kirpichev for PEP 757

    e4c05bb

    pythongh-102471: Change PyLongWriter_Discard(NULL) to do nothing

    b623660

    pythongh-102471: Change PyLongWriter_Discard(NULL) to do nothing

    e3ed89b

    gh-102471: Credit Sergey B Kirpichev for PEP 757 (#129338)

    11148e9

    gh-102471: Change PyLongWriter_Discard(NULL) to do nothing (#129339)

    7ec1742
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)type-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      Back | FazBrowse Home | New Git URL