Repository navigation
PEP-739 build-details generation is incorrect when cross-building #136267
Description
Bug report
Bug description:
When doing cross-builds, the PEP-739 build-details.json gets generated using inconsistent data, some of which based on the build python, and some from the target platform python.
See for example the following gist:
https://gist.github.com/itamaro/8f23ebec2e142fd7480241b76c16c4e0
The gist contains the build-details.json generated for a 3.15 GIL-disabled build, using a 3.15 GIL-enabled build as the build-python.
Concretely:
cd ~/pybuild/python/main-default
./configure --with-pydebug
make -j
cd ~/pybuild/python/main-nogil
./configure --with-pydebug --disable-gil
make -j
# now use the GIL-enabled python to regenerate build-details.json for the nogil build
_PYTHON_PROJECT_BASE=. _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_td_darwin_darwin _PYTHON_SYSCONFIGDATA_PATH=build/lib.macosx-14.5-arm64-3.15 ~/pybuild/python/main-default/python.exe -E Tools/build/generate-build-details.py /tmp/build-details.jsonSome examples from the gist:
Incorrect - abi.flags is ["d"] instead of ["t", "d"]:
"abi": {
"flags": [
"d"
]
},Correct - libpython.static has the correct SOABI, including t:
"libpython": {
"static": "/usr/local/lib/libpython3.15td.a"
},Incorrect - suffixes.extensions includes extensions for the GIL-enabled ABI:
"extensions": [
".cpython-315d-darwin.so",
".cpython-315-darwin.so",
".abi3.so",
".so"
]Notes
I used a "cross-build" on the same MacOS host here, with GIL enabled vs GIL disabled, because it was the easiest for a full end-to-end repro on a single machine.
The issue is more general though, as I originally observed it when generating build-info.json files for a MacOS-target runtime (both GIL enabled and disabled) using a Linux-GIL-enabled build-python. That setup is more involved, but the mixing is also more apparent, with some fields using "darwin" and other fields using "linux-gnu-x86_64" as the multiarch string.
CPython versions tested on:
CPython main branch, 3.15, 3.14
Operating systems tested on:
macOS, Linux
Activity
Zheaoli commented on Jul 4, 2025
I think this issue is caused by that we are mixing up the target and build info. I made a PoC patch for abi.flags and suffixes.extensions. I'm not sure this is acceptable
Zheaoli@f7bfb6c
itamaro commented on Jul 13, 2025
cc @FFY00 more generally, should build-details be generated strictly from sysconfigdata, so if sysconfigdata is accurate then build-details would also be accurate?
AA-Turner commented on Sep 16, 2025
Cross posting @sobolevn's comment from #135785 (closed as duplicate):
FFY00 commented on Oct 4, 2025
It should be generated from whatever is most accurate.
geofft commented on Oct 17, 2025
Oh, I just tracked down why this is happening specifically for .suffixes.extensions. Tools/build/generate-build-details.py does
which is in turn implemented in importlib._bootstrap_external as EXTENSION_SUFFIXES = _imp.extension_suffixes(), which is in turn implemented in C by turning const char *_PyImport_DynLoadFiletab[] into a Python list of strings.
For many of the other things that generate-build-details.py looks at, they are set correctly set on cross builds because $(PYTHON_FOR_BUILD) has a number of environment variables set that override the behavior of various functions, mostly in sysconfig. Specifically, configure.ac sets the variable to:
But this particular call path doesn't go through anything with an override.
One option would be to try to read _PyImport_DynLoadFiletab out of the just-built Python binary or libpython, e.g. by opening it with ctypes (but that would break in a number of circumstances) or using a binary parser (which would require a binary parser, but honestly the new remote debugging stuff might have enough of a parser to pull this off, at least for the platforms it supports). Probably a cleaner option is to get the info that's in that variable out of the build in some other way.
Similarly, ['abi']['flags'] is set from sys.abiflags, which is set in C as SET_SYS_FROM_STRING("abiflags", ABIFLAGS); and also doesn't get impacted by these environment variables. (This one, at least, is available in the Makefile, so we could just pass it as an argument to generate-build-details.py.)
If it's helpful, python-build-standalone's interpreters exhibit both of these bugs: our debug and free-threaded variants use a --with-build-python that is neither of those, so we have the ['abi']['flags'] bug on those, and any of our cross-built interpreters (basically anything not x86_64 or arm64) has the['suffixes']['extensions'] bug, as reported in astral-sh/python-build-standalone#816.
(FWIW I just opened #140266 which is also about incorrect data in build-details.json and kind of involves all those environment variables set in $(PYTHON_FOR_BUILD), but probably isn't super related, unless this is a motivation to more completely rework how the file is generated. As with that issue I'm happy to contribute a PR given guidance on how you want to approach this.)
manueljacob commented on Apr 7, 2026
This very likely has the same root cause as #90067.
mgorny commented on Aug 13, 2026
Just hit this one as well. To be honest, I can only think of two ways of reliably solving this: we either mirror the whole logic from CPython in generate-build-details.py (or somewhere else in the build system), or we play with some hard C parsing to get it out of the preprocessed C code used to build CPython.