Cross compiles try to load libraries for target python when the version + SOABI combo match the host and can crash · Issue #115382 · python/cpython · GitHub

Bug report

Bug description:

Note: I'm using 3.11.6 to showcase the behavior because that's easiest, but the build problems exist in 3.12+

When cross compiling Python, typically the foreign build is targeting a different architecture, but this is not always the case.

It's possible that an x86-64 host may be building a Python for a "foreign" x86-64 machine. This typically means that there may be some difference in libc version or CPU instruction support.

When performing a cross compile for the same architecture (by this I mean the combination of Python version + SOABI), Python 3.12+ will attempt to load foreign libraries as part of some of the target dependencies for the build_all make target and will potentially fail.

When cross compiling, builds specify --with-build-python during configure which specifies a host-safe version of python to use to perform python based tasks on behalf of the foreign python build. When configured, PYTHON_FOR_BUILD will be set to a rather complex command that generally evaluates to something like:

_PYTHON_PROJECT_BASE=/mnt/development/buildroot/output/build/python3-3.11.6 _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11/:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_x86_64-linux-gnu /mnt/development/buildroot/output/host/bin/python3

This is currently a problem for checksharedmods:

# dependency on BUILDPYTHON ensures that the target is run last
.PHONY: checksharedmods
checksharedmods: sharedmods $(PYTHON_FOR_BUILD_DEPS) $(BUILDPYTHON)
	@$(RUNSHARED) $(PYTHON_FOR_BUILD) $(srcdir)/Tools/build/check_extension_modules.py

When run, it tries to run check_extension_modules via PYTHON_FOR_BUILD, however this script has nested in its import dependencies a dependency on externally built modules which poses a problem.

vfazio@vfazio4 ~/development/buildroot/output/build/python3-3.11.6 $ _PYTHON_PROJECT_BASE=/mnt/development/buildroot/output/build/python3-3.11.6 _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11/:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_x86_64-linux-uclibc /mnt/development/buildroot/output/host/bin/python3 -c "import sys; print(sys.version); import math"
3.11.6 (main, Feb 12 2024, 09:05:53) [GCC 11.4.0]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: libc.so.0: cannot open shared object file: No such file or directory

Note, this was introduced as part of 3.12:

7bd67d1
81dca70d704

This also can show up with glibc like so

When the PYTHON_FOR_BUILD command is updated to not include the local build directory in PYTHONPATH, everything works fine.

This makes sense because it will use the host's libraries instead of the target's libraries... Inserting that path into PYTHONPATH alters the search order for extensions, and because the python versions match (this is a requirement for cross builds) and SOABI matches (which is due to the triplets being the same), it will try to use the libraries from the target path:

>>> sys.path
['', '/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11', '/mnt/development/buildroot/output/build/python3-3.11.6/Lib', '/mnt/development/buildroot/output/host/lib/python311.zip', '/mnt/development/buildroot/output/host/lib/python3.11', '/mnt/development/buildroot/output/host/lib/python3.11/lib-dynload', '/home/vfazio/.local/lib/python3.11/site-packages', '/mnt/development/buildroot/output/host/lib/python3.11/site-packages']

>>> sys.meta_path
[<_distutils_hack.DistutilsMetaFinder object at 0x7f031311cf50>, <class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>]
>>> sys.meta_path[3]
<class '_frozen_importlib_external.PathFinder'>
>>> sys.meta_path[3].find_spec("math")
ModuleSpec(name='math', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f0313092d50>, origin='/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11/math.cpython-311-x86_64-linux-gnu.so')

>>> sys.meta_path[3].find_module("math").load_module()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap_external>", line 605, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1120, in load_module
  File "<frozen importlib._bootstrap_external>", line 945, in load_module
  File "<frozen importlib._bootstrap>", line 290, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 721, in _load
  File "<frozen importlib._bootstrap>", line 676, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 573, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1233, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: libc.so.0: cannot open shared object file: No such file or directory

So, the "easy solution" is to remove:

diff --git a/configure.ac b/configure.ac
index 384718db1f..6b01083336 100644
--- a/configure.ac
+++ b/configure.ac
@@ -164,7 +164,7 @@ AC_ARG_WITH([build-python],
     dnl Build Python interpreter is used for regeneration and freezing.
     ac_cv_prog_PYTHON_FOR_REGEN=$with_build_python
     PYTHON_FOR_FREEZE="$with_build_python"
-    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
+    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
     AC_MSG_RESULT([$with_build_python])
   ], [
     AS_VAR_IF([cross_compiling], [yes],

However, I assume that was added for a reason and is maybe necessary for other steps (though in testing, it didn't appear to be necessary for the compile to complete).

I'm not sure it ever makes sense to load external modules from the foreign target python when cross compiling, This only works currently because for foreign architecture builds, the triplet will mismatch and cause it to fall back to the host's versions of the libraries. It'd only be safe to do this if the modules were completely source based.

If PYTHONPATH has to be set, then there's maybe an argument that python version and SOABI do not provide enough differentiation and I'm not sure of what the best solution is there.

Alternatively, the check_extension_modules script could be rewritten to reduce the use of external modules to perform its parsing and temporarily work around this issue, but a similar issue could be reintroduced in a subsequent script without policies on what can and can't be used in scripts called during build.

CPython versions tested on:

3.12
3.11

Operating systems tested on:

Linux

Linked PRs

@tiran Apologies in advance for tagging you, but would like your input as the developer who introduced the script as a dependency to the default build target.

I forgot to actually run autoreconf after patching configure.ac so the updated PYTHONPATH didn't actually stick. Builds will definitely fail as part of check_extension_modules.py if the local path is not there for foreign architecture builds, so this doesn't seem like an actual solution.

Traceback (most recent call last):
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Tools/build/check_extension_modules.py", line 484, in <module>
    main()
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Tools/build/check_extension_modules.py", line 466, in main
    checker = ModuleChecker(
              ^^^^^^^^^^^^^^
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Tools/build/check_extension_modules.py", line 143, in __init__
    self.ext_suffix = sysconfig.get_config_var("EXT_SUFFIX")
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Lib/sysconfig.py", line 740, in get_config_var
    return get_config_vars().get(name)
           ^^^^^^^^^^^^^^^^^
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Lib/sysconfig.py", line 723, in get_config_vars
    _init_config_vars()
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Lib/sysconfig.py", line 670, in _init_config_vars
    _init_posix(_CONFIG_VARS)
  File "/work/.cache/build/rpi3/python/3.12.2/python-3.12.2/Lib/sysconfig.py", line 536, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata__linux_aarch64-linux-gnu'
make: *** [Makefile:1144: checksharedmods] Error 1

So the challenge is we need the local path to find the foreign libraries to check they exist, but specifying the local path in PYTHONPATH causes scripts used during the build process to attempt to load external modules, like math, from the target instead of the host.

Maybe one option is to not specify the path in PYTHONPATH, so that the initial module imports work and use the host's libraries, but then as part of check_extension_modules.py append the build directory + cat pybuilddir.txt into the head of sys.path prior to calling anything from sysconfig.

I am seeing something very similar when cross compiling from illumos x86-64 to illumos aarch64.

aarch64-unknown-solaris2.11-ar rcs libpython3.12.a Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/typevarobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o  Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/assemble.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/flowgraph.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/ceval_gil.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/instrumentation.o Python/intrinsics.o Python/legacy_tracing.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/tracemalloc.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/perf_trampoline.o Python/dynload_shlib.o     Modules/config.o Modules/main.o Modules/gcmodule.o Modules/ucred.o  Modules/dlpimodule.o  Modules/privileges.o  Modules/pyrbac.o Modules/authattr.o Modules/execattr.o Modules/userattr.o  Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre/sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_typingmodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o Python/deepfreeze/deepfreeze.o Modules/getpath.o Python/frozen.o
/opt/cross/aarch64/bin/gcc --sysroot=/data/omnios-build/omniosorg/bloody/_build/sysroot.aarch64 -shared      -o libpython3.so -Wl,-hlibpython3.so libpython3.12.so
/opt/cross/aarch64/bin/gcc --sysroot=/data/omnios-build/omniosorg/bloody/_build/sysroot.aarch64      -o python Programs/python.o -Wl,-R,/usr/lib -L. -lpython3.12 -lsocket -lnsl -lintl -ldl -lsendfile  -lpthread -ltsol  -ldlpi   -lnsl -lsocket -lsecdb                        -lm
/opt/cross/aarch64/bin/gcc --sysroot=/data/omnios-build/omniosorg/bloody/_build/sysroot.aarch64      -o Programs/_testembed Programs/_testembed.o -Wl,-R,/usr/lib -L. -lpython3.12 -lsocket -lnsl -lintl -ldl -lsendfile  -lpthread -ltsol  -ldlpi   -lnsl -lsocket -lsecdb                        -lm
Traceback (most recent call last):
  File "/data/omnios-build/omniosorg/bloody/_build/Python-3.12.2/Python-3.12.2/./Tools/build/check_extension_modules.py", line 25, in <module>
    import pathlib
  File "/data/omnios-build/omniosorg/bloody/_build/Python-3.12.2/Python-3.12.2/Lib/pathlib.py", line 20, in <module>
    from urllib.parse import quote_from_bytes as urlquote_from_bytes
  File "/data/omnios-build/omniosorg/bloody/_build/Python-3.12.2/Python-3.12.2/Lib/urllib/parse.py", line 36, in <module>
    import math
ImportError: ld.so.1: python3.12: fatal: libm.so.0: open failed: No such file or directory
make: *** [Makefile:1157: checksharedmods] Error 1
--- Make failed

The link worked, the checksharedmods target tries to load libm.so.0 which only exists on illumos aarch64.

@citrus-it

I am seeing something very similar when cross compiling from illumos x86-64 to illumos aarch64.

The link worked, the checksharedmods target tries to load libm.so.0 which only exists on illumos aarch64.

That's pretty odd. It shouldn't be doing this for cross architecture builds, though my testing was limited to Debian and Buildroot based distros.

Can you look at your config.log and find the PLATFORM_TRIPLET and MULTIARCH variables?

cpython/configure

Line 6752 in ae6c01d

    
           printf %s "checking for the platform triplet based on compiler characteristics... " >&6; }

Also cat the pybuilddir.txt file in the build directory.

My guess is that something is not being correctly identified because it's an unknown libc or somehow the architecture isn't being set correctly and the import code thinks it can load the foreign architecture libs from the current build directory.

Can you look at your config.log and find the PLATFORM_TRIPLET and MULTIARCH variables?

PLATFORM_TRIPLET is none and MULTIARCH is empty,

% cat tmp/src/pybuilddir.txt
build/lib.sunos5-arm-3.12%

I'll go and look more closely at configure.ac and the existing cross patch that we apply - there are probably some adjustments we need to make in there. Thanks, and sorry for chipping into what I thought was a related issue.

No apologies necessary, I just hadn't seen this variant before. It could still be related (odd that it only fails at the very end when running this script)

No apologies necessary, I just hadn't seen this variant before. It could still be related (odd that it only fails at the very end when running this script)

With a patched configure.ac, so that the triplet is now populated, I get this instead. I will look into it properly later today, and should probably get these local patches upstreamed too.

ModuleNotFoundError: No module named '_sysconfigdata__sunos5_aarch64-unknown-solaris2'

Just as an FYI, platform checking was reworked recently c163d7f

For us, it's a bit of a bummer that uClibc is now explicitly disabled. We may try to get that support added in at some point,

After correcting the host triple detection, everything's working fine for my cross compilation case. Thanks for the pointers @vfazio !

I was thinking about this last night and I think the options are:

change check_extension_modules.py to not use pathlib which is importing math which is an external module and is impacted by the adjusted import priority due to PYTHONPATH. This fixes the immediate issue but doesn't prevent other build scripts from possibly making the same mistake.
Update PYTHON_FOR_BUILD to not use PYTHONPATH and force any build scripts that need to interrogate the build to manually adjust sys.path (or pass the build path as an argument). Scripts should be running out of the build directory so should know where to look for adjusting the import path. This should only be necessary for cross compile scenarios I think?
Specify yet another PYTHON variable in the Makefile that does not specify PYTHONPATH for "special cases" and use that for scripts that are "build directory aware" and which manually adjust sys.path "when appropriate"
Make the SOABIs more unique. This is difficult because of what needs to be encoded. It's not just the architecture and libc, but the libc version (host may be on glibc 2.34 but the cross build may target 2.38), architecture IP/instruction support (psABI https://gitlab.com/x86-psABIs/x86-64-ABI) etc.
<other magic>

In support of option 2 above...

Setting PYTHONPATH feels like we're abusing the import mechanism a bit "because it works" in most cases. These build time scripts are special in that they need to run on the host and may want to leverage host provided libraries but perform parsing on files generated by the target build. The mechanism used in check_extension_modules.py is to interrogate sysconfig for modules but sysconfig is a fun hodge-podge of information; sometimes it reflects the host python data (see sysconfig.get_platform), sometimes it reflects the target.

For this script in particular, we rely on get_config_var to query the modules specified in the target Makefile, meaning if we were to drop the build directory from PYTHONPATH we cannot reliably get information if sysconfig._init_config_vars was called because it would reflect the values from the host and not the target.

Luckily, this does not (currently) get called by any dependency within the script, so we could reliably inject the build path into sys.path but best is maybe to check as a precaution:

diff --git a/Tools/build/check_extension_modules.py b/Tools/build/check_extension_modules.py
index 59239c62e2..fe3423e4aa 100644
--- a/Tools/build/check_extension_modules.py
+++ b/Tools/build/check_extension_modules.py
@@ -140,9 +140,17 @@ class ModuleChecker:
     def __init__(self, cross_compiling: bool = False, strict: bool = False):
         self.cross_compiling = cross_compiling
         self.strict_extensions_build = strict
+        self.builddir = self.get_builddir()
+
+        # Add path for cross modules prior to sysconfig parsing the makefile.
+        if self.cross_compiling:
+            if sysconfig._CONFIG_VARS_INITIALIZED:
+                sysconfig._CONFIG_VARS_INITIALIZED = False
+                sysconfig._CONFIG_VARS = None
+            sys.path.insert(1, os.path.join(sysconfig._PROJECT_BASE, self.builddir))
+
         self.ext_suffix = sysconfig.get_config_var("EXT_SUFFIX")
         self.platform = sysconfig.get_platform()
-        self.builddir = self.get_builddir()
         self.modules = self.get_modules()
 
         self.builtin_ok = []
diff --git a/configure b/configure
index e962a6aed1..467962f75b 100755
--- a/configure
+++ b/configure
@@ -3686,7 +3686,7 @@ fi
     fi
         ac_cv_prog_PYTHON_FOR_REGEN=$with_build_python
     PYTHON_FOR_FREEZE="$with_build_python"
-    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
+    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
     { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $with_build_python" >&5
 printf "%s\n" "$with_build_python" >&6; }
 
diff --git a/configure.ac b/configure.ac
index 384718db1f..6b01083336 100644
--- a/configure.ac
+++ b/configure.ac
@@ -164,7 +164,7 @@ AC_ARG_WITH([build-python],
     dnl Build Python interpreter is used for regeneration and freezing.
     ac_cv_prog_PYTHON_FOR_REGEN=$with_build_python
     PYTHON_FOR_FREEZE="$with_build_python"
-    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
+    PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
     AC_MSG_RESULT([$with_build_python])
   ], [
     AS_VAR_IF([cross_compiling], [yes],

I tested this on a cross architecture and "same architecture but foreign libc" build and both builds completed though i haven't run the test suites.

Changing PYTHONPATH obviously would affect more than this script, so all other callers would need to be evaluated to ensure they're still working as intended. This change is almost a decade old so there may be some ingrained assumptions issue 15484 commit: 9731330

This doesn't mean we can't do some combination of all options. Option 1 may be the easiest bandaid and has less chance of impacting other scripts until we've had time to vet them.

I tested this on a cross architecture and "same architecture but foreign libc" build and both builds completed though i haven't run the test suites.

Thanks for the patch, it fixes the problem

    import math
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found

when cross-compiling for x86-64-v2/glibc-2.38:

$ grep "PLATFORM_TRIPLET\|MULTIARCH" config.log
MULTIARCH='x86_64-linux-gnu'
PLATFORM_TRIPLET='x86_64-linux-gnu'
$ cat pybuilddir.txt
build/lib.linux-x86_64-3.12
$ uname -a
Linux debian 6.7.5-x64v3-xanmod1 #0~20240216.gf29bf01 SMP PREEMPT_DYNAMIC Fri Feb 16 20:35:14 UTC x86_64 GNU/Linux
$ /usr/lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Debian GLIBC 2.36-9+deb12u4) stable release version 2.36.

A patch to convert check_extension_modules.py to drop pathlib. I have not tested this for cross builds, just the basic functionality, but I think it should work. I may try to submit this as it's a more "isolated" change than changing PYTHONPATH for all make targets, knowing that the long term goal should be to drop PYTHONPATH where possible for cross compile scenarios, which could lead to the revert of this commit.

diff --git a/Tools/build/check_extension_modules.py b/Tools/build/check_extension_modules.py
index a9fee4981e..ab6138750c 100644
--- a/Tools/build/check_extension_modules.py
+++ b/Tools/build/check_extension_modules.py
@@ -22,7 +22,6 @@
 import enum
 import logging
 import os
-import pathlib
 import re
 import sys
 import sysconfig
@@ -33,7 +32,7 @@
 from importlib.util import spec_from_file_location, spec_from_loader
 from typing import Iterable
 
-SRC_DIR = pathlib.Path(__file__).parent.parent.parent
+SRC_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
 
 # core modules, hard-coded in Modules/config.h.in
 CORE_MODULES = {
@@ -133,7 +132,7 @@ class ModuleChecker:
         "Modules/Setup.local",
         "Modules/Setup.stdlib",
         "Modules/Setup.bootstrap",
-        SRC_DIR / "Modules/Setup",
+        os.path.join(SRC_DIR, "Modules/Setup"),
     )
 
     def __init__(self, cross_compiling: bool = False, strict: bool = False):
@@ -263,14 +262,13 @@ def list_module_names(self, *, all: bool = False) -> set:
             names.update(WINDOWS_MODULES)
         return names
 
-    def get_builddir(self) -> pathlib.Path:
+    def get_builddir(self) -> str:
         try:
             with open(self.pybuilddir_txt, encoding="utf-8") as f:
                 builddir = f.read()
         except FileNotFoundError:
             logger.error("%s must be run from the top build directory", __file__)
             raise
-        builddir = pathlib.Path(builddir)
         logger.debug("%s: %s", self.pybuilddir_txt, builddir)
         return builddir
 
@@ -338,7 +336,7 @@ def get_sysconfig_modules(self) -> Iterable[ModuleInfo]:
             logger.debug("Found %s in Makefile", modinfo)
             yield modinfo
 
-    def parse_setup_file(self, setup_file: pathlib.Path) -> Iterable[ModuleInfo]:
+    def parse_setup_file(self, setup_file: str) -> Iterable[ModuleInfo]:
         """Parse a Modules/Setup file"""
         assign_var = re.compile(r"^\w+=")  # EGG_SPAM=foo
         # default to static module
@@ -382,10 +380,10 @@ def get_spec(self, modinfo: ModuleInfo) -> ModuleSpec:
         else:
             raise ValueError(modinfo)
 
-    def get_location(self, modinfo: ModuleInfo) -> pathlib.Path:
+    def get_location(self, modinfo: ModuleInfo) -> str:
         """Get shared library location in build directory"""
         if modinfo.state == ModuleState.SHARED:
-            return self.builddir / f"{modinfo.name}{self.ext_suffix}"
+            return os.path.join(self.builddir, f"{modinfo.name}{self.ext_suffix}")
         else:
             return None
 
@@ -430,23 +428,33 @@ def rename_module(self, modinfo: ModuleInfo) -> None:
 
         failed_name = f"{modinfo.name}_failed{self.ext_suffix}"
         builddir_path = self.get_location(modinfo)
-        if builddir_path.is_symlink():
+        if os.path.islink(builddir_path):
             symlink = builddir_path
-            module_path = builddir_path.resolve().relative_to(os.getcwd())
-            failed_path = module_path.parent / failed_name
+            real_path = os.path.realpath(builddir_path)
+            cwd = os.getcwd()
+            if not real_path.startswith(cwd):
+                raise ValueError(f"{real_path} is not in the subpath of {cwd}")
+            module_path = real_path.partition(cwd)[2]
+            failed_path = os.path.join(os.path.dirname(module_path), failed_name)
         else:
             symlink = None
             module_path = builddir_path
-            failed_path = self.builddir / failed_name
+            failed_path = os.path.join(self.builddir, failed_name)
 
         # remove old failed file
-        failed_path.unlink(missing_ok=True)
+        try:
+            os.unlink(failed_path)
+        except FileNotFoundError:
+            pass
         # remove symlink
         if symlink is not None:
-            symlink.unlink(missing_ok=True)
+            try:
+                os.unlink(symlink)
+            except FileNotFoundError:
+                pass
         # rename shared extension file
         try:
-            module_path.rename(failed_path)
+            os.rename(module_path, failed_path)
         except FileNotFoundError:
             logger.debug("Shared extension file '%s' does not exist.", module_path)
         else:

I'm not sure 3.13 is currently affected. 15de493 changed pathlib to delay import of quote_from_bytes so math doesn't get imported when pathlib does.

However, the root problem is still there. If a library is compiled as a shared module and it's a dependency of a build script, there's a risk of the foreign module being imported.

Build targets that use PYTHON_FOR_BUILD:

platform
pybuilddir.txt
wasm_stdlib
checksharedmods
install
altinstall
libinstall

I don't know anything about wasm_stdlib to make a call there, but besides checksharedmods, builds configured --with-ensurepip will likely fail as part of install/altinstall and libinstall will probably fail when compiling the optimizations.

The reason Buildroot doesn't see more failures is because we specifically configure --without-ensurepip and we patch out the compilation of optimizations.

After review, I don't feel confident in dropping the build directory from PYTHONPATH for these other build targets, especially in a patch to 3.12.x

Honestly, as much of a hack as it may look like, it may be easiest to delay the import of from urllib.parse import quote_from_bytes as urlquote_from_bytes into PurePath.as_uri like the above commit to work around the issue without needing to rework all of these targets in 3.12.x. This is untested, but i may try it out tomorrow.

diff --git a/Lib/pathlib.py b/Lib/pathlib.py
index bd5a096f9e..544290d718 100644
--- a/Lib/pathlib.py
+++ b/Lib/pathlib.py
@@ -17,7 +17,6 @@
 from _collections_abc import Sequence
 from errno import ENOENT, ENOTDIR, EBADF, ELOOP
 from stat import S_ISDIR, S_ISLNK, S_ISREG, S_ISSOCK, S_ISBLK, S_ISCHR, S_ISFIFO
-from urllib.parse import quote_from_bytes as urlquote_from_bytes
 
 
 __all__ = [
@@ -479,7 +478,8 @@ def as_uri(self):
             # It's a posix path => 'file:///etc/hosts'
             prefix = 'file://'
             path = str(self)
-        return prefix + urlquote_from_bytes(os.fsencode(path))
+        from urllib.parse import quote_from_bytes
+        return prefix + quote_from_bytes(os.fsencode(path))
 
     @property
     def _str_normcase(self):

@vfazio If it helps... I have used your patch in a Buildroot build with success!

The build was done from Buildroot commit 8ab4a0a348 which is of course a commit above 36e635d2d5c0166476858aa239ccbe78e8f2af14 (package/python3: bump version to 3.12.1).

Without the patch, I get the exact error than the one described here. With the patch, no error!

pythongh-115382: Remove foreign import paths from cross compiles

pythongh-115382: Fix cross compiles when host and target use same SOABI

So, I've pushed an MR that I believe should address this issue.

The commit message sort of speaks for itself, but I'll detail what's going on just as a summary post.

When --with-build-python is specified, which is mandatory for cross compiles but could be used in non-cross compile scenarios, an external but "compatible" host interpreter is used to perform certain build stages. There is a minimal version specified #104487 to account for required language features for certain build targets.

When a host interpreter is used, the command line to invoke that interpreter includes PYTHONPATH which points to the target build. This puts the target's paths higher in the sys.path import search order so the target's libraries are prioritized over the host's versions.

For foreign architecture builds (think build=x86_64, host=arm64), when imports are searched and the module is a compiled module, the ExtensionFileLoader searches for the specific SOABI+file extension:

const char *_PyImport_DynLoadFiletab[] = {
#ifdef __CYGWIN__
    ".dll",
#else  /* !__CYGWIN__ */
    "." SOABI ".so",
#ifdef ALT_SOABI
    "." ALT_SOABI ".so",
#endif
    ".abi" PYTHON_ABI_STRING ".so",
    ".so",
#endif  /* __CYGWIN__ */
    NULL,
};

This means that the host interpreter will never load the target compiled libraries so long as they disagree on SOABI.

However, when the host interpreter and the target have the same SOABI (so host=x86_64, build=x86_64), there's a risk that due to their precedence in sys.path that the target's libraries will be loaded and/or executed.

As seen in this issue, this is problematic because the target may have been built and linked against a different libc implementation which the host cannot load. Even if the libc matched, there's a chance that the instruction set for the generated binaries may not be executable on the host (x86 psABI differences).

The only time it's "OK" to run the target libraries is if the imports and their dependencies are pure Python.

From what I can naively tell, the only real reason it's necessary to include the target's path in sys.path via PYTHONPATH is so the target's generated sysconfigdata module can be interrogated to check for built modules, compile options, etc.

If this is indeed the case, it should be sufficient to drop the path from PYTHONPATH and build an override mechanism for its path similar to how the name is already handled via _PYTHON_SYSCONFIGDATA_NAME.

In build testing, this works fine. The Python CI pipelines also seem to be ok with this.

pythongh-115382: Fix cross compiles when host and target use same SOABI

Note that CPython 3.11 also fails when running make sharedmods if the host doesn't support instructions generated by the target toolchain:

An example build:
http://autobuild.buildroot.net/results/c854080e003e9a7d525325073190b472a8f982aa/build-end.log

make[1]: *** [Makefile:868: sharedmods] Illegal instruction (core dumped)
make[1]: *** Waiting for unfinished jobs....

I tested the patch in #116294 and adapted it for 3.11 and it seemed to resolve the issue.

gh-115382: Fix cross compiles when host and target use same SOABI

pythongh-115382: Fix cross compiles when host and target use same SOABI

[3.13] gh-115382: Fix cross compiles when host and target use same SOABI

Closing this issue as my PR has landed in main and been ported to 3.13.

pythongh-115382: Fix cross compiles when host and target use same SOABI

Cross compiles try to load libraries for target python when the version + SOABI combo match the host and can crash · Issue #115382 · python/cpython · GitHub

Repository navigation

Cross compiles try to load libraries for target python when the version + SOABI combo match the host and can crash #115382

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

17 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Cross compiles try to load libraries for target python when the version + SOABI combo match the host and can crash · Issue #115382 · python/cpython · GitHub

Repository navigation

Cross compiles try to load libraries for target python when the version + SOABI combo match the host and can crash #115382

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

vfazio commented on Feb 16, 2024

vfazio commented on Feb 16, 2024

citrus-it commented on Feb 18, 2024

vfazio commented on Feb 18, 2024

citrus-it commented on Feb 18, 2024

vfazio commented on Feb 18, 2024

citrus-it commented on Feb 18, 2024

vfazio commented on Feb 18, 2024

citrus-it commented on Feb 19, 2024

vfazio commented on Feb 19, 2024

bkuhls commented on Feb 19, 2024

vfazio commented on Feb 25, 2024

vfazio commented on Feb 26, 2024

bendebled commented on Feb 26, 2024

17 remaining items

vfazio commented on Mar 18, 2024

vfazio commented on Apr 2, 2024

vfazio commented on Oct 21, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions