bpo-14527: fix (some) _ctypes liffi build problems by rupertnash · Pull Request #20451 · python/cpython · GitHub

Fundamentally, the problem appears to be that configure uses pkgconfig to find the libffi include directory, while setup.py's detect_ctypes only uses the global list of library directories.

I have made an attempt at fixing this by having configure produce the directory containing libffi (LIBFFI_LIBDIR) and setup.py use this. However I've hardly any experience with autotools, so I would be very happy to be corrected if this is no use at all.

https://bugs.python.org/issue14527

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

CLA Missing

Our records indicate the following people have not signed the CLA:

@rupertnash

For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

Worked for my setup.

Before following this commit, I had set PKG_CONFIG_PATH, LD_LIBRARY_PATH, LD_RUN_PATH and LIBRARY_PATH point towards the appropriate locations.

./configure did set correctly LIBFFI_INCLUDEDIR using pkgconfig

Still couldn't build module _ctypes

This commit solved my issue.

This patch seems correct to me.

Why use setup.py heuristics to find libname when you could just let pkgconfig gives the right hint for compilation host/target with pkg-config libffi --libs-only-l like you did with LIBFFI_LIBDIR and LIBFFI_INCLUDEDIR ?

That is a reasonable question.

The true answer is that while making my changes I tried to keep them as small as practical.

However when I came here to respond, I notice that this library search checks for libffi and libffi_pic, whether or not pkg-config is available on the compilation host.

If we simply create a new configure variable with say:

LIBFFI_LIBNAME="`"$PKG_CONFIG" libffi --libs-only-l 2> /dev/null | sed -e 's/^-l//;s/ *$//'`"

then we will lose the ability to find libffi_pic.{so,a} if there's no pkg-config.

We could set a default like LIBFFI_LIBNAME="ffi ffi_pic" and then have setup.py split it's value on space before looping much as now...

My feeling is to stick as we are, but happy to defer to core team

whether or not pkg-config is available on the compilation host.

maybe if pkg-config is here then use it fully else use old way ( and good luck for cross compilation setup.py will pick anything in /usr instead of sysroot in most cases ).

ref: https://bugs.python.org/issue31710

my 3.9.1 android/wasm (with HAVE_FFI_PREP_CLOSURE_LOC and ffi git on wasm ) CI is quite happy with that patch, it looks like a good step forward into cross-compilation to me

Dear devs - I had forgotten about this PR until I had to rebuild python yesterday on a machine with libffi in a non-standard location. Due to some major changes in setup.py around libffi, I've basically had to re-implement my patch. I've done this and force-pushed this branch. (Old commits https://github.com/rupertnash/cpython/tree/fix-ffi-build-old)

I would appreciate your comments again and I will have a bit of time in the next week or two to actually get this merged

I needed to make additional changes to pass -Wl,-rpath, to setup.py to get this to work. I wonder why there is support for building with an external SSL, but FFI lags behind.

I'd like to chime in and say this worked for Python 3.10.8 with a slight modification, but the patch would not apply against 3.11.0. It would be super nice if this PR was set up so it would merge cleanly into the main branch. My rationale for that is we use an embedded interpreter that runs on myriad Linux distributions, so we can't count on the system libffi being the same across distributions. Hence we need to use our own built from source libffi and point the python build at that libffi.

Sorry for the delay.
Meanwhile, this PR no longer applies; there is no setup.py now.

I've tested a build with env vars given by configure --help (see the issue), and everything looked good, but I'm not sure I tested correctly. Please do comment on the issue if there are more problems.

I'll close this PR. Sorry again for not getting to it earlier.

the-knights-who-say-ni added the CLA not signed label May 27, 2020

bedevere-bot added the awaiting review label May 27, 2020

the-knights-who-say-ni added CLA signed and removed CLA not signed labels May 28, 2020

lapsintra approved these changes Jul 22, 2020

View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels Jul 22, 2020

FFY00 approved these changes Nov 6, 2020

View reviewed changes

pmp-p reviewed Nov 22, 2020

View reviewed changes

pmp-p approved these changes Dec 9, 2020

View reviewed changes

pmp-p mentioned this pull request Dec 9, 2020

ctypes support pyodide/pyodide#728

Closed

Shillaker added a commit to faasm/cpython that referenced this pull request Dec 16, 2020

Merge changes from python#20451

f51a124

rupertnash added 2 commits March 22, 2021 16:18

fix for _ctypes liffi build problems bpo-14527

ef2c636

add NEWS entry

8fe5428

rupertnash force-pushed the fix-ffi-build branch from 76c8001 to 8fe5428 Compare March 23, 2021 15:02

pmp-p mentioned this pull request Dec 6, 2021

Shell scripts are missing error handling emmatyping/python-wasm#9

Closed

pmp-p mentioned this pull request Mar 30, 2022

Get ctypes working emmatyping/python-wasm#68

Open

pda mannequin mentioned this pull request Apr 15, 2022

How to link with a non-system libffi? #58732

Closed

ezio-melotti removed the CLA signed label Jul 13, 2022

encukou closed this Mar 28, 2024

                       if ffi_inc is not None:
                           for lib_name in ('ffi', 'ffi_pic'):
-                              if (self.compiler.find_library_file(self.lib_dirs, lib_name)):
+                              fullpath = self.compiler.find_library_file(self.lib_dirs + ffi_libdir, lib_name)

bpo-14527: fix (some) _ctypes liffi build problems by rupertnash · Pull Request #20451 · python/cpython · GitHub

Uh oh!

Uh oh!

bpo-14527: fix (some) _ctypes liffi build problems - #20451

bpo-14527: fix (some) _ctypes liffi build problems #20451
rupertnash wants to merge 2 commits into
python:mainfrom
rupertnash:fix-ffi-build

rupertnash commented May 27, 2020 •
edited by bedevere-bot
Loading

Uh oh!

Uh oh!

the-knights-who-say-ni commented May 27, 2020

Uh oh!

lapsintra left a comment

Uh oh!

FFY00 left a comment

Uh oh!

pmp-p Nov 22, 2020

Uh oh!

rupertnash Dec 2, 2020

Uh oh!

pmp-p Dec 5, 2020 •
edited
Loading

Uh oh!

Uh oh!

pmp-p left a comment

Uh oh!

rupertnash commented Mar 23, 2021

Uh oh!

josefwells commented Feb 11, 2022

Uh oh!

tom-kacvinsky commented Nov 1, 2022

Uh oh!

encukou commented Mar 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

bpo-14527: fix (some) _ctypes liffi build problems by rupertnash · Pull Request #20451 · python/cpython · GitHub

Uh oh!

Uh oh!

bpo-14527: fix (some) _ctypes liffi build problems - #20451

bpo-14527: fix (some) _ctypes liffi build problems #20451rupertnash wants to merge 2 commits intopython:mainpython/cpython:mainfrom rupertnash:fix-ffi-buildrupertnash/cpython:fix-ffi-buildCopy head branch name to clipboard

Conversation

rupertnash commented May 27, 2020 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

the-knights-who-say-ni commented May 27, 2020

CLA Missing

Uh oh!

lapsintra left a comment

Choose a reason for hiding this comment

Uh oh!

FFY00 left a comment

Choose a reason for hiding this comment

Uh oh!

pmp-p Nov 22, 2020

Choose a reason for hiding this comment

Uh oh!

rupertnash Dec 2, 2020

Choose a reason for hiding this comment

Uh oh!

pmp-p Dec 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pmp-p left a comment

Choose a reason for hiding this comment

Uh oh!

rupertnash commented Mar 23, 2021

Uh oh!

josefwells commented Feb 11, 2022

Uh oh!

tom-kacvinsky commented Nov 1, 2022

Uh oh!

encukou commented Mar 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

bpo-14527: fix (some) _ctypes liffi build problems #20451
rupertnash wants to merge 2 commits into
python:mainfrom
rupertnash:fix-ffi-build

rupertnash commented May 27, 2020 •
edited by bedevere-bot
Loading

pmp-p Dec 5, 2020 •
edited
Loading