FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-127629: Add ctypes to the Emscripten build by hoodmane · Pull Request #127683 · python/cpython · GitHub

/ cpython Public

gh-127629: Add ctypes to the Emscripten build #127683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
freakboy3742 merged 6 commits into from
Dec 10, 2024
Conversation 11 Commits 6 Checks 0 Files changed
Merged

gh-127629: Add ctypes to the Emscripten build #127683

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d9b6e9b
gh-127629: Add ctypes to the Emscripten build
hoodmane Oct 4, 2024
8172368
Add news
hoodmane Dec 6, 2024
91a70b3
Merge branch 'main' into emscripten-ctypes
freakboy3742 Dec 9, 2024
a3f4891
Use release libffi
hoodmane Dec 9, 2024
556de1e
Distinguish between `build` and `prefix` directories
hoodmane Dec 9, 2024
f670c21
Add separate make-libffi step to readme
hoodmane Dec 9, 2024
File filter

Filter by extension

Filter by extension .md  (1) .py  (1) .rst  (1) .sh  (1) All 4 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Build/2024-12-06-12-47-52.gh-issue-127629.tD-ERQ.rst
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Emscripten builds now include ctypes support.
1 change: 1 addition & 0 deletions Tools/wasm/README.md
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ or you can break it out into four separate steps:
```shell
python Tools/wasm/emscripten configure-build-python
python Tools/wasm/emscripten make-build-python
python Tools/wasm/emscripten make-libffi
python Tools/wasm/emscripten configure-host
python Tools/wasm/emscripten make-host
```
Expand Down
64 changes: 53 additions & 11 deletions Tools/wasm/emscripten/__main__.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import sys
import sysconfig
import tempfile
from urllib.request import urlopen
from pathlib import Path
from textwrap import dedent

Expand All @@ -22,9 +23,13 @@
CHECKOUT = EMSCRIPTEN_DIR.parent.parent.parent

CROSS_BUILD_DIR = CHECKOUT / "cross-build"
BUILD_DIR = CROSS_BUILD_DIR / "build"
NATIVE_BUILD_DIR = CROSS_BUILD_DIR / "build"
HOST_TRIPLE = "wasm32-emscripten"
HOST_DIR = CROSS_BUILD_DIR / HOST_TRIPLE

DOWNLOAD_DIR = CROSS_BUILD_DIR / HOST_TRIPLE / "build"
HOST_BUILD_DIR = CROSS_BUILD_DIR / HOST_TRIPLE / "build"
HOST_DIR = HOST_BUILD_DIR / "python"
PREFIX_DIR = CROSS_BUILD_DIR / HOST_TRIPLE / "prefix"

LOCAL_SETUP = CHECKOUT / "Modules" / "Setup.local"
LOCAL_SETUP_MARKER = "# Generated by Tools/wasm/emscripten.py\n".encode("utf-8")
Expand Down Expand Up @@ -118,16 +123,16 @@ def build_platform():

def build_python_path():
"""The path to the build Python binary."""
binary = BUILD_DIR / "python"
binary = NATIVE_BUILD_DIR / "python"
if not binary.is_file():
binary = binary.with_suffix(".exe")
if not binary.is_file():
raise FileNotFoundError("Unable to find `python(.exe)` in " f"{BUILD_DIR}")
raise FileNotFoundError("Unable to find `python(.exe)` in " f"{NATIVE_BUILD_DIR}")

return binary


@subdir(BUILD_DIR, clean_ok=True)
@subdir(NATIVE_BUILD_DIR, clean_ok=True)
def configure_build_python(context, working_dir):
"""Configure the build/host Python."""
if LOCAL_SETUP.exists():
Expand All @@ -143,7 +148,7 @@ def configure_build_python(context, working_dir):
call(configure, quiet=context.quiet)


@subdir(BUILD_DIR)
@subdir(NATIVE_BUILD_DIR)
def make_build_python(context, working_dir):
"""Make/build the build Python."""
call(["make", "--jobs", str(cpu_count()), "all"], quiet=context.quiet)
Expand All @@ -159,6 +164,23 @@ def make_build_python(context, working_dir):
print(f"🎉 {binary} {version}")


@subdir(HOST_BUILD_DIR, clean_ok=True)
def make_emscripten_libffi(context, working_dir):
shutil.rmtree(working_dir / "libffi-3.4.6", ignore_errors=True)
with tempfile.NamedTemporaryFile(suffix=".tar.gz") as tmp_file:
with urlopen(
"https://github.com/libffi/libffi/releases/download/v3.4.6/libffi-3.4.6.tar.gz"
) as response:
shutil.copyfileobj(response, tmp_file)
shutil.unpack_archive(tmp_file.name, working_dir)
call(
[EMSCRIPTEN_DIR / "make_libffi.sh"],
env=updated_env({"PREFIX": PREFIX_DIR}),
cwd=working_dir / "libffi-3.4.6",
quiet=context.quiet,
Comment thread
hoodmane marked this conversation as resolved.
)


@subdir(HOST_DIR, clean_ok=True)
def configure_emscripten_python(context, working_dir):
"""Configure the emscripten/host build."""
Expand All @@ -168,7 +190,7 @@ def configure_emscripten_python(context, working_dir):

emscripten_build_dir = working_dir.relative_to(CHECKOUT)

python_build_dir = BUILD_DIR / "build"
python_build_dir = NATIVE_BUILD_DIR / "build"
lib_dirs = list(python_build_dir.glob("lib.*"))
assert (
len(lib_dirs) == 1
Expand All @@ -183,12 +205,18 @@ def configure_emscripten_python(context, working_dir):
sysconfig_data += "-pydebug"

host_runner = context.host_runner
env_additions = {"CONFIG_SITE": config_site, "HOSTRUNNER": host_runner}
pkg_config_path_dir = (PREFIX_DIR / "lib/pkgconfig/").resolve()
env_additions = {
"CONFIG_SITE": config_site,
"HOSTRUNNER": host_runner,
"EM_PKG_CONFIG_PATH": str(pkg_config_path_dir),
}
build_python = os.fsdecode(build_python_path())
configure = [
"emconfigure",
os.path.relpath(CHECKOUT / "configure", working_dir),
"CFLAGS=-DPY_CALL_TRAMPOLINE -sUSE_BZIP2",
"PKG_CONFIG=pkg-config",
f"--host={HOST_TRIPLE}",
f"--build={build_platform()}",
f"--with-build-python={build_python}",
Expand All @@ -197,7 +225,7 @@ def configure_emscripten_python(context, working_dir):
"--disable-ipv6",
"--enable-big-digits=30",
"--enable-wasm-dynamic-linking",
f"--prefix={HOST_DIR}",
f"--prefix={PREFIX_DIR}",
]
if pydebug:
configure.append("--with-pydebug")
Expand Down Expand Up @@ -264,6 +292,7 @@ def build_all(context):
steps = [
configure_build_python,
make_build_python,
make_emscripten_libffi,
configure_emscripten_python,
make_emscripten_python,
]
Expand Down Expand Up @@ -292,18 +321,30 @@ def main():
configure_build = subcommands.add_parser(
"configure-build-python", help="Run `configure` for the " "build Python"
)
make_libffi_cmd = subcommands.add_parser(
"make-libffi", help="Clone libffi repo, configure and build it for emscripten"
)
make_build = subcommands.add_parser(
"make-build-python", help="Run `make` for the build Python"
)
configure_host = subcommands.add_parser(
"configure-host",
help="Run `configure` for the host/emscripten (pydebug builds are inferred from the build Python)",
)
make_host = subcommands.add_parser("make-host", help="Run `make` for the host/emscripten")
make_host = subcommands.add_parser(
"make-host", help="Run `make` for the host/emscripten"
)
clean = subcommands.add_parser(
"clean", help="Delete files and directories created by this script"
)
for subcommand in build, configure_build, make_build, configure_host, make_host:
for subcommand in (
build,
configure_build,
make_libffi_cmd,
make_build,
configure_host,
make_host,
):
subcommand.add_argument(
"--quiet",
action="store_true",
Expand Down Expand Up @@ -336,6 +377,7 @@ def main():
context = parser.parse_args()

dispatch = {
"make-libffi": make_emscripten_libffi,
Comment thread
hoodmane marked this conversation as resolved.
"configure-build-python": configure_build_python,
"make-build-python": make_build_python,
"configure-host": configure_emscripten_python,
Expand Down
21 changes: 21 additions & 0 deletions Tools/wasm/emscripten/make_libffi.sh
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash
set +e

export CFLAGS="-O2 -fPIC -DWASM_BIGINT"
export CXXFLAGS="$CFLAGS"

# Build paths
export CPATH="$PREFIX/include"
export PKG_CONFIG_PATH="$PREFIX/lib/pkgconfig"
export EM_PKG_CONFIG_PATH="$PKG_CONFIG_PATH"

# Specific variables for cross-compilation
export CHOST="wasm32-unknown-linux" # wasm32-unknown-emscripten

MichaelBuhler Apr 8, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Why did this end up being wasm32-unknown-linux instead of wasm32-unknown-emscripten?

hoodmane Apr 8, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Good question. Maybe I put it that way while debugging and didn't notice before merging.

emconfigure ./configure --host=$CHOST --prefix="$PREFIX" --enable-static --disable-shared --disable-dependency-tracking \
--disable-builddir --disable-multi-os-directory --disable-raw-api --disable-docs

make install
# Some forgotten headers?
cp fficonfig.h $PREFIX/include/
cp include/ffi_common.h $PREFIX/include/
Back | FazBrowse Home | New Git URL