FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Draft - Build and publish free-threaded Python wheels (cp313t/cp314t) across CI by kosiew · Pull Request #1355 · apache/datafusion-python · GitHub

/ datafusion-python Public

Draft - Build and publish free-threaded Python wheels (cp313t/cp314t) across CI #1355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kosiew wants to merge 11 commits into from
Conversation 2 Commits 11 Checks 26 Files changed
Closed

Draft - Build and publish free-threaded Python wheels (cp313t/cp314t) across CI #1355

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
8e1fd4d
Expand wheel matrix for Python 3.13/3.14
kosiew Jan 28, 2026
6efe878
Update PyO3 dependency and document wheel rationale
kosiew Jan 28, 2026
c03f56d
Update build.yml to use python-tag for artifact naming and enhance RE…
kosiew Feb 4, 2026
0ebd476
Enhance build workflow to validate interpreter and wheel mode compati…
kosiew Feb 11, 2026
25264c3
Remove unsupported interpreter input in build.yml
kosiew Feb 28, 2026
01216c0
Update Python setup and maturin invocation paths
kosiew Feb 28, 2026
78630ab
Update free-threaded interpreter configurations
kosiew Feb 28, 2026
8524cc8
Merge with main
kosiew Feb 28, 2026
286eea6
refactor: update DataFusion re-export to include self reference
kosiew Feb 28, 2026
eeb2da6
fix: conditionally install dependencies for specific Python version
kosiew Mar 2, 2026
8b1a122
feat: add 3.14t experimental support for macOS arm64 and Windows builds
kosiew Mar 2, 2026
File filter

Filter by extension

Filter by extension .md  (1) .rs  (1) .toml  (1) .yml  (1) All 4 file types selected
Only manifest files
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
105 changes: 91 additions & 14 deletions .github/workflows/build.yml
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,11 @@ on:
workflow_call:
inputs:
build_mode:
description: 'Build mode: debug or release'
description: "Build mode: debug or release"
required: true
type: string
run_wheels:
description: 'Whether to build distribution wheels'
description: "Whether to build distribution wheels"
required: false
type: boolean
default: false
Expand Down Expand Up @@ -249,14 +249,63 @@ jobs:
needs: [generate-license, lint-rust, lint-python]
name: macOS arm64 & Windows
runs-on: ${{ matrix.os }}
continue-on-error: ${{ matrix.experimental || false }}
strategy:
fail-fast: false
matrix:
python-version: ["3.10"]
os: [macos-latest, windows-latest]
include:
- os: macos-latest
python-version: "3.10"
python-tag: cp310
- os: macos-latest
python-version: "3.11"
python-tag: cp311
- os: macos-latest
python-version: "3.12"
python-tag: cp312
- os: macos-latest
python-version: "3.13"
python-tag: cp313
- os: macos-latest
python-version: "3.14"
python-tag: cp314
- os: macos-latest
python-version: "3.13t"
python-tag: cp313t
- os: macos-latest
python-version: "3.14t"
python-tag: cp314t
experimental: true
- os: windows-latest
python-version: "3.10"
python-tag: cp310
- os: windows-latest
python-version: "3.11"
python-tag: cp311
- os: windows-latest
python-version: "3.12"
python-tag: cp312
- os: windows-latest
python-version: "3.13"
python-tag: cp313
- os: windows-latest
python-version: "3.14"
python-tag: cp314
- os: windows-latest
python-version: "3.13t"
python-tag: cp313t
- os: windows-latest
python-version: "3.14t"
python-tag: cp314t
experimental: true
steps:
- uses: actions/checkout@v6

- uses: actions/setup-python@v6
id: setup-python
with:
python-version: ${{ matrix.python-version }}

- uses: dtolnay/rust-toolchain@stable

- run: rm LICENSE.txt
Expand All @@ -282,6 +331,7 @@ jobs:
repo-token: ${{ secrets.GITHUB_TOKEN }}

- name: Install dependencies
if: matrix.python-version != '3.14t'
run: uv sync --dev --no-install-package datafusion

# Run clippy BEFORE maturin so we can avoid rebuilding. The features must match
Expand All @@ -293,11 +343,15 @@ jobs:

- name: Build Python package (release mode)
if: inputs.build_mode == 'release'
run: uv run --no-project maturin build --release --strip --features substrait
env:
MATURIN_FEATURES: ${{ endsWith(matrix.python-version, 't') && 'substrait' || 'substrait,py-limited-api' }}
run: uv run --no-project maturin build --release --strip --features "${MATURIN_FEATURES}" --interpreter "${{ steps.setup-python.outputs.python-path }}"

- name: Build Python package (debug mode)
if: inputs.build_mode != 'release'
run: uv run --no-project maturin build --features substrait
env:
MATURIN_FEATURES: ${{ endsWith(matrix.python-version, 't') && 'substrait' || 'substrait,py-limited-api' }}
run: uv run --no-project maturin build --features "${MATURIN_FEATURES}" --interpreter "${{ steps.setup-python.outputs.python-path }}"

- name: List Windows wheels
if: matrix.os == 'windows-latest'
Expand All @@ -314,7 +368,7 @@ jobs:
uses: actions/upload-artifact@v6
if: inputs.build_mode == 'release'
with:
name: dist-${{ matrix.os }}
name: dist-${{ matrix.os }}-${{ matrix.python-tag }}
path: target/wheels/*

# ============================================
Expand All @@ -324,13 +378,34 @@ jobs:
if: inputs.build_mode == 'release'
needs: [generate-license, lint-rust, lint-python]
runs-on: macos-15-intel
continue-on-error: ${{ matrix.experimental || false }}
strategy:
fail-fast: false
matrix:
python-version: ["3.10"]
include:
- python-version: "3.10"
python-tag: cp310
- python-version: "3.11"
python-tag: cp311
- python-version: "3.12"
python-tag: cp312
- python-version: "3.13"
python-tag: cp313
- python-version: "3.14"
python-tag: cp314
- python-version: "3.13t"
python-tag: cp313t
- python-version: "3.14t"
python-tag: cp314t
experimental: true
steps:
- uses: actions/checkout@v6

- uses: actions/setup-python@v6
id: setup-python
with:
python-version: ${{ matrix.python-version }}

- uses: dtolnay/rust-toolchain@stable

- run: rm LICENSE.txt
Expand All @@ -356,19 +431,22 @@ jobs:
repo-token: ${{ secrets.GITHUB_TOKEN }}

- name: Install dependencies
if: matrix.python-version != '3.14t'
run: uv sync --dev --no-install-package datafusion

- name: Build (release mode)
env:
MATURIN_FEATURES: ${{ endsWith(matrix.python-version, 't') && 'substrait' || 'substrait,py-limited-api' }}
run: |
uv run --no-project maturin build --release --strip --features substrait
uv run --no-project maturin build --release --strip --features "${MATURIN_FEATURES}" --interpreter "${{ steps.setup-python.outputs.python-path }}"

- name: List Mac wheels
run: find target/wheels/

- name: Archive wheels
uses: actions/upload-artifact@v6
with:
name: dist-macos-aarch64
name: dist-macos-aarch64-${{ matrix.python-tag }}
path: target/wheels/*

# ============================================
Expand Down Expand Up @@ -407,9 +485,8 @@ jobs:
shell: bash

# ============================================
# Build - Source Distribution
# Build - Merge Artifacts
# ============================================

merge-build-artifacts:
runs-on: ubuntu-latest
name: Merge build artifacts
Expand All @@ -434,7 +511,7 @@ jobs:
build-docs:
name: Build docs
runs-on: ubuntu-latest
needs: [build-manylinux-x86_64] # Only need the Linux wheel for docs
needs: [build-manylinux-x86_64] # Only need the Linux wheel for docs
# Only run docs on main branch pushes, tags, or PRs
if: github.event_name == 'push' || github.event_name == 'pull_request'
steps:
Expand Down Expand Up @@ -477,7 +554,7 @@ jobs:
- name: Download pre-built Linux wheel
uses: actions/download-artifact@v7
with:
name: dist-manylinux-x86_64
name: dist-manylinux-x86_64-cp310-314
path: wheels/

# Install from the pre-built wheels
Expand Down
7 changes: 2 additions & 5 deletions Cargo.toml
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ include = [
default = ["mimalloc"]
protoc = ["datafusion-substrait/protoc"]
substrait = ["dep:datafusion-substrait"]
py-limited-api = ["pyo3/abi3-py310"]

[dependencies]
tokio = { version = "1.49", features = [
Expand All @@ -48,11 +49,7 @@ tokio = { version = "1.49", features = [
"rt-multi-thread",
"sync",
] }
pyo3 = { version = "0.26", features = [
"extension-module",
"abi3",
"abi3-py310",
] }
pyo3 = { version = "0.26", features = ["extension-module"] }
pyo3-async-runtimes = { version = "0.26", features = ["tokio-runtime"] }
pyo3-log = "0.13.3"
arrow = { version = "57", features = ["pyarrow"] }
Expand Down
10 changes: 10 additions & 0 deletions dev/release/README.md
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,10 @@ datafusion-22.0.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
datafusion-22.0.0-cp37-abi3-win_amd64.whl
```

Note: PyO3's free-threaded CPython builds (3.13t/3.14t) use a distinct ABI and cannot use the limited API (`abi3`).
The release workflow enables Cargo feature `py-limited-api` for GIL-enabled wheels and disables it for free-threaded
builds, producing version-specific `cp313t`/`cp314t` wheels.

Upload the wheels to testpypi.

```bash
Expand Down Expand Up @@ -237,6 +241,12 @@ cargo publish

### Publishing Python Artifacts to PyPi

GitHub Actions groups wheel artifacts by platform and interpreter tag using the pattern `dist-<platform>-<python-tag>`.
For example, standard manylinux wheels live under `dist-manylinux-x86_64-cp310-314` while the free-threaded builds
use `dist-manylinux-x86_64-cp313t-314t`. macOS and Windows jobs publish one artifact per CPython version as well
(`dist-macos-latest-cp311`, `dist-windows-latest-cp313t`, etc.). Download the exact tags you intend to push to PyPI,
and remember that the docs workflow currently installs from the `cp310-314` manylinux artifact.

Go to the Test PyPI page of Datafusion, and download
[all published artifacts](https://test.pypi.org/project/datafusion/#files) under `dist-release/` directory. Then proceed
uploading them using `twine`:
Expand Down
4 changes: 2 additions & 2 deletions src/lib.rs
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@
// under the License.

// Re-export Apache Arrow DataFusion dependencies
pub use datafusion;
pub use datafusion::{
common as datafusion_common, logical_expr as datafusion_expr, optimizer, sql as datafusion_sql,
self, common as datafusion_common, logical_expr as datafusion_expr, optimizer,
sql as datafusion_sql,
};
#[cfg(feature = "substrait")]
pub use datafusion_substrait;
Expand Down
Loading
Back | FazBrowse Home | New Git URL