chore(release): add multi-platform release JAR build pipeline - #87
Merged
andygrove merged 8 commits intoMay 25, 2026
Merged
andygrove merged 8 commits into
andygrove merged 8 commits into
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
The apache-rat-plugin runs at the `verify` phase, which surfaces during `mvn install` (the binary release flow) but not during the `make test` lifecycle. Without these exclusions, the PR / issue templates and the source-tarball exclude list itself fail the license-header check. The source-tarball flow already considers these files exempt via dev/release/rat_exclude_files.txt; this brings the pom's plugin configuration into alignment.
`core/pom.xml` defaults `datafusion.native.profile=debug`, so the antrun copy-native-lib step expects a debug-mode dylib at `native/target/debug/...`. The release orchestrator builds with `cargo build --release`, which writes to `native/target/release/...`. Pass `-Ddatafusion.native.profile=release` so the antrun check looks at the right path and the produced JAR bundles the release native libraries.
Address review feedback on the multi-platform JAR pipeline: - build-release.sh: pre-clean leftover builder containers before the first `docker run`. Without this, a prior SIGKILL'd run leaves named containers behind and the next invocation fails immediately on `docker run --name` with a name conflict. - publish-to-maven.sh: pass `--fail-with-body` to the upload-loop and close-staging curl calls. curl exits 0 on HTTP 4xx/5xx by default, which would silently let an unauthorised or rejected upload run to completion and corrupt the staging repo. - Dockerfile: fold gcc-10 / g++-10 / cpp-10 into the main `apt-get install --no-install-recommends -y` layer so the apt cache stays consistent and the layer count is reduced. - README: document `xmllint` as a release-manager prerequisite (used by publish-to-maven.sh to parse Nexus staging responses).
pgwhalen
approved these changes
May 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
1 more reviewer2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
PR #77 added the runtime side of the fat-JAR design: NativeLibraryLoader reads org/apache/datafusion/<os>/<arch>/lib<datafusion_jni>.<ext> from the JAR and extracts it on demand. But nothing in the repo actually produces a JAR with more than one platform's lib inside — core/pom.xml's host-activated profile bundles only the host's lib, and the existing release scripts (dev/release/create-tarball.sh etc.) only do source tarballs. A consumer pulling the artifact from Maven Central today gets a JAR that works on the platform it was built on and nowhere else.
This PR adds the build side: a release-manager script that drives two Docker containers for the Linux arches and the host's own Rust toolchain for the macOS arches, assembles all four .so/.dylib files into a single JAR, and installs it into a temporary local Maven repo. A second script (not yet exercised in CI) signs and uploads that repo to Apache Nexus staging.
The structure mirrors datafusion-comet's release tooling (dev/release/comet-rm/Dockerfile, build-release-comet.sh, publish-to-maven.sh), simplified for this project: single module pair (no Spark/Scala matrix) and macOS libs built natively on the RM's macOS host (no OSXCross / Xcode SDK plumbing).
What changes are included in this PR?
New release tooling under dev/release/:
Two follow-on fixes uncovered by the dry run:
Are these changes tested?
End-to-end dry run on macOS aarch64 produced datafusion-java-0.1.0-SNAPSHOT.jar (175 MB) containing exactly the four expected resource entries:
./mvnw test is green on the branch (308 tests run, 0 failures, 13 skipped) after the standard `cargo build` (debug) precondition from CLAUDE.md.
`publish-to-maven.sh` is not exercised in this PR. Validating it requires real Apache Nexus credentials and a real GPG-signed release candidate, both of which are out of scope for the build-pipeline dry run.
Are there any user-facing changes?
No code or API changes. Release-manager-facing tooling only.