feat: initial seed of Apache DataFusion Java bindings - #1
Merged
andygrove merged 5 commits intoMay 13, 2026
Merged
andygrove merged 5 commits into
andygrove merged 5 commits into
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
Seed the project with a minimal end-to-end JNI binding from the JVM to Apache DataFusion, plus the build, format, and license-check tooling needed for ongoing contribution. Java surface (org.apache.datafusion): - SessionContext: AutoCloseable session, sql(String) returning a lazy DataFrame, registerParquet(String, String) for registering local Parquet files as SQL tables. - DataFrame: AutoCloseable, collect(BufferAllocator) executes the plan and returns result batches as an Arrow ArrowReader via the Arrow C Data Interface. collect() consumes the DataFrame; close() releases the native plan if never collected. Native side (native/, crate datafusion-jni): - JNI entry points for SessionContext create/close/registerParquet/ createDataFrame and DataFrame collect/close. - Results are exported as FFI_ArrowArrayStream so the JVM reads batches without per-row JNI crossings or row-by-row copies. Build and contributor tooling: - pom.xml with Maven wrapper, JUnit 5, Arrow 19, JDK 17 toolchain. - apache-rat-plugin (license-header check) and spotless-maven-plugin (google-java-format) both bound to the verify phase. - Makefile targets for native build, JVM build, test, clean, and TPC-H SF1 test data generation via tpchgen-cli. - GitHub Actions workflow running spotless:check and cargo fmt --check on push and pull_request to main.
andygrove
marked this pull request as ready for review
May 12, 2026 23:28
comphead
reviewed
May 13, 2026
Comment thread
native/Cargo.toml
Outdated
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
comphead
reviewed
May 13, 2026
Comment thread
src/main/java/org/apache/datafusion/NativeLibraryLoader.java
Outdated
|
|
||
| public static synchronized void loadLibrary() { | ||
| if (!loaded) { | ||
| System.loadLibrary("datafusion_jni"); |
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
lets make the string as constant?
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
There was a problem hiding this comment.
updated
Sorry, something went wrong.
Copy link
Copy Markdown
Member
Author
|
Note that no GitHub workflows will run until after this PR is merged. Also we cannot create any issues until this PR is merged because it contains the .asf.yaml change to enable GitHub issues. |
Sorry, something went wrong.
Re-enable datafusion's default features (parquet, sql) and add arrow dependency with the ffi feature so FFI_ArrowArrayStream, ctx.sql, and register_parquet compile again.
comphead
reviewed
May 13, 2026
Comment thread
src/main/java/org/apache/datafusion/SessionContext.java
|
|
||
| private static native long createSessionContext(); | ||
|
|
||
| private static native long createDataFrame(long handle, String sql); |
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
maybe we can specify what kind of handle expected in methods? like sessionHandle?
Sorry, something went wrong.
comphead
approved these changes
May 13, 2026
comphead
left a comment
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Thanks @andygrove, you actually dont need an approve to merge the first PR :)
Sorry, something went wrong.
Copy link
Copy Markdown
|
🎉 |
Sorry, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Seed the project with a minimal end-to-end JNI binding from the JVM to Apache DataFusion, plus the build, format, and license-check tooling needed for ongoing contribution.
What is in this PR
Java surface (org.apache.datafusion)
Native side (native/, crate datafusion-jni)
Build and contributor tooling
Project status
This is the first code drop into a brand-new repository. The README labels the project as early development: the API is small and will change without notice, and there is no published release.
A Roadmap section in the README outlines near-term priorities: session configuration, full SessionContext/DataFrame API parity with the Rust side, JVM-side plan construction via DataFusion's Protobuf representation, and Java-defined vectorized expressions over Arrow.
Verification
Locally, on this branch:
The optional TPC-H integration test runs after make tpch-data (requires tpchgen-cli); it reads lineitem.parquet via registerParquet and asserts SELECT COUNT(*) returns 6,001,215.