The Java DataFrame API currently exposes only select and filter as
transformations. This PR rounds out the trivially-implementable subset of
DataFusion's Rust DataFrame API so users can build small pipelines without
falling back to SQL strings.
What changes are included in this PR?
Four new methods on DataFrame, each backed by a thin JNI bridge that calls
the corresponding DataFusion operator:
limit(int fetch) / limit(int skip, int fetch) — take the first N rows, optionally after skipping. Negative arguments throw IllegalArgumentException.
distinct() — deduplicate rows across all columns.
dropColumns(String... columnNames) — inverse of select. Unknown column names are silently ignored (matches DataFusion's drop_columns semantics).
withColumnRenamed(String oldName, String newName) — rename a column. Unknown old names are a no-op (matches DataFusion's with_column_renamed semantics).
All four follow the existing pattern: throw IllegalStateException on closed/collected handles, and leave the receiver usable so chaining off a shared source is non-destructive.
Are these changes tested?
Yes. DataFrameTransformationsTest gains 12 new tests covering happy paths,
non-destructive semantics, invalid arguments, and the silent-ignore behaviors
for dropColumns/withColumnRenamed. The existing methodsThrowAfterClose
and methodsThrowAfterCollect lifecycle tests are extended to cover the new
methods.
Are there any user-facing changes?
Yes — four new public methods on org.apache.datafusion.DataFrame. No
breaking changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The Java DataFrame API currently exposes only select and filter as
transformations. This PR rounds out the trivially-implementable subset of
DataFusion's Rust DataFrame API so users can build small pipelines without
falling back to SQL strings.
What changes are included in this PR?
Four new methods on DataFrame, each backed by a thin JNI bridge that calls
the corresponding DataFusion operator:
All four follow the existing pattern: throw IllegalStateException on closed/collected handles, and leave the receiver usable so chaining off a shared source is non-destructive.
Are these changes tested?
Yes. DataFrameTransformationsTest gains 12 new tests covering happy paths,
non-destructive semantics, invalid arguments, and the silent-ignore behaviors
for dropColumns/withColumnRenamed. The existing methodsThrowAfterClose
and methodsThrowAfterCollect lifecycle tests are extended to cover the new
methods.
Are there any user-facing changes?
Yes — four new public methods on org.apache.datafusion.DataFrame. No
breaking changes.