feat(dataframe): expose withColumn and unnestColumns - #54
Merged
andygrove merged 1 commit intoMay 17, 2026
Merged
andygrove merged 1 commit into
andygrove merged 1 commit into
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
Closes apache#41 withColumn(name, expr) takes a SQL fragment, parses it via df.parse_sql_expr (the same convention filter(String) uses), and calls DataFusion's with_column. Replaces a column of the same name in place; otherwise appends. unnestColumns(String...) and unnestColumns(UnnestOptions, String...) route to unnest_columns_with_options. UnnestOptions exposes the preserveNulls knob (default true, matching upstream). Recursions are deferred to a follow-up since they need a richer column-pair representation (RecursionUnnestOption with input/output/depth). Tests: 11 new transformation tests, 2 UnnestOptions setter tests, plus the two close/collect tests extended for the new methods. The unnest tests cover both preserveNulls=true (keep null rows) and preserveNulls=false (drop null rows). withColumn tests cover both the append-new and replace-existing branches. make test (90 tests, 0 failures) and cargo clippy/fmt are clean.
andygrove
approved these changes
May 17, 2026
andygrove
left a comment
Copy link
Copy Markdown
Member
There was a problem hiding this comment.
Thanks @LantaoJin
Sorry, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
DataFrame gained dropColumns and withColumnRenamed in #30 but the most common column-shaping primitive -- adding or replacing a column from a SQL expression -- was still missing. unnestColumns is in the same family for struct / list flattening. The issue (#41) lists both as a single unit of work.
withColumn and unnestColumns follow the same JNI shape as filter(String) from #19: the SQL fragment is parsed on the native side via df.parse_sql_expr, so no Java-side Expr model is required. That keeps this PR independent of any future joins / Expr-builder work.
What changes are included in this PR?
Out of scope (filed separately):
Are these changes tested?
Yes, 13 new tests, plus two existing close/collect tests extended.
Are there any user-facing changes?
Yes, purely additive. New public API:
No API removals, no deprecations, no behavior change for existing callers.