Is your feature request related to a problem or challenge?
The DataFrame API gained dropColumns and withColumnRenamed in #30
but the most common column-shaping primitive — adding or replacing a
column with an expression — is still missing. unnestColumns for
struct / list flattening is in the same family.
Describe the solution you'd like
- DataFrame.withColumn(String name, String expr). The expression is
a SQL fragment parsed via DataFusion's parse_sql_expr, matching the
convention already used by DataFrame.filter(String). Behaviour
mirrors DataFusion::DataFrame::with_column — replaces a column of
the same name in place, otherwise appends.
- DataFrame.unnestColumns(String... columns) and an overload
accepting an UnnestOptions value (preserve-nulls, recursions). Maps
to unnest_columns / unnest_columns_with_options.
- Tests in DataFrameTransformationsTest (round-trip on a small
fixture, schema check, error on unknown column).
Describe alternatives you've considered
SELECT a + b AS c, * via ctx.sql(...). Works for withColumn, but
requires the DataFrame to be registered as a table first; awkward for
intermediate transformations.
Additional context
Follows the same JNI shape as #19's filter(String) — parse_sql_expr
on the native side, no Java-side Expr model required. Keeps this
issue independent of #N (joins issue) where an Expr builder may
eventually be needed.
Reactions are currently unavailable
Is your feature request related to a problem or challenge?
The DataFrame API gained dropColumns and withColumnRenamed in #30
but the most common column-shaping primitive — adding or replacing a
column with an expression — is still missing. unnestColumns for
struct / list flattening is in the same family.
Describe the solution you'd like
a SQL fragment parsed via DataFusion's parse_sql_expr, matching the
convention already used by DataFrame.filter(String). Behaviour
mirrors DataFusion::DataFrame::with_column — replaces a column of
the same name in place, otherwise appends.
accepting an UnnestOptions value (preserve-nulls, recursions). Maps
to unnest_columns / unnest_columns_with_options.
fixture, schema check, error on unknown column).
Describe alternatives you've considered
SELECT a + b AS c, * via ctx.sql(...). Works for withColumn, but
requires the DataFrame to be registered as a table first; awkward for
intermediate transformations.
Additional context
Follows the same JNI shape as #19's filter(String) — parse_sql_expr
on the native side, no Java-side Expr model required. Keeps this
issue independent of #N (joins issue) where an Expr builder may
eventually be needed.