feat: expose SessionContext.write_csv, write_json, write_parquet - #1569
Closed
timsaucer wants to merge 1 commit into
Closed
feat: expose SessionContext.write_csv, write_json, write_parquet#1569timsaucer wants to merge 1 commit into
timsaucer wants to merge 1 commit into
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
Adds three plan-level writers on SessionContext that mirror the upstream datafusion::execution::context API. Each takes an ExecutionPlan and an output directory path; the plan is executed and its results are written one partition per file inside that directory. These complement the existing DataFrame.write_* methods, which are the right choice when callers need finer control (CSV header, Parquet compression, write options). The new SessionContext methods are the right choice when a caller already holds a physical ExecutionPlan (for example after custom physical optimizer rules or hand-built plans) and just wants the rows materialized. Related to apache#462. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Member
Author
|
I am closing this PR because it provides extremely little value. We should likely instead update the check-upstream skill to not recommend exposing these functions. |
Sorry, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #. No dedicated tracking issue; related to umbrella issue #462 (interface design / user stories).
Rationale for this change
DataFusion's SessionContext exposes write_csv, write_json, and write_parquet methods that take an already-built Arc<dyn ExecutionPlan> and a target path. These complement the existing per-DataFrame write methods and are the right entry point when a caller already holds a physical plan -- for example after running custom physical optimizer rules (recently exposed via PR #1557) or after constructing a plan directly. The Python bindings did not surface them.
What changes are included in this PR?
Are there any user-facing changes?
Yes. Three new public methods on datafusion.SessionContext:
No breaking changes.