Integrate datafusion-distributed with Python - #1611
Conversation
Sorry, something went wrong.
| .with_analyzer_rule(Arc::new(crate::analyzer::ResolveLambdaVariables::new())); | ||
|
|
||
| if distributed { | ||
| builder = builder.with_distributed_planner(); | ||
| } |
There was a problem hiding this comment.
Rather than letting external system inject their own QueryPlanners, this allows just plumbing datafusion-distributed query planner from within the Rust world, which is actually a pretty easy thing to do.
Sorry, something went wrong.
| "rt-multi-thread", | ||
| "sync", | ||
| ] } | ||
| tonic = { workspace = true } |
There was a problem hiding this comment.
This is the kind of thing that could be easily hidden behind a flag.
Sorry, something went wrong.
| @classmethod | ||
| def from_session_builder( | ||
| cls, | ||
| session_builder: Callable[[WorkerQueryContext], SessionContext], |
There was a problem hiding this comment.
In datafusion-distributed, this callback configures DataFusion at the SessionStateBuilder level, but this project does not contain an analogous structure, the closest is just a SessionContext, so users are expected to build a full SessionContext here out of another SessionContext, even if in datafusion-distributed the contract is SessionStateBuilder in and SessionState out.
Sorry, something went wrong.
Quick Try
Run two workers in separate terminals:
Download the test dataset:
Then run the query or print the distributed plan:
Or print the distributed plan:
Which issue does this PR close?
N/A.
Rationale for this change
This lets Python users wire datafusion-distributed into datafusion-python: discover workers from Python, spawn worker servers, and inspect distributed plans.
One integration wrinkle: upstream examples usually build from SessionStateBuilder, while this package exposes SessionConfig/SessionContext as the main public path. This PR installs the distributed planner when a SessionContext is built from a distributed config.
What changes are included in this PR?
Are there any user-facing changes?
Yes. New distributed APIs are exposed from Python, plus new examples. No intended breaking changes.