Add FFI query planner support by timsaucer · Pull Request #1677 · apache/datafusion-python · GitHub

Which issue does this PR close?

Related to #1612. This PR does not close it, but provides the FFI query planner plumbing that a datafusion-distributed integration can build on.

This is part 1 of 3 in the split of #1672. These are enabled as a github stack so you should be able to swab between the 3 PRs in github interface (above, next to the "Open" oval).

Rationale for this change

Extension libraries (for example distributed execution engines) need to supply their own QueryPlanner to a SessionContext without compiling against the datafusion-python crate. This PR exposes the query planner over the FFI boundary, following the same PyCapsule pattern used for table providers and catalogs.

What changes are included in this PR?

SessionContext.with_query_planner(planner) installs a planner exported via a __datafusion_query_planner__ PyCapsule, preserving existing session state and codec settings.
SessionContext.__datafusion_query_planner__() exports the current planner so another planner can wrap it as an explicit fallback (a session holds exactly one planner; layering is explicit delegation).
A RuntimeAwareQueryPlanner adapter binds foreign planners to the Tokio runtime owned by datafusion-python.
New example crate datafusion-ffi-query-planner-example demonstrating a real three-library plan exchange (host, provider library, planner library as separate cdylibs), including session config transfer via SessionConfig.with_extension.
New docs/source/contributor-guide/ffi.md sections covering the query planner capsule protocol and the three-library setup.

Are there any user-facing changes?

New public APIs: SessionContext.with_query_planner and SessionContext.__datafusion_query_planner__. A new example crate ships under examples/. No breaking changes to existing APIs.

AI Disclosure: This code was written in part by an AI agent.:

This API is the main reason for this PR. Here we allow changing out the default query planner with a user provided query planner.

In order to prove that the 3 library approach works where we have different codecs and different execution plans provided, we are adding a second test library. This way we can make sure there is no accidental ability to reach into a foreign code block.

As the docstring says, the purpose of this is to make sure we attach the runtime handle when needed.

We need our session context to export it's own query planner because we have a use case where one query planner can depend on another. This is already supported by datafusion-distributed, so we want to be certain we support it here.

I'm adding this to the query planner example because it's a very common pattern that we will need custom configs for the query planner, so it is reasonable to need insurance that configs pass over the FFI boundary properly and to use as a demonstration to anyone who is providing such a library.

this is needed for ballista, thanks Tim for example

The FFI test wheel artifact now bundles two projects, so upload-artifact preserves a `<project>/dist/` prefix instead of placing the wheels at the artifact root. The install step globbed `wheels/*.whl`, which no longer matched them, so the FFI wheels were silently skipped and the FFI unit tests failed with `ModuleNotFoundError: No module named 'datafusion_ffi_example'`. Install the recursive `find` results instead of re-globbing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Appears consistent with the rest of the FFI plumbing

Generally wonder if this builder pattern feels pythonic. Consistent with what's already here so no action requested. Didn't look at how many withs there are but

ctx = SessionContext(config, planner)

feels a little more intuitive than

ctx = SessionContext().with_query_planner(planner)

Good point! Also worth updating the skill to match this pattern

thanks @timsaucer cant want to get this integrated

Nit, MyPlannerConfig to have names aligned,

this is needed for ballista, thanks Tim for example

is this session context be parameter of method call on the line 119 ? are those two different sessions ?

timsaucer added 4 commits August 6, 2026 10:54

Add FFI query planner support

9ba147c

AI Disclosure: This code was written in part by an AI agent.:

Add three-library FFI planner example

47aea03

AI Disclosure: This code was written in part by an AI agent.:

Update FFI query planner integration

05ab5da

AI Disclosure: This code was written in part by an AI agent.:

add rat

27147f3

timsaucer mentioned this pull request Aug 7, 2026

Add FFI query planner support with composable extension codecs #1672

Closed

timsaucer commented Aug 7, 2026

View reviewed changes

ntjohnson1 approved these changes Aug 14, 2026

View reviewed changes

Merge branch 'main' into feat/ffi-query-planner-core

c744579

milenkovicm approved these changes Aug 17, 2026

View reviewed changes

+                    - name: Build FFI query planner test library
+                      if: matrix.python-tag == 'abi3'
+                      uses: PyO3/maturin-action@v1
+                      with:
+                        target: x86_64-unknown-linux-gnu
+                        manylinux: "2_28"
+                        working-directory: examples/datafusion-ffi-query-planner-example
+                        args: --out dist
+                        rustup-components: rust-std

+                  pub fn __datafusion_query_planner__<'py>(
+                      &self,
+                      py: Python<'py>,
+                  ) -> PyResult<Bound<'py, PyCapsule>> {

+                          observations: Arc::clone(&self.observations),
+                      });
+                      let runtime = get_tokio_runtime().handle().clone();
+                      let ctx_provider = Arc::new(SessionContext::new()) as Arc<dyn TaskContextProvider>;

                       Ok(())
                   }
+                  pub fn with_query_planner(&self, planner: Bound<'_, PyAny>) -> PyDataFusionResult<Self> {

+              struct RuntimeAwareQueryPlanner {
+                  planner: FFI_QueryPlanner,
+              }

+              #[derive(Clone, Debug)]
+              pub(crate) struct PlannerConfig {
+                  pub max_rows: usize,
+              }

                       """
                       self.ctx.add_physical_optimizer_rule(rule)
+                  def with_query_planner(

+              }
+              #[pymethods]
+              impl PlannerConfig {

Add FFI query planner support by timsaucer · Pull Request #1677 · apache/datafusion-python · GitHub

Uh oh!

Add FFI query planner support - #1677

Add FFI query planner support#1677timsaucer wants to merge 6 commits intomainapache/datafusion-python:mainfrom feat/ffi-query-planner-coreapache/datafusion-python:feat/ffi-query-planner-coreCopy head branch name to clipboard

Conversation

timsaucer commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ntjohnson1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milenkovicm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add FFI query planner support#1677
timsaucer wants to merge 6 commits into
mainfrom
feat/ffi-query-planner-core

timsaucer commented Aug 7, 2026 •
edited
Loading