Repository navigation

Feat/table provider ffi - #103

Draft

timsaucer wants to merge 22 commits into

apache:mainfrom

timsaucer:feat/table-provider-ffi

Commits on Jun 10, 2026
- WIP using rerun specific table options
  timsaucer
  committed
  d1c4a9b
- WIP on making table providers spark data sources with a working example
  timsaucer
  committed
  baa9ed3
- feat(spark): add Spark DataSource V2 connector + pyspark FFI demo
  
  timsaucer
  and
  claude
  committed
  e7b5848
- refactor(build): consolidate Rust crates into a Cargo workspace
  
  timsaucer
  and
  claude
  committed
  7ee7c9d
- feat(examples): pass user options through FFI table provider demo
  
  timsaucer
  and
  claude
  committed
  8db9d4a
- update examples to build after last commit
  timsaucer
  committed
  e8c70a9
- feat(spark): per-partition payload + preferred locations in FFI factory
  
  timsaucer
  and
  claude
  committed
  088474d
- docs(examples): update SPARK_INTEGRATION for PartitionInfo + per-slice createProvider
  
  timsaucer
  and
  claude
  committed
  f7d3972
- feat(spark): SupportsReportPartitioning via optional reportPartitioning hook
  
  timsaucer
  and
  claude
  committed
  daa3ba5
Commits on Jun 11, 2026
- feat(spark): shared-scan mode with per-executor provider cache
  timsaucer
  committed
  c926d3d
- refactor(spark): fold scan planning/execution into connector cdylib
  
  timsaucer
  and
  claude
  committed
  1cffd93
- docs: rewrite examples README and move Spark guide into spark/
  
  timsaucer
  and
  claude
  committed
  1f73a6f
- feat(spark): add datafusion-spark-bridge SDK for static bridges
  
  timsaucer
  and
  claude
  committed
  e9f3f61
- feat(spark): ScanBackend dispatch + one-method-minimum factory
  
  timsaucer
  and
  claude
  committed
  dc909ce
- feat(spark): reusable native loader + bridge packaging recipe
  
  timsaucer
  and
  claude
  committed
  45c9613
- feat(spark): bridge scaffold generator
  
  timsaucer
  and
  claude
  committed
  cc35958
- refactor(spark)!: remove the FFI provider path
  
  timsaucer
  and
  claude
  committed
  9e47f0e
- docs: scrub dual-path language after FFI removal
  
  timsaucer
  and
  claude
  committed
  13bca92
- refactor(spark): rename optionsProtoBytes, fix stale createProvider docs
  
  timsaucer
  and
  claude
  committed
  a1815a8
- refactor(spark): rename LegacyMode to PerPartitionMode
  
  timsaucer
  and
  claude
  committed
  4b99734
Commits on Jun 12, 2026
- refactor(spark): move bridge scaffold from dev/ to spark/scaffold/
  
  timsaucer
  and
  claude
  committed
  827068c
- add support for fixed sized list widening
  timsaucer
  committed
  0cc474a

Feat/table provider ffi by timsaucer · Pull Request #103 · apache/datafusion-java · GitHub

Repository navigation

Feat/table provider ffi - #103

Feat/table provider ffi#103
timsaucer wants to merge 22 commits into
apache:mainfrom
timsaucer:feat/table-provider-ffi

Commits on Jun 10, 2026

WIP using rerun specific table options

WIP on making table providers spark data sources with a working example

feat(spark): add Spark DataSource V2 connector + pyspark FFI demo

refactor(build): consolidate Rust crates into a Cargo workspace

feat(examples): pass user options through FFI table provider demo

update examples to build after last commit

feat(spark): per-partition payload + preferred locations in FFI factory

docs(examples): update SPARK_INTEGRATION for PartitionInfo + per-slice createProvider

feat(spark): SupportsReportPartitioning via optional reportPartitioning hook

Commits on Jun 11, 2026

feat(spark): shared-scan mode with per-executor provider cache

refactor(spark): fold scan planning/execution into connector cdylib

docs: rewrite examples README and move Spark guide into spark/

feat(spark): add datafusion-spark-bridge SDK for static bridges

feat(spark): ScanBackend dispatch + one-method-minimum factory

feat(spark): reusable native loader + bridge packaging recipe

feat(spark): bridge scaffold generator

refactor(spark)!: remove the FFI provider path

docs: scrub dual-path language after FFI removal

refactor(spark): rename optionsProtoBytes, fix stale createProvider docs

refactor(spark): rename LegacyMode to PerPartitionMode

Commits on Jun 12, 2026

refactor(spark): move bridge scaffold from dev/ to spark/scaffold/

add support for fixed sized list widening

Feat/table provider ffi by timsaucer · Pull Request #103 · apache/datafusion-java · GitHub

Repository navigation

Feat/table provider ffi - #103

Feat/table provider ffi#103timsaucer wants to merge 22 commits intoapache:mainapache/datafusion-java:mainfrom timsaucer:feat/table-provider-ffitimsaucer/datafusion-java:feat/table-provider-ffiCopy head branch name to clipboard

Commits

Commits on Jun 10, 2026

Commits on Jun 11, 2026

Commits on Jun 12, 2026

Feat/table provider ffi#103
timsaucer wants to merge 22 commits into
apache:mainfrom
timsaucer:feat/table-provider-ffi