Summary

Add native, read-only Object Table support to Paimon Rust and the DataFusion integration. The implementation follows Paimon Java's fixed Object Table schema and path semantics, so Object Tables no longer require an externally registered table engine for reads.

Changes

• Add ObjectTable and ObjectEntry with the Java-compatible fixed schema: path, name, length, mtime, atime, and owner.
• Align creation with Java behavior:
  • ignore user-defined fields and persist the fixed six-column schema
  • default a missing or blank path to the catalog table directory
  • preserve an explicit non-empty path
  • load legacy filesystem metadata without path from the table directory
• Recursively expose object metadata with relative paths through a lazy OpenDAL stream.
• Execute DataFusion Object Table scans with StreamingTableExec:
  • defer recursive listing from planning to execution
  • emit bounded RecordBatches using the session batch size
  • construct only columns requested by the pushed projection
  • avoid global sorting in the scan; explicit SQL ORDER BY remains a DataFusion operator
  • stop polling after K files for pushed LIMIT K queries without ORDER BY
  • do not preallocate memory from an untrusted SQL limit
• Route Object Tables natively from filesystem and REST catalogs.
  • Build filesystem I/O from the Object Table's own path, including cross-scheme locations.
  • Use GetTableResponse.path for REST Object Tables, matching Paimon Java and preserving REST data-token handling.
• Add a DataFusion table provider with projection-aware, bounded-memory streaming and limit support.
• Keep Object Tables read-only and reject branches, time travel, and writes.
• Mark the new LoadedTable API #[non_exhaustive] before its first v0.4 publication so future native table kinds do not repeatedly break downstream exhaustive matches.
• Keep non-native table types on the existing external table-engine path.

Testing

• cargo +1.98.0 clippy --locked -p paimon -p paimon-datafusion --all-targets --features fulltext,vortex -- -D warnings
• cargo test -p paimon-datafusion --test object_table
• cargo test -p paimon-datafusion --test table_type_routing
• object-table filesystem creation/default-path/legacy-path unit tests
• limited listing stops polling after K files
• a huge limit on an empty location does not preallocate or panic
• ordered LIMIT query regression test
• recursive listing yields its first row without draining the namespace
• projected scans emit multiple bounded batches at the configured session batch size
• zero-column projection regression through COUNT(*)
• cargo test -p paimon --test rest_catalog_test test_load_table_constructs_native_object_table -- --exact
• cargo check -p paimon -p paimon-datafusion -p paimon-rest-server --all-targets --features fulltext,vortex
• cargo fmt --all -- --check
• git diff --check

Notes

• OpenDAL does not expose access time or owner portably, so atime is currently 0 and owner is NULL; modification time and length are populated when available.
• LoadedTable was introduced after v0.3.0 and has not appeared in a published release. It is made non-exhaustive here before the v0.4 API is published.
• The Windows REST construction test remains active; only its local recursive-listing assertion is skipped because the existing OpenDAL drive-root StripPrefixError is tracked separately.
• This is complementary to fix(datafusion): keep metadata queries available without table engines #747, which keeps metadata queries available when unsupported external engines are not registered; neither PR depends on the other.