Repository navigation
Commit e2a53bb
authored
feat: Add a non-JVM read path for Lance data sources (#6943)
* feat: Add a non-JVM read path for Lance data sources
Adds LanceSource and teaches the DuckDB offline store to read it, completing the
read half of #6899. LanceFormat landed in #6925 as a format descriptor; nothing
read Lance until now.
Lance already worked through SparkSource, which drives its reader generically
from table_format.format_type.value and table_format.properties. What was
missing is a path that needs no JVM, which is also the real test of whether the
DataSource abstraction is engine-agnostic rather than Spark-agnostic in name
only.
Placement: a new source read by the existing DuckDB store, rather than a Lance
offline store or an extension of FileSource.
FileSource is the wrong host. Its format axis is already taken by file_format,
so adding table_format would give one source two overlapping format axes. It is
also read by two stores with incompatible contracts: duckdb._read_data_source
dispatches on type, while dask._read_datasource has no dispatch seam and reads
file_options.uri unconditionally as Parquet, and asserts isinstance(...,
FileSource) in three places. Decisively, Lance's catalog addressing has no path
to put in FileSource.path, so the catalog-based layer would not fit the class
even if the path-based one did.
A Lance offline store would be the wrong 120 lines. duckdb.py is a binding that
injects reader and writer callbacks into the engine in ibis.py, so a Lance store
would be a near-copy of it plus a repo_config entry, and would force a choice
between Lance and Parquet instead of mixing them in one feature service.
Reading a source as ibis.memtable(arrow_table) in the DuckDB store already has
two precedents, IcebergSource and MlflowDatasetSource. Following them leaves
ibis.py untouched, so the point-in-time join, TTL handling, field mapping and
ODFVs work unchanged, and no edit to repo_config.py or data_source.py is needed
because CUSTOM_SOURCE plus data_source_class_type is self-describing.
Both addressing modes work: a uri, and catalog/namespace/table through
namespace_client and table_id.
Pin semantics follow what was argued on #5782 and #6925: a pin selects data,
never shape. get_table_column_names_and_types reads the pinned schema so
feast apply infers what reads will actually see; a pre-flight check fails with a
message naming the pin when a pinned version cannot satisfy the declared schema;
and vector widths go through _validate_vector_field_lengths from #6909 rather
than a second validator.
Tests use the dir namespace implementation, which exercises the same
namespace_client and table_id code path as a remote catalog with no server
required. 43 tests, including a demonstration that a tag pin returns earlier
data after the dataset has been overwritten for the same entity and timestamp.
Read-only for now: _write_data_source is untouched, so a LanceSource is not yet
a persist target and there is no SavedDatasetLanceStorage.
Signed-off-by: hao-xu5 <hxu44@apple.com>
* fix: Address Lance read review feedback
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
---------
Signed-off-by: hao-xu5 <hxu44@apple.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>1 parent a5eaa36 commit e2a53bb
4 files changed
Lines changed: 1282 additions & 0 deletions
File tree
- sdk/python
- feast/infra
- data_sources/contrib/lance
- offline_stores
- tests/unit/infra/data_sources
Lines changed: 9 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -0,0 +1,9 @@ | |||
| 1 | + from feast.infra.data_sources.contrib.lance.lance_source import ( | ||
| 2 | + LanceSource, | ||
| 3 | + validate_lance_source_schema, | ||
| 4 | + ) | ||
| 5 | + | ||
| 6 | + __all__ = [ | ||
| 7 | + "LanceSource", | ||
| 8 | + "validate_lance_source_schema", | ||
| 9 | + ] | ||
0 commit comments