feat: Extend Feast's DataSource to natively support Iceberg REST Catalog-backed tables · Issue #6499 · feast-dev/feast · GitHub

Problem

Teams running Unity Catalog can read UC-governed Delta tables today via the Spark offline store and SparkSource (e.g. catalog.schema.table), but there is no first-class integration for:

UC feature table registration — Feast feast apply does not register feature views as UC feature tables (primary keys, Features UI, Catalog Explorer discovery).
UC-backed materialization — feast materialize writes to the online store but does not persist governed Delta tables back to UC or sync UC lineage.
Databricks-aware configuration — Users must hand-wire Spark session / cluster config instead of a dedicated offline store type with UC defaults.

UC tables are Spark-accessible Delta tables, so the read path works generically. The gap is the UC governance/registration layer on top (feature table metadata, primary keys, discovery, lineage) that Unity Catalog provides natively.

Motivation / use cases

Platform teams want one feature definition in Feast (entities, TTLs, PIT joins, serving API) while keeping UC as the governance catalog for discovery and access control.
Data scientists expect feature tables to appear in Catalog Explorer / Features UI, not only in the Feast registry.
ML engineers need lineage from source UC tables → feature tables → models without maintaining parallel metadata.
Teams evaluating Feast vs Databricks Feature Store need a clear path when UC registration is a hard requirement.

Current state

Capability	Supported today?
Read UC Delta tables via Spark	✅ via Spark offline store + SparkSource
Point-in-time training joins	✅
UC feature table registration on feast apply	❌
Materialize to UC Delta + online store	❌ (offline_write_batch not supported for Spark offline store)
Databricks-specific offline store config	❌
Import existing UC feature tables as Feast views	❌

Related issues: #2406 (Delta/Iceberg/Hudi table formats), #764 (Databricks Spark runner, closed).

Proposed solution (phased)

L1 — Databricks-aware Spark offline store (read path)

New contrib type e.g. databricks_uc extending Spark offline store
feature_store.yaml config for workspace host, default catalog/schema, auth
UnityCatalogSource (or extended SparkSource) with UC path validation

L2 — UC registration on feast apply

Hook after registry apply: register/update UC feature tables via FeatureEngineeringClient (or UC REST APIs)
Map Entity.join_keys → UC primary key constraints
Sync Feast tags, description, owner to UC table properties
Opt-out per feature view: uc_config.register_as_feature_table: false

L3 — UC-backed materialization

Extend materialization to MERGE/append into catalog.schema.<feature_view> Delta tables
Continue writing to configured online store for low-latency serving
Emit OpenLineage events with UC FQNs

L4 (optional) — Bidirectional sync

feast import-uc-table to scaffold FeatureView from existing UC feature tables
UC lineage bridge for apply/materialize events

Example API sketch

# feature_store.yaml
offline_store:
  type: databricks_uc
  workspace_host: https://adb-xxx.azuredatabricks.net
  catalog: prod_ml
  schema: features
  uc_registration:
    enabled: true
    on_apply: true

customer_features = FeatureView(
    name="customer_features",
    entities=[customer],
    source=UnityCatalogSource(
        table="prod_raw.bronze.transactions",
        timestamp_field="event_timestamp",
    ),
    schema=[...],
    uc_config={
        "catalog": "prod_ml",
        "schema": "features",
        "table": "customer_features",
        "register_as_feature_table": True,
        "materialize_offline": True,
    },
)

Expected behavior

feast apply
  → Feast registry updated (existing)
  → UC: create/register feature table, set primary key, sync metadata

feast materialize-incremental <end>
  → Read from UC sources via Spark
  → Write Delta to UC feature table (new)
  → Write to online store (existing)

@ntkathole I have implemented L1, L2 and L3
I am also planning to open a PR for L4 soon

@falloficaruss I did further research on it and I think instead of reading UC-governed tables via the Spark offline store, we should build on the Iceberg REST Catalog standard rather than tying it specifically to Spark/Databricks UC connect.

We should extend Feast's DataSource to natively support Iceberg REST Catalog-backed tables. Something like:

class IcebergRestCatalogSource(DataSource):
    """Base — works with UC, Polaris, Nessie, any Iceberg REST catalog."""
    endpoint: str              # Iceberg REST catalog endpoint
    warehouse: str             # catalog / warehouse name
    namespace: str             # schema / namespace
    table: str                 # table name
    timestamp_field: str
    token_env_var: str         # auth via env var 
    credential_vending: bool = True


class UnityCatalogSource(IcebergRestCatalogSource):
    """Extends base with UC-specific governance features."""
    register_as_feature_table: bool = True
    sync_lineage: bool = True

Any offline store that encounters an IcebergRestCatalogSource resolves it via the standard Iceberg REST client, then reads data with its own engine.

feast apply
  → Feast registry updated (existing)
  → Provider detects UnityCatalogSource with register_as_feature_table=True
  → Calls UC REST API: register feature table, set primary keys, sync metadata

feast materialize
  → Offline store sees IcebergRestCatalogSource
  → IcebergRestClient resolves metadata + vends credentials (standard protocol)
  → Reads via chosen engine (Spark/Trino/DuckDB)
  → Writes to online store (existing)
  → Optionally writes governed Delta/Iceberg back to UC (Last Phase)

Thoughts ?

@ntkathole Moving connection config to the DataSource keeps the offline store config generic and leaves room for other engines (Trino, DuckDB) to serve UC tables later. So I would agree with your points.
For L4 (bidirectional sync), that would naturally build on the UnityCatalogSource subclass with its governance flags. Does that fit with the direction you have in mind, or would you prefer to keep that separate?

Right

feat: Extend Feast's DataSource to natively support Iceberg REST Catalog-backed tables · Issue #6499 · feast-dev/feast · GitHub

Repository navigation

feat: Extend Feast's DataSource to natively support Iceberg REST Catalog-backed tables #6499

Problem

Motivation / use cases

Current state

Proposed solution (phased)

L1 — Databricks-aware Spark offline store (read path)

L2 — UC registration on feast apply

L3 — UC-backed materialization

L4 (optional) — Bidirectional sync

Example API sketch

Expected behavior

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

feat: Extend Feast's DataSource to natively support Iceberg REST Catalog-backed tables · Issue #6499 · feast-dev/feast · GitHub

Repository navigation

feat: Extend Feast's DataSource to natively support Iceberg REST Catalog-backed tables #6499

Description

Problem

Motivation / use cases

Current state

Proposed solution (phased)

L1 — Databricks-aware Spark offline store (read path)

L2 — UC registration on feast apply

L3 — UC-backed materialization

L4 (optional) — Bidirectional sync

Example API sketch

Expected behavior

Activity

falloficaruss commented on Jun 28, 2026

ntkathole commented on Jun 29, 2026

falloficaruss commented on Jun 29, 2026

ntkathole commented on Jun 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions