FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat: Add Chronon online and offline store integrations by franciscojavierarceo · Pull Request #6188 · feast-dev/feast · GitHub

Repository navigation

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension .md  (11) .py  (21) .sh  (2) .yml  (1) No extension  (1) All 5 file types selected
Only manifest files
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
134 changes: 134 additions & 0 deletions .github/workflows/pr_chronon_integration_tests.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
name: pr-chronon-integration-tests

on:
pull_request:
types:
- opened
- synchronize
- labeled
paths:
- "sdk/python/feast/infra/online_stores/chronon_online_store/**"
- "sdk/python/feast/infra/offline_stores/contrib/chronon_offline_store/**"
- "sdk/python/feast/infra/chronon_provider.py"
- "sdk/python/tests/unit/infra/online_stores/chronon_online_store/**"
- "sdk/python/tests/unit/infra/offline_stores/contrib/chronon_offline_store/**"
- "sdk/python/tests/unit/examples/test_chronon_demo.py"
- "sdk/python/tests/unit/infra/scripts/test_chronon_local_service_script.py"
- "examples/chronon/**"
- "sdk/python/tests/integration/online_store/test_chronon_online_store.py"
- "sdk/python/tests/integration/online_store/test_chronon_online_store_real_service.py"
- "sdk/python/tests/integration/offline_store/test_chronon_offline_store.py"
- "infra/scripts/chronon/**"
- "chronon/**"
- ".github/workflows/pr_chronon_integration_tests.yml"

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
chronon-python-tests:
if:
((github.event.action == 'labeled' && (github.event.label.name == 'approved' || github.event.label.name == 'lgtm' || github.event.label.name == 'ok-to-test')) ||
(github.event.action != 'labeled' && (contains(github.event.pull_request.labels.*.name, 'ok-to-test') || contains(github.event.pull_request.labels.*.name, 'approved') || contains(github.event.pull_request.labels.*.name, 'lgtm')))) &&
github.event.pull_request.base.repo.full_name == 'feast-dev/feast'
# The Chronon quickstart image is published for ARM64 only.
runs-on: ubuntu-24.04-arm
steps:
- uses: actions/checkout@v4
with:
repository: ${{ github.event.repository.full_name }}
ref: ${{ github.ref }}
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
architecture: arm64
- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
- name: Setup Java
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "8"
- name: Setup sbt
uses: sbt/setup-sbt@v1
- name: Install dependencies
run: make install-python-dependencies-ci
- name: Run Chronon unit tests
run: |
uv run pytest -c sdk/python/pytest.ini \
sdk/python/tests/unit/infra/online_stores/chronon_online_store \
sdk/python/tests/unit/infra/offline_stores/contrib/chronon_offline_store \
sdk/python/tests/unit/examples/test_chronon_demo.py \
sdk/python/tests/unit/infra/scripts/test_chronon_local_service_script.py
- name: Run Chronon stub integration tests
run: |
uv run pytest -c sdk/python/pytest.ini \
sdk/python/tests/integration/online_store/test_chronon_online_store.py \
sdk/python/tests/integration/offline_store/test_chronon_offline_store.py \
--integration
- name: Checkout Chronon
env:
CHRONON_REPO: ${{ github.workspace }}/.chronon
CHRONON_REPO_URL: https://github.com/airbnb/chronon.git
CHRONON_REF: 6c0b8de9f0301521baf61a46ff3083c566fb4052 # pragma: allowlist secret
run: |
git clone "${CHRONON_REPO_URL}" "${CHRONON_REPO}"
git -C "${CHRONON_REPO}" checkout "${CHRONON_REF}"
- name: Install Chronon build dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
apt-transport-https \
autoconf \
automake \
bison \
build-essential \
ca-certificates \
curl \
flex \
g++ \
gnupg \
libtool \
pkg-config \
python3-pip \
python3-venv
curl -fsSL --retry 3 --connect-timeout 15 \
"https://github.com/apache/thrift/archive/refs/tags/v0.13.0.tar.gz" \
-o /tmp/thrift.tar.gz
sudo rm -rf /usr/src/thrift
sudo mkdir -p /usr/src/thrift
sudo tar zxf /tmp/thrift.tar.gz -C /usr/src/thrift --strip-components=1
cd /usr/src/thrift
sudo ./bootstrap.sh
sudo ./configure --disable-libs --disable-tests --disable-tutorial
sudo make -j2
sudo make install
python3 -m pip install --break-system-packages build
thrift -version
- name: Build Chronon quickstart and service jars
env:
CHRONON_REPO: ${{ github.workspace }}/.chronon
run: |
cd "${CHRONON_REPO}/quickstart/mongo-online-impl"
sbt assembly
cd "${CHRONON_REPO}"
sbt "project service" assembly
- name: Start live Chronon service
env:
CHRONON_REPO: ${{ github.workspace }}/.chronon
run: infra/scripts/chronon/start-local-chronon-service.sh
- name: Run Chronon live-service integration test
env:
CHRONON_SERVICE_URL: http://127.0.0.1:9000
run: |
uv run pytest -c sdk/python/pytest.ini \
sdk/python/tests/integration/online_store/test_chronon_online_store_real_service.py \
--integration
- name: Stop live Chronon service
if: always()
run: infra/scripts/chronon/stop-local-chronon-service.sh
1 change: 1 addition & 0 deletions Makefile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ lock-python-dependencies-all: ## Recompile and lock all Python dependency sets f
"uv pip compile -p $(ver) --universal --no-strip-extras pyproject.toml --extra ci \
--torch-backend cpu \
--generate-hashes --output-file sdk/python/requirements/py$(ver)-ci-requirements.txt" && \
python3 infra/scripts/add_cpu_torch_hashes.py --python-version $(ver) sdk/python/requirements/py$(ver)-ci-requirements.txt && \
pixi run --environment $(call get_env_name,$(ver)) --manifest-path infra/scripts/pixi/pixi.toml \
"uv pip compile -p $(ver) --no-strip-extras pyproject.toml \
--generate-hashes --output-file sdk/python/requirements/py$(ver)-requirements.txt" && \
Expand Down
3 changes: 3 additions & 0 deletions README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] Vector Search (Alpha release. See [RFC](https://docs.google.com/document/d/18IWzLEA9i2lDWnbfbwXnMCg3StlqaLVI-uRpQjr_Vos/edit#heading=h.9gaqqtox9jg6))
* [ ] [Enhanced Feature Server and SDK for native support for NLP](https://github.com/feast-dev/feast/issues/4964)
* **Data Sources**
* [x] [Chronon source (contrib)](https://docs.feast.dev/reference/data-sources/chronon)
* [x] [Snowflake source](https://docs.feast.dev/reference/data-sources/snowflake)
* [x] [Redshift source](https://docs.feast.dev/reference/data-sources/redshift)
* [x] [BigQuery source](https://docs.feast.dev/reference/data-sources/bigquery)
Expand All @@ -189,6 +190,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Ray source (contrib plugin)](https://docs.feast.dev/reference/data-sources/ray)
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
* **Offline Stores**
* [x] [Chronon (contrib)](https://docs.feast.dev/reference/offline-stores/chronon)
* [x] [Snowflake](https://docs.feast.dev/reference/offline-stores/snowflake)
* [x] [Redshift](https://docs.feast.dev/reference/offline-stores/redshift)
* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
Expand All @@ -209,6 +211,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Hybrid](https://docs.feast.dev/reference/offline-stores/hybrid)
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/customizing-feast/adding-a-new-offline-store)
* **Online Stores**
* [x] [Chronon](https://docs.feast.dev/reference/online-stores/chronon)
* [x] [Snowflake](https://docs.feast.dev/reference/online-stores/snowflake)
* [x] [DynamoDB](https://docs.feast.dev/reference/online-stores/dynamodb)
* [x] [Redis](https://docs.feast.dev/reference/online-stores/redis)
Expand Down
3 changes: 3 additions & 0 deletions docs/SUMMARY.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@
* [Clickhouse (contrib)](reference/data-sources/clickhouse.md)
* [Ray (contrib)](reference/data-sources/ray.md)
* [MongoDB (contrib)](reference/data-sources/mongodb.md)
* [Chronon (contrib)](reference/data-sources/chronon.md)
* [Offline stores](reference/offline-stores/README.md)
* [Overview](reference/offline-stores/overview.md)
* [Dask](reference/offline-stores/dask.md)
Expand All @@ -143,6 +144,7 @@
* [Oracle (contrib)](reference/offline-stores/oracle.md)
* [Athena (contrib)](reference/offline-stores/athena.md)
* [MongoDB (contrib)](reference/offline-stores/mongodb.md)
* [Chronon (contrib)](reference/offline-stores/chronon.md)
* [Remote Offline](reference/offline-stores/remote-offline-store.md)
* [Hybrid](reference/offline-stores/hybrid.md)
* [Online stores](reference/online-stores/README.md)
Expand All @@ -166,6 +168,7 @@
* [SingleStore](reference/online-stores/singlestore.md)
* [Milvus](reference/online-stores/milvus.md)
* [MongoDB](reference/online-stores/mongodb.md)
* [Chronon (contrib)](reference/online-stores/chronon.md)
* [Aerospike](reference/online-stores/aerospike.md)
* [Elasticsearch](reference/online-stores/elasticsearch.md)
* [Qdrant](reference/online-stores/qdrant.md)
Expand Down
57 changes: 57 additions & 0 deletions docs/reference/data-sources/chronon.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Chronon source (contrib)

## Description

Chronon sources describe feature data produced by [Chronon](https://chronon.ai/) and consumed by Feast.
They point Feast at Chronon's offline materialization output and, when online reads are needed, identify the Chronon Join or GroupBy that should be queried from Chronon's online service.

Feast does not compute or materialize Chronon features. Chronon owns feature computation, backfills, consistency, and online serving. Feast uses the source metadata for registry, discovery, historical retrieval, and online lookup.

## Examples

Defining a Chronon source for a Chronon Join:

```python
from feast.infra.offline_stores.contrib.chronon_offline_store.chronon_source import (
ChrononSource,
)

driver_stats_source = ChrononSource(
name="driver_stats",
materialization_path="data/chronon/driver_stats",
chronon_join="team/driver_stats.v1",
online_endpoint="http://localhost:8080",
timestamp_field="event_timestamp",
created_timestamp_column="created_timestamp",
)
```

Defining a Chronon source for a Chronon GroupBy:

```python
driver_profile_source = ChrononSource(
name="driver_profile",
materialization_path="data/chronon/driver_profile",
chronon_group_by="team/driver_profile.v1",
timestamp_field="event_timestamp",
)
```

## Configuration reference

| Parameter | Required | Description |
| :------------------------- | :------- | :---------- |
| `materialization_path` | yes | Local or repository-relative path to Chronon's Parquet materialization output. |
| `chronon_join` | no | Chronon Join name used for online reads, for example `team/training_set.v1`. |
| `chronon_group_by` | no | Chronon GroupBy name used for online reads, for example `team/user_features.v1`. |
| `online_endpoint` | no | Chronon online service base URL for this source. If omitted, Feast uses the Chronon online store `path`. |
| `timestamp_field` | yes | Event timestamp column in the materialized Chronon data. |
| `created_timestamp_column` | no | Optional created timestamp column used to select the latest row when duplicate event timestamps exist. |
| `field_mapping` | no | Standard Feast field mapping applied before retrieval. |

Set at most one of `chronon_join` and `chronon_group_by`. Offline-only sources may omit both, but `online_endpoint` requires one of them so Feast can build the Chronon request URL.

## Supported Types

Chronon sources read Parquet data through PyArrow and use Feast's standard PyArrow type mapping.
For a comparison against other batch data sources, please see [here](overview.md#functionality-matrix).
4 changes: 4 additions & 0 deletions docs/reference/offline-stores/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ Please see [Offline Store](../../getting-started/components/offline-store.md) fo
[overview.md](overview.md)
{% endcontent-ref %}

{% content-ref url="chronon.md" %}
[chronon.md](chronon.md)
{% endcontent-ref %}

{% content-ref url="dask.md" %}
[dask.md](dask.md)
{% endcontent-ref %}
Expand Down
121 changes: 121 additions & 0 deletions docs/reference/offline-stores/chronon.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Chronon offline store (contrib)

## Description

The Chronon offline store provides support for reading [ChrononSources](../data-sources/chronon.md) from Chronon's Parquet materialization output.

Chronon remains the system of record for feature computation and materialization. Feast reads Chronon-produced data for historical retrieval, feature reuse, and registry-driven training workflows.

## Getting started

Chronon-backed feature repos use the Chronon offline store with Chronon sources:

{% code title="feature_store.yaml" %}
```yaml
project: my_project
registry: data/registry.db
provider: chronon
offline_store:
type: chronon
online_store:
type: chronon
path: http://localhost:8080
```
{% endcode %}

Example feature view:

```python
from feast import Entity, FeatureView, Field
from feast.infra.offline_stores.contrib.chronon_offline_store.chronon_source import (
ChrononSource,
)
from feast.types import Float32

driver = Entity(name="driver", join_keys=["driver_id"])

driver_stats = FeatureView(
name="driver_stats",
entities=[driver],
schema=[Field(name="rating", dtype=Float32)],
source=ChrononSource(
name="driver_stats_source",
materialization_path="data/chronon/driver_stats",
chronon_join="team/driver_stats.v1",
timestamp_field="event_timestamp",
created_timestamp_column="created_timestamp",
),
)
```

## Historical retrieval

`get_historical_features` performs point-in-time joins against Chronon's materialized Parquet data. For each entity row, Feast selects the latest Chronon row with an event timestamp at or before the entity timestamp. If `created_timestamp_column` is configured and duplicate event timestamps exist, the latest created row wins.

Reads select only the required feature, entity-key, event-time, and (when needed) created-time columns from Parquet, translating field mappings to physical column names before loading. Joins still execute locally in pandas; column selection reduces I/O and memory use but does not provide distributed execution.

Python on-demand feature views can transform the retrieved features locally through Feast's standard retrieval job. Put required request-time inputs in the pandas `entity_df`. `to_df()`, `to_arrow()`, and saved datasets include the requested transformed outputs. Time-range retrieval without an `entity_df` cannot supply request-time inputs.

This is intended for training and validation workflows that want Feast's registry and retrieval APIs while using Chronon as the feature computation engine.

## Save retrieval results

Use `provider: chronon` and `SavedDatasetFileStorage` to persist a retrieval result as Parquet and load it again through Feast:

```python
from feast.infra.offline_stores.file_source import SavedDatasetFileStorage

job = store.get_historical_features(
entity_df=entity_df,
features=["driver_stats:rating"],
)
training = store.create_saved_dataset(
from_=job,
name="training_v1",
storage=SavedDatasetFileStorage(path="data/training_v1.parquet"),
)
training_df = store.get_saved_dataset("training_v1").to_df()
```

Relative paths resolve against the feature repository. File storage can also use a Parquet directory or supported PyArrow filesystem URI, including S3 with the existing file-storage credentials/endpoint configuration. Existing destinations are rejected unless `allow_overwrite=True`; overwriting a directory replaces its contents. Use a dedicated output location. Writes are synchronous and are not an atomic publication mechanism.

The saved result retains request columns, custom entity timestamp names, full feature names, and already-computed on-demand features. This saves training results; it does not write feature values into Chronon's computation or online storage.

## Configuration reference

| Parameter | Required | Default | Description |
| :-------- | :------- | :------ | :---------- |
| `type` | yes | - | Must be set to `chronon`. |
| `path` | no | - | Reserved for future offline store configuration. Source-level `materialization_path` controls where data is read from. |

## Functionality Matrix

The set of functionality supported by offline stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Chronon offline store.

| | Chronon |
| :----------------------------------------------------------------- | :------ |
| `get_historical_features` (point-in-time correct join) | yes |
| `pull_latest_from_table_or_query` (retrieve latest feature values) | yes |
| `pull_all_from_table_or_query` (retrieve a saved dataset) | yes |
| `offline_write_batch` (persist dataframes to offline store) | no |
| `write_logged_features` (persist logged features to offline store) | no |

Below is a matrix indicating which functionality is supported by `ChrononRetrievalJob`.

| | Chronon |
| ----------------------------------------------------- | ------- |
| export to dataframe | yes |
| export to arrow table | yes |
| export to arrow batches | no |
| export to SQL | no |
| export to data lake (S3, GCS, etc.) | no |
| export to data warehouse | no |
| export as Spark dataframe | no |
| local execution of Python-based on-demand transforms | yes |
| remote execution of Python-based on-demand transforms | no |
| persist results in the offline store | yes (Parquet saved datasets) |
| preview the query plan before execution | no |
| read partitioned data | yes |

To compare this set of functionality against other offline stores, please see the full [functionality matrix](overview.md#functionality-matrix).
Loading
Loading

Back | FazBrowse Home | New Git URL