FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

fix: Use UTC end bound and correct docs for disable_event_timestamp by patelchaitany · Pull Request #6956 · feast-dev/feast · GitHub

Repository navigation

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension .jinja2  (1) .md  (7) .py  (5) All 3 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
2 changes: 1 addition & 1 deletion README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ feast materialize 2021-04-12T00:00:00 $CURRENT_TIME
feast materialize --disable-event-timestamp
```

The `--disable-event-timestamp` flag allows you to materialize all available feature data using the current datetime as the event timestamp, without needing to specify start and end timestamps. This is useful when your source data lacks proper event timestamp columns.
The `--disable-event-timestamp` flag allows you to materialize all available data (from 1970-01-01 up to the current UTC time) without specifying start and end timestamps. Rows keep the event timestamps from the source.

```commandline
Materializing feature view driver_hourly_stats from 2021-04-14 to 2021-04-15 done!
Expand Down
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Online store

Feast uses online stores to serve features at low latency.
Feature values are loaded from data sources into the online store through _materialization_, which can be triggered through the `materialize` command (either with specific timestamps or using `--disable-event-timestamp` to materialize all data with current timestamps).
Feature values are loaded from data sources into the online store through _materialization_, which can be triggered through the `materialize` command (either with specific timestamps or using `--disable-event-timestamp` to materialize all available data).

The storage schema of features within the online store mirrors that of the original data source.
One key difference is that for each [entity key](../concepts/entity.md), only the latest feature values are stored.
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/components/overview.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
* **Create Batch Features:** ELT/ETL systems like Spark and SQL are used to transform data in the batch store.
* **Create Stream Features:** Stream features are created from streaming services such as Kafka or Kinesis, and can be pushed directly into Feast via the [Push API](../../reference/data-sources/push.md).
* **Feast Apply:** The user (or CI) publishes versioned controlled feature definitions using `feast apply`. This CLI command updates infrastructure and persists definitions in the object store registry.
* **Feast Materialize:** The user (or scheduler) executes `feast materialize` (with timestamps or `--disable-event-timestamp` to materialize all data with current timestamps) which loads features from the offline store into the online store.
* **Feast Materialize:** The user (or scheduler) executes `feast materialize` (with timestamps or `--disable-event-timestamp` to materialize all available data) which loads features from the offline store into the online store.
* **Model Training:** A model training pipeline is launched. It uses the Feast Python SDK to retrieve a training dataset that can be used for training models.
* **Get Historical Features:** Feast exports a point-in-time correct training dataset based on the list of features and entity dataframe provided by the model training pipeline.
* **Deploy Model:** The trained model binary (and list of features) are deployed into a model serving system. This step is not executed by Feast.
Expand Down
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
feast materialize-incremental $CURRENT_TIME
```

**Simple materialization (for data without event timestamps):**
**Simple materialization (all available data, no start/end timestamps):**
```bash
feast materialize --disable-event-timestamp
```
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/quickstart.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -513,7 +513,7 @@ feast materialize-incremental $CURRENT_TIME
{% endtab %}
{% tab title="Bash (simple)" %}
```bash
# Alternative: Materialize all data using current timestamp (for data without event timestamps)
# Alternative: Materialize all available data without specifying start/end timestamps
feast materialize --disable-event-timestamp
```
{% endtab %}
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/feast-cli-commands.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ Load data from feature views into the online store.
feast materialize 2020-01-01T00:00:00 2022-01-01T00:00:00
```

**Without timestamps (uses current datetime):**
**Without timestamps (materializes all available data):**
```bash
feast materialize --disable-event-timestamp
```
Expand All @@ -287,7 +287,7 @@ feast materialize -v driver_hourly_stats 2020-01-01T00:00:00 2022-01-01T00:00:00
feast materialize --disable-event-timestamp -v driver_hourly_stats
```

The `--disable-event-timestamp` flag is useful when your source data lacks event timestamp columns, allowing you to materialize all available data using the current datetime as the event timestamp.
The `--disable-event-timestamp` flag lets you materialize all available data (from 1970-01-01 up to the current UTC time) without specifying start and end timestamps. Rows keep the event timestamps from the source.

```text
Materializing 1 feature views from 2020-01-01 to 2022-01-01
Expand Down
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,7 @@ curl -X POST "http://localhost:6566/materialize" -d '{
}' | jq
```

When `disable_event_timestamp` is set to `true`, the `start_ts` and `end_ts` parameters are not required, and all available data is materialized using the current datetime as the event timestamp. This is useful when your source data lacks proper event timestamp columns.
When `disable_event_timestamp` is set to `true`, the `start_ts` and `end_ts` parameters are not required, and all available data (from 1970-01-01 up to the current UTC time) is materialized. Rows keep the event timestamps from the source.

Or from Python:
```python
Expand Down
2 changes: 1 addition & 1 deletion infra/templates/README.md.jinja2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ feast materialize 2021-04-12T00:00:00 $CURRENT_TIME
feast materialize --disable-event-timestamp
```

The `--disable-event-timestamp` flag allows you to materialize all available feature data using the current datetime as the event timestamp, without needing to specify start and end timestamps. This is useful when your source data lacks proper event timestamp columns.
The `--disable-event-timestamp` flag allows you to materialize all available data (from 1970-01-01 up to the current UTC time) without specifying start and end timestamps. Rows keep the event timestamps from the source.

```commandline
Materializing feature view driver_hourly_stats from 2021-04-14 to 2021-04-15 done!
Expand Down
15 changes: 6 additions & 9 deletions sdk/python/feast/cli/cli.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# limitations under the License.
import json
import logging
from datetime import datetime
from datetime import datetime, timezone
from importlib.metadata import version as importlib_version
from pathlib import Path
from typing import List, Optional
Expand Down Expand Up @@ -363,7 +363,7 @@ def registry_dump_command(ctx: click.Context):
@click.option(
"--disable-event-timestamp",
is_flag=True,
help="Materialize all available data using current datetime as event timestamp (useful when source data lacks event timestamps)",
help="Materialize all available data, from 1970-01-01 up to the current UTC time, without specifying START_TS and END_TS (rows keep their source event timestamps)",
)
@click.option(
"--version",
Expand All @@ -388,7 +388,7 @@ def materialize_command(

START_TS and END_TS should be in ISO 8601 format, e.g. '2021-07-16T19:20:01'

If --disable-event-timestamp is used, timestamps are not required and all available data will be materialized using the current datetime as the event timestamp.
If --disable-event-timestamp is used, timestamps are not required and all available data (from 1970-01-01 up to the current UTC time) will be materialized. Rows keep their source event timestamps.
"""
store = create_feature_store(ctx)

Expand All @@ -397,12 +397,9 @@ def materialize_command(
raise click.UsageError(
"Cannot specify START_TS or END_TS when --disable-event-timestamp is used"
)
now = datetime.now()
# Query all available data and use current datetime as event timestamp
start_date = datetime(
1970, 1, 1
) # Beginning of time to capture all historical data
end_date = now
# Query all available data, from the beginning of time up to now (UTC)
start_date = datetime(1970, 1, 1, tzinfo=timezone.utc)
end_date = utils._utc_now()
else:
if not start_ts or not end_ts:
raise click.UsageError(
Expand Down
5 changes: 2 additions & 3 deletions sdk/python/feast/feature_server.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor
from contextlib import asynccontextmanager
from datetime import datetime
from datetime import datetime, timezone
from importlib import resources as importlib_resources
from types import SimpleNamespace
from typing import Any, DefaultDict, Dict, List, NamedTuple, Optional, Set, Union
Expand Down Expand Up @@ -478,8 +478,7 @@ def _parse_materialize_timestamps(
) -> tuple:
"""Parse and validate start/end timestamps from a MaterializeRequest."""
if request.disable_event_timestamp:
now = datetime.now()
return datetime(1970, 1, 1), now
return datetime(1970, 1, 1, tzinfo=timezone.utc), utils._utc_now()

if not request.start_ts or not request.end_ts:
raise ValueError(
Expand Down
2 changes: 1 addition & 1 deletion sdk/python/feast/feature_store.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -2855,7 +2855,7 @@ def materialize(
end_date (datetime): End date for time range of data to materialize into the online store
feature_views (List[str]): Optional list of feature view names. If selected, will only run
materialization for the specified feature views.
disable_event_timestamp (bool): If True, materializes all available data using current datetime as event timestamp instead of source event timestamps
disable_event_timestamp (bool): If True, materializes all available data (from 1970-01-01 up to the current UTC time). Rows keep their source event timestamps.
full_feature_names (bool): If True, feature names will be prefixed with the corresponding
feature view name.
version (str): Optional version to materialize (e.g., 'v2'). Requires feature_views
Expand Down
2 changes: 1 addition & 1 deletion sdk/python/feast/infra/provider.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@ def materialize_single_feature_view(
registry: The registry for the current feature store.
project: Feast project to which the objects belong.
tqdm_builder: A function to monitor the progress of materialization.
disable_event_timestamp: If True, materializes all available data using current datetime as event timestamp instead of source event timestamps.
disable_event_timestamp: If True, materializes all available data (from 1970-01-01 up to the current UTC time). Rows keep their source event timestamps.
"""
pass

Expand Down
16 changes: 16 additions & 0 deletions sdk/python/tests/unit/test_feature_server.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,22 @@ def test_materialize_request_model():
assert req2.end_ts == "2021-01-02T00:00:00"


def test_parse_materialize_timestamps_disable_event_timestamp_uses_utc():
"""The full-window bounds must be tz-aware UTC, not naive local time."""
from datetime import datetime, timedelta, timezone

from feast.feature_server import MaterializeRequest, _parse_materialize_timestamps

req = MaterializeRequest(feature_views=["test"], disable_event_timestamp=True)
before = datetime.now(timezone.utc)
start_date, end_date = _parse_materialize_timestamps(req)
after = datetime.now(timezone.utc)

assert start_date == datetime(1970, 1, 1, tzinfo=timezone.utc)
assert end_date.utcoffset() == timedelta(0)
assert before <= end_date <= after


def _enable_offline_batching_config(
fs, enabled: bool = True, batch_size: int = 1, batch_interval_seconds: int = 60
):
Expand Down
Loading

Back | FazBrowse Home | New Git URL