feat(server): Remote Materialization - #6649
Conversation
Sorry, something went wrong.
When ?async=true is passed to /materialize and /materialize-incremental, the endpoint fires off materialization in a background thread and returns 202 Accepted immediately. Concurrent requests for an already-MATERIALIZING FV are rejected with 409. Without ?async=true, behavior is unchanged. Client-side additions: - remote=True param on store.materialize() / materialize_incremental() to delegate to the feature server (URL/TLS from online_store config) - wait=False support with store.poll_materialization() for status polling - FeatureView state set to MATERIALIZING before 202, reset on failure Server-side additions: - ?async=true on existing /materialize and /materialize-incremental - ?force=true to override stuck MATERIALIZING state - Four module-level helpers for testability: _authorize_materialize_views, _check_already_materializing, _update_fv_state, _parse_materialize_timestamps Registry fixes: - SQL and Snowflake registries now set FV state to AVAILABLE_ONLINE in apply_materialization() (parity with file-based registry) Addresses feast-dev#4526 Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
When ?async=true is passed to /materialize and /materialize-incremental, the endpoint fires off materialization in a background thread and returns 202 Accepted immediately. Concurrent requests for an already-MATERIALIZING FV are rejected with 409. Without ?async=true, behavior is unchanged. Client-side additions: - remote=True param on store.materialize() / materialize_incremental() to delegate to the feature server (URL/TLS from online_store config) - wait=False support with store.poll_materialization() for status polling - FeatureView state set to MATERIALIZING before 202, reset on failure Server-side additions: - ?async=true on existing /materialize and /materialize-incremental - ?force=true to override stuck MATERIALIZING state - Four module-level helpers for testability: _authorize_materialize_views, _check_already_materializing, _update_fv_state, _parse_materialize_timestamps Registry fixes: - SQL and Snowflake registries now set FV state to AVAILABLE_ONLINE in apply_materialization() (parity with file-based registry) Addresses feast-dev#4526 Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
|
⚠️ Please install the Codecov Report❌ Patch coverage is 33.87097% with 123 lines in your changes missing coverage. Please review.
@@ Coverage Diff @@
## master #6649 +/- ##
==========================================
- Coverage 46.77% 46.75% -0.02%
==========================================
Files 414 414
Lines 50191 50348 +157
Branches 7181 7208 +27
==========================================
+ Hits 23475 23539 +64
- Misses 25077 25157 +80
- Partials 1639 1652 +13
Continue to review full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Sorry, something went wrong.
…ore is remote When online_store.type == remote, materialize() and materialize_incremental() POST to the feature server with ?async=true (fire-and-forget) instead of running the engine locally. Optional force=True maps to ?force=true for stuck MATERIALIZING recovery. URL/TLS/auth come from online_store config. Shared _delegate_remote_materialize() builds query params and posts; RemoteComputeEngine remains a follow-up for provider-layer remoting. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Remove server-side MATERIALIZING pre-set before 202 which conflicted with store.materialize() state machine (MATERIALIZING → MATERIALIZING rejected). Async now accepts and runs materialize in the background; store owns transitions. ?force=true resets stuck MATERIALIZING FVs to GENERATED so normal materialize can proceed. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
| force: bool = False, | ||
| ) -> None: | ||
| """Fire-and-forget POST to feature server with ?async=true.""" | ||
| query_params = {"async": "true"} |
There was a problem hiding this comment.
may be allow user to pass param, There's no way for an SDK user to do synchronous materialization now.
Sorry, something went wrong.
There was a problem hiding this comment.
Agreed — added run_async: bool = True on materialize() / materialize_incremental().
- Default run_async=True: remote topology POSTs with ?async=true, returns after 202
- run_async=False: POSTs without async and blocks until the server finishes sync materialization
- force=True still requires run_async=True (force only applies on the async path)
Sorry, something went wrong.
franciscojavierarceo
left a comment
There was a problem hiding this comment.
I found two blocking correctness issues:
- The async concurrency guard is check-then-act. _check_already_materializing() reads registry state, but MATERIALIZING is not written until the executor task starts. Two concurrent async=true requests can therefore both pass the check and enqueue duplicate materializations. Please atomically reserve the feature views before returning 202 (and roll the reservation back if submission fails), or use a registry compare-and-set/lock.
- MaterializeIncrementalRequest.version is accepted and sent by the SDK, but both server paths call store.materialize_incremental() without version=request.version; authorization resolution also ignores it. A versioned request can therefore materialize a different feature-view version than requested. Please thread version through both sync and async paths and cover it in the server tests.
The existing review thread about the SDK no longer exposing synchronous mode also remains unresolved.
Sorry, something went wrong.
jyejare
left a comment
There was a problem hiding this comment.
This PR implements remote materialization support, allowing clients with a remote online store topology to delegate materialization requests to a feature server via async HTTP endpoints. The implementation adds proper conflict detection, state management, and force override capabilities.
Sorry, something went wrong.
|
|
||
| Rolls back all already-transitioned FVs if this one can't transition. | ||
| """ | ||
| previous_state = getattr(feature_view, "state", None) | ||
| previous_states[feature_view.name] = getattr(feature_view, "state", None) |
There was a problem hiding this comment.
[Critical] Race condition in state tracking initialization
The line previous_states[feature_view.name] = getattr(feature_view, "state", None) was moved before the state transition logic, but this creates a critical bug. If the transition fails and we need to rollback other FVs, the previous_states dict won't have entries for FVs that failed before this one was reached.
Suggested:
| Rolls back all already-transitioned FVs if this one can't transition. | |
| """ | |
| previous_state = getattr(feature_view, "state", None) | |
| previous_states[feature_view.name] = getattr(feature_view, "state", None) | |
| previous_state = getattr(feature_view, "state", None) | |
| if ( | |
| hasattr(feature_view, "state") | |
| and feature_view.state != FeatureViewState.STATE_UNSPECIFIED | |
| ): | |
| # ... validation logic ... | |
| feature_view.state = FeatureViewState.MATERIALIZING | |
| self.registry.apply_feature_view(feature_view, self.project, commit=True) | |
| previous_states[feature_view.name] = previous_state |
Sorry, something went wrong.
There was a problem hiding this comment.
Thanks for the careful look — walked through both placements with the caller loop in mind; they have the same observable outcome (including failure cases).
Caller (simplified):
for feature_view, fv_start in fv_with_dates:
self._transition_fv_to_materializing(feature_view, regular_fvs, previous_states)
regular_fvs.append(feature_view) # only after a successful returnOn a failed transition, the failing FV is not in already_transitioned / regular_fvs. Rollback only restores FVs that already succeeded.
Sorry, something went wrong.
| # by store.materialize(); server only accepts and runs in background. | ||
| def _run_materialize(): | ||
| try: | ||
| store.materialize( | ||
| start_date, | ||
| end_date, | ||
| fv_names, | ||
| disable_event_timestamp=request.disable_event_timestamp, | ||
| full_feature_names=request.full_feature_names, | ||
| version=request.version, | ||
| ) | ||
| except Exception as e: | ||
| logger.error( | ||
| f"Async materialization failed for {fv_names}: {e}", |
There was a problem hiding this comment.
[Critical] Background task exception handling insufficient
The async materialization runs in a background thread but doesn't handle critical failures properly. If materialization fails, only the FV state is reset to GENERATED, but there's no way for the client to know the operation failed since the API already returned 202. This could lead to silent failures in production.
Suggested:
| # by store.materialize(); server only accepts and runs in background. | |
| def _run_materialize(): | |
| try: | |
| store.materialize( | |
| start_date, | |
| end_date, | |
| fv_names, | |
| disable_event_timestamp=request.disable_event_timestamp, | |
| full_feature_names=request.full_feature_names, | |
| version=request.version, | |
| ) | |
| except Exception as e: | |
| logger.error( | |
| f"Async materialization failed for {fv_names}: {e}", | |
| except Exception as e: | |
| logger.error( | |
| f"Async materialization failed for {fv_names}: {e}", | |
| exc_info=True, | |
| ) | |
| _update_fv_state(store, fv_names, FeatureViewState.GENERATED) | |
| # TODO: Consider implementing a status endpoint or webhook callback | |
| # for clients to check materialization status |
Sorry, something went wrong.
There was a problem hiding this comment.
Thanks — agreed that fire-and-forget 202 means clients don’t get the failure on the HTTP response. That’s intentional for this PR: async acceptance + observe completion via FeatureView.state / materialization_intervals (and run_async=False when the client needs the sync HTTP path to block/error).
A dedicated status endpoint / job handle / webhook is out of scope here and planned as a follow-up (along the RemoteComputeEngine / job-tracking direction discussed earlier). I’ve added a TODO in the failure path comment pointing to that follow-up.
Sorry, something went wrong.
| content={ | ||
| "error": ( | ||
| f"Cannot start async materialization — the following feature " | ||
| f"views are already in MATERIALIZING state: {conflicting}. " | ||
| f"Use ?force=true to override." | ||
| ), | ||
| "feature_views": conflicting, | ||
| }, | ||
| ) | ||
| return None | ||
|
|
||
|
|
||
| def _update_fv_state( |
There was a problem hiding this comment.
[Warning] Silent exception handling could hide real errors
The _reset_stuck_materializing_to_generated and _check_already_materializing functions silently ignore all exceptions when accessing feature views. This could hide legitimate errors like registry corruption or network issues, making debugging difficult.
Suggested:
| content={ | |
| "error": ( | |
| f"Cannot start async materialization — the following feature " | |
| f"views are already in MATERIALIZING state: {conflicting}. " | |
| f"Use ?force=true to override." | |
| ), | |
| "feature_views": conflicting, | |
| }, | |
| ) | |
| return None | |
| def _update_fv_state( | |
| except (FeatureViewNotFoundException, KeyError): | |
| # Expected when FV doesn't exist | |
| pass | |
| except Exception as e: | |
| logger.warning(f"Unexpected error checking state for {fv_name}: {e}") | |
| pass |
Sorry, something went wrong.
There was a problem hiding this comment.
Accepted
Sorry, something went wrong.
| or normalize_version_string(self.version) | ||
| != normalize_version_string(other.version) | ||
| or self.org != other.org | ||
| or self.state != other.state |
There was a problem hiding this comment.
[Suggestion] State comparison added to eq affects hash consistency
Adding state to the equality comparison means two otherwise identical FeatureViews will be considered different if they're in different states (e.g., GENERATED vs MATERIALIZING). This could break set operations, dict lookups, and caching logic that expects state changes to not affect object identity.
Suggested:
| or self.state != other.state | |
| # Consider if state should be included in equality. If FeatureViews should be | |
| # considered equal regardless of materialization state, remove this line. | |
| # If state is semantically important for equality, ensure __hash__ is also updated | |
| # or make the class unhashable. |
Sorry, something went wrong.
There was a problem hiding this comment.
Agreed
Sorry, something went wrong.
| feature_views: Optional[List[str]] = None | ||
| disable_event_timestamp: bool = False | ||
| full_feature_names: bool = False | ||
| version: Optional[str] = None | ||
|
|
||
|
|
||
| class MaterializeIncrementalRequest(BaseModel): | ||
| end_ts: str | ||
| feature_views: Optional[List[str]] = None | ||
| full_feature_names: bool = False | ||
| version: Optional[str] = None | ||
|
|
||
|
|
||
| class GetOnlineFeaturesRequest(BaseModel): |
There was a problem hiding this comment.
[Suggestion] Missing version parameter documentation and validation
The version parameter was added to both MaterializeRequest and MaterializeIncrementalRequest but there's no documentation about what values are valid or how it affects the materialization behavior.
Suggested:
| feature_views: Optional[List[str]] = None | |
| disable_event_timestamp: bool = False | |
| full_feature_names: bool = False | |
| version: Optional[str] = None | |
| class MaterializeIncrementalRequest(BaseModel): | |
| end_ts: str | |
| feature_views: Optional[List[str]] = None | |
| full_feature_names: bool = False | |
| version: Optional[str] = None | |
| class GetOnlineFeaturesRequest(BaseModel): | |
| version: Optional[str] = Field( | |
| None, | |
| description="Optional version to materialize (e.g., 'v2'). Requires feature_views with exactly one entry." | |
| ) |
Sorry, something went wrong.
There was a problem hiding this comment.
Done
Sorry, something went wrong.
| @@ -839,27 +971,49 @@ async def materialize(request: MaterializeRequest) -> None: | |||
| ) | |||
|
|
|||
There was a problem hiding this comment.
[Nitpick] Inconsistent parameter passing in sync mode
In the synchronous code path for materialize, the version parameter is passed but the feature_views parameter uses the original request.feature_views instead of the resolved fv_names list.
Suggested:
| await run_in_threadpool( | |
| store.materialize, | |
| start_date, | |
| end_date, | |
| fv_names, # Use resolved names for consistency | |
| disable_event_timestamp=request.disable_event_timestamp, | |
| full_feature_names=request.full_feature_names, | |
| version=request.version, # Don't forget version parameter | |
| ) |
Sorry, something went wrong.
There was a problem hiding this comment.
sync path now passes resolved fv_names and version=request.version (aligned with async).
Sorry, something went wrong.
… threading - Revert FeatureView.state from __eq__ - Add run_async for remote sync vs async HTTP - Reserve MATERIALIZING before 202; idempotent store transitions - Thread version through authorize and all materialize server paths - Narrow silent excepts; document version on request models Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
|
Thanks — both blocking items addressed in d8e7a99:
|
Sorry, something went wrong.
| _update_fv_state(store, fv_names, FeatureViewState.GENERATED) | ||
|
|
||
| loop = asyncio.get_running_loop() | ||
| loop.run_in_executor(None, _run_materialize_incremental) |
There was a problem hiding this comment.
non-blocking but I think it's better if we used dedicated executor, instead of default shared pool, for materialization so that it won't block other operations if multiple executors in progress
Sorry, something went wrong.
There was a problem hiding this comment.
Agreed, added separate executor
Sorry, something went wrong.
| endpoint: str, | ||
| payload: Dict[str, Any], | ||
| force: bool = False, | ||
| run_async: bool = True, |
There was a problem hiding this comment.
should this be by default False ? since it's change in behavior for existing users - the call returns successfully even if the server-side materialization fails later.
Sorry, something went wrong.
There was a problem hiding this comment.
Agree, updating soon
Sorry, something went wrong.
Avoid wiping AVAILABLE_ONLINE when a post-success exception fires (e.g. SparkApp CR already cleaned up). Reuse the stuck-state helper so true failures still return to GENERATED. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Preserve sync semantics for remote users; opt into fire-and-forget with run_async=True. Add unit tests that version is threaded through authorize and both sync/async materialize server paths. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Isolate long materialize waits from the default/shared thread pool used by online serving. Pool size via FEAST_MATERIALIZE_MAX_WORKERS (default 2); shut down on app lifespan exit without blocking. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
# [0.66.0](v0.65.0...v0.66.0) (2026-08-21) ### Bug Fixes * Add connection pre-warming for DynamoDB async client ([89240fa](89240fa)), closes [#6060](#6060) * Add remote registry client extra ([#6697](#6697)) ([b8dfcb0](b8dfcb0)) * Address review feedback on FIPS cipher suite configuration ([4a35fba](4a35fba)) * Allow remote-registry first apply for new projects ([39d408d](39d408d)) * Avoid importing feast.feature_store at mcp_server import time ([ddb2e9a](ddb2e9a)) * Bump pymssql to >=2.3.6 for macOS arm64 wheel support ([181eb35](181eb35)), closes [#5636](#5636) [#5193](#5193) [#5636](#5636) * Call ApplySavedDataset RPC instead of ApplyFeatureService in RemoteRegistry.apply_saved_dataset() ([934d341](934d341)) * Catch missing dbt parser dependency in dbt CLI commands ([#6534](#6534)) ([3c2ae3c](3c2ae3c)) * Default authentication to kubernetes auth ([6a4690a](6a4690a)) * Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock ([#6648](#6648)) ([104ad10](104ad10)), closes [#6647](#6647) * Do not pass undeclared feature view columns to ODFV UDFs ([#6527](#6527)) ([75b9463](75b9463)) * downgrade mcp pin to 1.29.0 and fix CI lockfiles and unit tests ([98e5bca](98e5bca)), closes [#6706](#6706) * Feast apply silently ignoring ttl updates to None or timedelta(0) ([#6709](#6709)) ([97b0f25](97b0f25)), closes [#6703](#6703) * Fix mypy TorchTensor type alias error ([#6712](#6712)) ([34de6fa](34de6fa)), closes [#5563](#5563) * Fixed data source creation form gaps ([5d0f7d6](5d0f7d6)) * Handle parameterized and complex Trino types in type map ([326554d](326554d)) * Isolate default user permissions ([e37adbf](e37adbf)) * Isolate projection join key maps ([d1c709d](d1c709d)) * Map Postgres real to FLOAT instead of DOUBLE ([62db435](62db435)) * Merge shared ODFV source projections in feature resolution ([d269946](d269946)), closes [#6621](#6621) * More exhaustive athena types ([a9aaefc](a9aaefc)) * Normalize SQL registry read_path to the psycopg3 driver like path ([#6644](#6644)) ([996c6ea](996c6ea)), closes [#6643](#6643) * **operator:** add spec.services.onlineStore.disabled to opt out of the online store ([d81d4e3](d81d4e3)), closes [#6586](#6586) * Preinstall DuckDB delta extension for tests ([fd4d49d](fd4d49d)), closes [#6743](#6743) * Preserve event-time ordering within Redis online_write_batch ([40fb788](40fb788)), closes [#5163](#5163) * Prevent mutation of cached feature resolution results ([ea17419](ea17419)) * Remote feastRef FeatureStore fails first apply for a new feastProject ([9affee5](9affee5)) * Remove inert subjectaccessreviews and reorganize RBAC rules ([f771ea4](f771ea4)) * Report single-feature-view spark_application materialization success ([a9219d9](a9219d9)), closes [#6673](#6673) * Reset the global security manager after the permissions fixture ([7667215](7667215)) * Resolve kserve with pip --dry-run instead of installing it ([01da132](01da132)), closes [#6732](#6732) * Resolve write_to_offline_store feature view with a single registry lookup ([a42dc85](a42dc85)), closes [#4235](#4235) * Return False from __eq__ on cross-type comparison ([#6637](#6637)) ([0f149a9](0f149a9)), closes [#6636](#6636) * Reuse IdP-issued client tokens until near expiry ([602d752](602d752)) * Reuse the OIDC JWKS client across requests ([#6683](#6683)) ([a1e6fc2](a1e6fc2)) * Separate CronJob and feature-server ServiceAccounts ([398f643](398f643)) * Serialize UnixTimestamp proto values as raw int64 in remote online store transport ([1e7134f](1e7134f)) * Set FIPS cipher suites before pyarrow.flight import to prevent crash on IBM Power ([979b82a](979b82a)) * Support Entra ID (Azure AD) token claims in OIDC auth ([#6631](#6631)) ([f843c63](f843c63)) * UDF/ODFV source rehydrate (+ Postgres / online cache) ([#6655](#6655)) ([5fd7af7](5fd7af7)) * Updated projects-list.json in order to display newly added projects ([#6657](#6657)) ([3a6a103](3a6a103)) * Use correct image name in multi-arch imagetools push step ([faf85e0](faf85e0)) * Use join keys instead of entity names in ODFV materialization ([#6645](#6645)) ([abffebc](abffebc)), closes [#5965](#5965) * use matching proto class per feature view list in SqliteOnlineStore.plan() ([adb8c1c](adb8c1c)), closes [#6658](#6658) * Widen Athena integer type mapping for unsigned ints ([3425783](3425783)) ### Features * Add ConnectionRef to DataSource for pluggable external credential resolution ([28bde01](28bde01)) * Add Feature Service Create in UI ([0399380](0399380)) * Add hybrid to ValidOfflineStoreDBStorePersistenceTypes for HybridOfflineStore support ([#6707](#6707)) ([310ab51](310ab51)), closes [#6701](#6701) * Add MLflow integration support to Feast operator ([#6611](#6611)) ([52999f1](52999f1)) * Add opt-in filter_by_created_timestamp cutoff to get_historical_features ([#6617](#6617)) ([79b33ce](79b33ce)), closes [#6615](#6615) * Add optional OIDC token audience and issuer verification ([#6670](#6670)) ([ef307c6](ef307c6)) * Add packaged feature repository support to Feast Operator ([8112b1e](8112b1e)), closes [#6598](#6598) * add plan() support to DynamoDBOnlineStore ([51ce982](51ce982)), closes [#6658](#6658) [#6659](#6659) * Added optional namespace/colleciton to datasets ([165fcf2](165fcf2)) * Added SQL registry schema_mode and registry create command ([#6704](#6704)) ([037c4cd](037c4cd)) * Allow users to have protected project on shared registry ([f9923bc](f9923bc)) * Apply Intermediate TLS defaults on API fallback and handle transient errors ([#6587](#6587)) ([43ae993](43ae993)) * **cli:** Updated feast init demo by adding rag template ([#5946](#5946)) ([c8628eb](c8628eb)), closes [#5264](#5264) * Expose the OIDC JWKS tunables through the operator ([#6690](#6690)) ([fef4e78](fef4e78)), closes [#6683](#6683) * Making feast vector store with open ai search api compatible ([#6121](#6121)) ([54da19a](54da19a)) * Multi-arch publish for feast operator image ([b221036](b221036)) * OpenLineage lineage enhancements - full object coverage, richer UI, and API-level sync ([#6719](#6719)) ([120a868](120a868)) * **operator:** Add spec.services.initImage for init container image override ([#6598](#6598)) ([ca355cb](ca355cb)) * Pass optional OIDC audience and issuer through the operator ([#6677](#6677)) ([a13ed7b](a13ed7b)), closes [#6670](#6670) * **server:** Remote Materialization ([#6649](#6649)) ([b7ae488](b7ae488)), closes [#4526](#4526) * Support Lineage configs via operator ([bf1e54a](bf1e54a)) * Updated datasets UI to support grouping ([7ae64ec](7ae64ec))
Summary
Supersedes #6590 (clean single-commit history with proper DCO).
Add ?async=true query parameter support to the existing /materialize and /materialize-incremental endpoints. When set, materialization runs in a background thread and the endpoint returns 202 Accepted immediately. Concurrent requests for an already-MATERIALIZING FV are rejected with 409. Also adds ?force=true to allow operators to override stuck MATERIALIZING state (e.g. after a crashed SparkApplication pod).
Client-side (SDK)
(URL/TLS derived from online_store config)
Server-side
Registry
AVAILABLE_ONLINE (parity with file-based registry)
Which issue(s) this PR fixes
Addresses #4526
Checks
Testing Strategy
Misc