fix: UDF/ODFV source rehydrate (+ Postgres / online cache) - #6655
Merged
ntkathole merged 11 commits intoAug 17, 2026
Merged
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
Comment thread
sdk/python/feast/transformation/udf_rehydrate.py
Fixed
Copy link
Copy Markdown
|
⚠️ Please install the Codecov Report❌ Patch coverage is 64.59627% with 57 lines in your changes missing coverage. Please review.
@@ Coverage Diff @@
## master #6655 +/- ##
==========================================
+ Coverage 46.82% 46.92% +0.09%
==========================================
Files 415 416 +1
Lines 50406 50549 +143
Branches 7215 7252 +37
==========================================
+ Hits 23601 23718 +117
- Misses 25155 25157 +2
- Partials 1650 1674 +24
Continue to review full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Sorry, something went wrong.
franciscojavierarceo
requested changes
Jul 29, 2026
franciscojavierarceo
left a comment
Copy link
Copy Markdown
Member
There was a problem hiding this comment.
Two blocking concerns from the remote diff:
- feature_server.py and _get_online_request_context() now pass allow_cache=False for feature-service and feature-view resolution on every online request. For SQL-backed registries this moves backing-registry I/O into the serving hot path, which can materially regress latency and registry load. Please refresh/invalidate the affected cached definitions when UDF state changes (or isolate and justify this serving behavior separately) instead of globally bypassing the cache.
- CodeQL's open finding in transformation/udf_rehydrate.py is valid: the decorator-stripping regex has nested unbounded repetition and processes registry-provided source text. Please replace it with a linear parser/line-based approach and add an adversarial regression case.
Sorry, something went wrong.
aniketpalu
added a commit
to aniketpalu/feast
that referenced
this pull request
Jul 30, 2026
Post-materialize MATERIALIZING gate lag is handled by client retry / registry TTL refresh, not by bypassing the registry cache on every online request. Addresses review feedback on feast-dev#6655. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
aniketpalu
added a commit
to aniketpalu/feast
that referenced
this pull request
Jul 31, 2026
…he) into merged-spark-e2e Resolved .secrets.baseline conflict by taking PR tip. Co-authored-by: Cursor <cursoragent@cursor.com>
aniketpalu
added a commit
to aniketpalu/feast
that referenced
this pull request
Aug 6, 2026
Post-materialize MATERIALIZING gate lag is handled by client retry / registry TTL refresh, not by bypassing the registry cache on every online request. Addresses review feedback on feast-dev#6655. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
aniketpalu
force-pushed
the
fix/udf-source-rehydrate-postgres-online-cache
branch
from
fd2178d to
3028ff9
Compare
August 6, 2026 10:28
…istry reads Rehydrate BatchFeatureView/ODFV callables from body_text (strip leading decorators) before dill.loads to avoid Spark driver exit 139 and cross-Python serve failures. Treat empty Postgres feature_name_columns as SELECT *. Disable registry cache on the online request path so FeatureView state gates see AVAILABLE_ONLINE immediately after materialize. RayTransformation.from_proto left unchanged (follow-up). Signed-off-by: Aniket Paluskar <apaluska@redhat.com>
Post-materialize MATERIALIZING gate lag is handled by client retry / registry TTL refresh, not by bypassing the registry cache on every online request. Addresses review feedback on feast-dev#6655. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Avoid CodeQL ReDoS finding on nested @/newline regex when rehydrating body_text. Add multiline and adversarial @ spam unit tests. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
ntkathole
force-pushed
the
fix/udf-source-rehydrate-postgres-online-cache
branch
from
648ef47 to
708ff26
Compare
August 10, 2026 13:21
Source-first from_proto rebuilds callables whose bytecode differs from the live repo function; requiring co_code equality made no-op feast apply rewrite ODFVs and broke universal CLI integration tests. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
RayTransformation now compares by udf_string when present (same contract as Pandas/Python) so a future source-first from_proto will not break no-op apply. SparkTransformationNode caches resolve_udf so strip+exec runs once per node lifetime. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
ntkathole
approved these changes
Aug 17, 2026
franciscojavierarceo
pushed a commit
that referenced
this pull request
Aug 21, 2026
# [0.66.0](v0.65.0...v0.66.0) (2026-08-21) ### Bug Fixes * Add connection pre-warming for DynamoDB async client ([89240fa](89240fa)), closes [#6060](#6060) * Add remote registry client extra ([#6697](#6697)) ([b8dfcb0](b8dfcb0)) * Address review feedback on FIPS cipher suite configuration ([4a35fba](4a35fba)) * Allow remote-registry first apply for new projects ([39d408d](39d408d)) * Avoid importing feast.feature_store at mcp_server import time ([ddb2e9a](ddb2e9a)) * Bump pymssql to >=2.3.6 for macOS arm64 wheel support ([181eb35](181eb35)), closes [#5636](#5636) [#5193](#5193) [#5636](#5636) * Call ApplySavedDataset RPC instead of ApplyFeatureService in RemoteRegistry.apply_saved_dataset() ([934d341](934d341)) * Catch missing dbt parser dependency in dbt CLI commands ([#6534](#6534)) ([3c2ae3c](3c2ae3c)) * Default authentication to kubernetes auth ([6a4690a](6a4690a)) * Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock ([#6648](#6648)) ([104ad10](104ad10)), closes [#6647](#6647) * Do not pass undeclared feature view columns to ODFV UDFs ([#6527](#6527)) ([75b9463](75b9463)) * downgrade mcp pin to 1.29.0 and fix CI lockfiles and unit tests ([98e5bca](98e5bca)), closes [#6706](#6706) * Feast apply silently ignoring ttl updates to None or timedelta(0) ([#6709](#6709)) ([97b0f25](97b0f25)), closes [#6703](#6703) * Fix mypy TorchTensor type alias error ([#6712](#6712)) ([34de6fa](34de6fa)), closes [#5563](#5563) * Fixed data source creation form gaps ([5d0f7d6](5d0f7d6)) * Handle parameterized and complex Trino types in type map ([326554d](326554d)) * Isolate default user permissions ([e37adbf](e37adbf)) * Isolate projection join key maps ([d1c709d](d1c709d)) * Map Postgres real to FLOAT instead of DOUBLE ([62db435](62db435)) * Merge shared ODFV source projections in feature resolution ([d269946](d269946)), closes [#6621](#6621) * More exhaustive athena types ([a9aaefc](a9aaefc)) * Normalize SQL registry read_path to the psycopg3 driver like path ([#6644](#6644)) ([996c6ea](996c6ea)), closes [#6643](#6643) * **operator:** add spec.services.onlineStore.disabled to opt out of the online store ([d81d4e3](d81d4e3)), closes [#6586](#6586) * Preinstall DuckDB delta extension for tests ([fd4d49d](fd4d49d)), closes [#6743](#6743) * Preserve event-time ordering within Redis online_write_batch ([40fb788](40fb788)), closes [#5163](#5163) * Prevent mutation of cached feature resolution results ([ea17419](ea17419)) * Remote feastRef FeatureStore fails first apply for a new feastProject ([9affee5](9affee5)) * Remove inert subjectaccessreviews and reorganize RBAC rules ([f771ea4](f771ea4)) * Report single-feature-view spark_application materialization success ([a9219d9](a9219d9)), closes [#6673](#6673) * Reset the global security manager after the permissions fixture ([7667215](7667215)) * Resolve kserve with pip --dry-run instead of installing it ([01da132](01da132)), closes [#6732](#6732) * Resolve write_to_offline_store feature view with a single registry lookup ([a42dc85](a42dc85)), closes [#4235](#4235) * Return False from __eq__ on cross-type comparison ([#6637](#6637)) ([0f149a9](0f149a9)), closes [#6636](#6636) * Reuse IdP-issued client tokens until near expiry ([602d752](602d752)) * Reuse the OIDC JWKS client across requests ([#6683](#6683)) ([a1e6fc2](a1e6fc2)) * Separate CronJob and feature-server ServiceAccounts ([398f643](398f643)) * Serialize UnixTimestamp proto values as raw int64 in remote online store transport ([1e7134f](1e7134f)) * Set FIPS cipher suites before pyarrow.flight import to prevent crash on IBM Power ([979b82a](979b82a)) * Support Entra ID (Azure AD) token claims in OIDC auth ([#6631](#6631)) ([f843c63](f843c63)) * UDF/ODFV source rehydrate (+ Postgres / online cache) ([#6655](#6655)) ([5fd7af7](5fd7af7)) * Updated projects-list.json in order to display newly added projects ([#6657](#6657)) ([3a6a103](3a6a103)) * Use correct image name in multi-arch imagetools push step ([faf85e0](faf85e0)) * Use join keys instead of entity names in ODFV materialization ([#6645](#6645)) ([abffebc](abffebc)), closes [#5965](#5965) * use matching proto class per feature view list in SqliteOnlineStore.plan() ([adb8c1c](adb8c1c)), closes [#6658](#6658) * Widen Athena integer type mapping for unsigned ints ([3425783](3425783)) ### Features * Add ConnectionRef to DataSource for pluggable external credential resolution ([28bde01](28bde01)) * Add Feature Service Create in UI ([0399380](0399380)) * Add hybrid to ValidOfflineStoreDBStorePersistenceTypes for HybridOfflineStore support ([#6707](#6707)) ([310ab51](310ab51)), closes [#6701](#6701) * Add MLflow integration support to Feast operator ([#6611](#6611)) ([52999f1](52999f1)) * Add opt-in filter_by_created_timestamp cutoff to get_historical_features ([#6617](#6617)) ([79b33ce](79b33ce)), closes [#6615](#6615) * Add optional OIDC token audience and issuer verification ([#6670](#6670)) ([ef307c6](ef307c6)) * Add packaged feature repository support to Feast Operator ([8112b1e](8112b1e)), closes [#6598](#6598) * add plan() support to DynamoDBOnlineStore ([51ce982](51ce982)), closes [#6658](#6658) [#6659](#6659) * Added optional namespace/colleciton to datasets ([165fcf2](165fcf2)) * Added SQL registry schema_mode and registry create command ([#6704](#6704)) ([037c4cd](037c4cd)) * Allow users to have protected project on shared registry ([f9923bc](f9923bc)) * Apply Intermediate TLS defaults on API fallback and handle transient errors ([#6587](#6587)) ([43ae993](43ae993)) * **cli:** Updated feast init demo by adding rag template ([#5946](#5946)) ([c8628eb](c8628eb)), closes [#5264](#5264) * Expose the OIDC JWKS tunables through the operator ([#6690](#6690)) ([fef4e78](fef4e78)), closes [#6683](#6683) * Making feast vector store with open ai search api compatible ([#6121](#6121)) ([54da19a](54da19a)) * Multi-arch publish for feast operator image ([b221036](b221036)) * OpenLineage lineage enhancements - full object coverage, richer UI, and API-level sync ([#6719](#6719)) ([120a868](120a868)) * **operator:** Add spec.services.initImage for init container image override ([#6598](#6598)) ([ca355cb](ca355cb)) * Pass optional OIDC audience and issuer through the operator ([#6677](#6677)) ([a13ed7b](a13ed7b)), closes [#6670](#6670) * **server:** Remote Materialization ([#6649](#6649)) ([b7ae488](b7ae488)), closes [#4526](#4526) * Support Lineage configs via operator ([bf1e54a](bf1e54a)) * Updated datasets UI to support grouping ([7ae64ec](7ae64ec))
HSarwat
pushed a commit
to HSarwat/feast
that referenced
this pull request
Aug 22, 2026
…#6655) * fix: prefer UDF source over dill; Postgres SELECT *; fresh online registry reads Rehydrate BatchFeatureView/ODFV callables from body_text (strip leading decorators) before dill.loads to avoid Spark driver exit 139 and cross-Python serve failures. Treat empty Postgres feature_name_columns as SELECT *. Disable registry cache on the online request path so FeatureView state gates see AVAILABLE_ONLINE immediately after materialize. RayTransformation.from_proto left unchanged (follow-up). Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * Lint Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * fix: drop global allow_cache=False on online path Post-materialize MATERIALIZING gate lag is handled by client retry / registry TTL refresh, not by bypassing the registry cache on every online request. Addresses review feedback on feast-dev#6655. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * fix: replace UDF decorator strip regex with linear scan Avoid CodeQL ReDoS finding on nested @/newline regex when rehydrating body_text. Add multiline and adversarial @ spam unit tests. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * fix: cast resolve_udf result to FunctionType for mypy Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * fix: compare UDF transformations by udf_string when present Source-first from_proto rebuilds callables whose bytecode differs from the live repo function; requiring co_code equality made no-op feast apply rewrite ODFVs and broke universal CLI integration tests. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * lint Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * fix: align Ray __eq__ with source identity; cache Spark UDF resolve RayTransformation now compares by udf_string when present (same contract as Pandas/Python) so a future source-first from_proto will not break no-op apply. SparkTransformationNode caches resolve_udf so strip+exec runs once per node lifetime. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes three issues found while testing BatchFeatureView UDFs and OnDemandFeatureViews with Spark transformations (SparkApplication / SparkComputeEngine path) against a remote feature server.
All three blocked a clean UDF/ODFV materialize → online path. This PR is scoped to that story. Shared helper feast.transformation.udf_rehydrate is introduced so the same contract can later be wired into RayTransformation.from_proto (intentionally not changed here).
1. Spark BFV UDF — prefer source over dill (exit 139)
Problem: Materializing a BatchFeatureView with a Python UDF on Spark could segfault the driver (exit 139). Registry stores both dill (body) and source (body_text / udf_string); the Spark path executed the dill-restored callable, which is unsafe with DataFrame ops on our Spark stack.
Change: SparkTransformationNode resolves the UDF via shared resolve_udf: exec trusted udf_string when present; fall back to the existing callable / dill. SparkFeatureBuilder passes udf_string through.
Files: spark/nodes.py, spark/feature_builder.py, transformation/udf_rehydrate.py
2. ODFV / local transformations — source-first from_proto
Problem: ODFV serve (and feast-apply rehydrate from SQL registry) used dill.loads(body) only. That fails across Python minors (e.g. apply on Spark driver 3.10, serve on feature-server 3.12) and when body_text includes @on_demand_feature_view(...) (NameError / CrashLoop if source is exec'd naively).
Change: PandasTransformation.from_proto and PythonTransformation.from_proto use the same resolve_udf (strip leading decorators, then exec; dill fallback).
Out of scope (follow-up): RayTransformation.from_proto remains dill.loads for now.
Files: pandas_transformation.py, python_transformation.py, udf_rehydrate.py
3. Postgres empty feature_name_columns → SELECT *
Problem: BatchFeatureView python/pandas mode signals "all columns" with empty feature_name_columns. Postgres offline store still projected only join keys + timestamps, starving the UDF of input features.
Change: When feature_name_columns is empty, pull_latest / pull_all use a.* / alias.* (aligned with Ray offline-store semantics).
Files: postgres_offline_store/postgres.py + unit tests
4. Fresh online registry reads
Problem: After materialize completes, online serving could briefly (or stickily) gate on cached MATERIALIZING / stale FV metadata.
Change: Online request path uses allow_cache=False when resolving features / feature services (utils._get_online_request_context, feature server _get_features).
Files: utils.py, feature_server.py
Out of scope
Which issue(s) this PR fixes
Found during BYOS Spark UDF/ODFV E2E; no single upstream issue required. Related discussions: Spark UDF materialize reliability, ODFV cross-image Python skew.
Checks
Testing Strategy
Misc