| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
`pa_to_feast_value_type` had no handling for `fixed_size_list`, so decoding an Arrow vector column raised `KeyError: 'fixed_size_list<item: float>[8]'`. The variable-width spelling `list<item: float>` already decoded to `FLOAT_LIST`, so only the fixed-width spelling was affected. That spelling is how Arrow-native stores represent an embedding, and it is the type `_validate_vector_field_lengths` reads a vector's width from, so schema inference failed on exactly the columns the vector support added in feast-dev#6909 is meant to describe. A Feast value type carries no width -- `Field.vector_length` carries it -- so the fixed-width spelling is rewritten into the variable-width one and decodes identically, leaving the list handling single-sourced. Nesting is preserved, so a nested fixed-size list still decodes to `VALUE_LIST`. Signed-off-by: hao-xu5 <hxu44@apple.com>
|
⚠️ Please install the Codecov Report❌ Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.
@@ Coverage Diff @@
## master #6944 +/- ##
==========================================
+ Coverage 49.04% 49.06% +0.01%
==========================================
Files 433 433
Lines 54308 54314 +6
Branches 7910 7912 +2
==========================================
+ Hits 26637 26647 +10
+ Misses 25790 25788 -2
+ Partials 1881 1879 -2
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
What this does
pa_to_feast_value_type had no handling for fixed_size_list, so decoding an Arrow vector column raised a KeyError:
Only the fixed-width spelling was affected. "fixed_size_list" not in inspect.getsource(feast.type_map) held before this change.
Why it matters
fixed_size_list is how Arrow-native stores spell an embedding column, and it is the type _validate_vector_field_lengths reads a width from — the validator added in #6909. So schema inference failed on exactly the columns that vector support is meant to describe.
Any source whose schema inference routes through pa_to_feast_value_type is affected, not one particular store. It surfaces through get_pyarrow_schema_from_batch_source, so offline_write_batch and feast apply schema inference both hit it.
Approach
A Feast value type carries no width — Field.vector_length carries it — so the two spellings must agree. The fixed-width spelling is rewritten into the variable-width one and then flows through the existing list handling unchanged, which keeps that logic single-sourced rather than duplicating the element-type table.
Nesting is preserved: fixed_size_list<item: list<item: float>>[8] still decodes to VALUE_LIST, matching list<item: list<item: float>>.
Verification
Each is asserted equal to its variable-width counterpart, and one test pins the parser against the exact string PyArrow emits (str(pyarrow.list_(pyarrow.float32(), 8))) so the format cannot drift away unnoticed.
5 tests added. sdk/python/tests/unit/test_type_map.py: 24 failures before the change and the same 24 after, by name — all pre-existing TestSparkNativeTypeValidation cases that need pyspark. 247 passed.
ruff check and ruff format --check clean.