FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat: Add Lance optional dependencies by haoxu0 · Pull Request #6958 · feast-dev/feast · GitHub

Repository navigation

feat: Add Lance optional dependencies - #6958

Open
haoxu0 wants to merge 4 commits into
feast-dev:masterfrom
haoxu0:feat/lance-dependencies
Open

haoxu0 wants to merge 4 commits into
feast-dev:masterfrom
haoxu0:feat/lance-dependencies

Conversation

haoxu0 commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds an installable lance extra for the native Lance read/write paths:

  • pylance>=12.0.0
  • lance-namespace>=0.11.1
  • includes the extra in feast[ci]
  • documents pip install 'feast[lance]'
  • adds a packaging metadata regression test

This is split from #6943 so the data-source implementation and packaging change can be reviewed independently.

Verification

  • pytest sdk/python/tests/unit/infra/data_sources/test_lance_dependencies.py -q
  • ruff check sdk/python/tests/unit/infra/data_sources/test_lance_dependencies.py
  • ruff format --check sdk/python/tests/unit/infra/data_sources/test_lance_dependencies.py
  • pip install --dry-run '.[lance]'

Related: #6943, #6945

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
haoxu0 requested a review from a team as a code owner October 7, 2026 04:30

codecov-commenter commented Oct 7, 2026 •
edited
Loading

Copy link
Copy Markdown

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.69%. Comparing base (9d42729) to head (afe2961).
⚠️ Report is 1 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6958      +/-   ##
==========================================
+ Coverage   49.32%   49.69%   +0.36%     
==========================================
  Files         443      443              
  Lines       55094    55094              
  Branches     8017     8017              
==========================================
+ Hits        27176    27378     +202     
+ Misses      26035    25814     -221     
- Partials     1883     1902      +19     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 51.11% <ø> (+0.39%) ⬆️
see 4 files with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bf930e5...afe2961. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Member

Need to run make lock-python-dependencies-all and pixi lock

haoxu0 commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — you are right on both counts, and I want to be straight that I cannot run either command from my environment. Here is what I was able to establish, in case it saves you time.

pixi lock is required, and I initially thought it would not be. I assumed a new PyPI extra could not affect it, since none of the three [tool.pixi.feature.*.pypi-dependencies] entries references ci or lance — they use aws/gcp/..., duckdb/delta/... and ray/..., so nothing in the resolved package set changes. That reasoning was incomplete: feast is an editable path dependency, so pixi.lock records its full requires_dist, including this line:

- feast[aws,azure,cassandra,...,iceberg,image,k8s,mcp,milvus,...] ; extra == 'ci'

which is the pre-change expansion and does not contain lance. So the lock does need regenerating, but the change should be confined to that one block: the ci list gains lance, plus two new ; extra == 'lance' entries for pylance and lance-namespace. No resolution change, because no feature pulls those extras. I deliberately did not hand-edit the file, since a hand edit that does not match the generator is worse than none.

For the requirements, only py3.{10,11,12}-ci-requirements.txt are affected. lance is referenced solely by the ci extra, and those are the only requirement files containing torch:

py3.10-ci-requirements.txt                    torch=2
py3.10-minimal-requirements.txt               torch=0
py3.10-minimal-sdist-requirements.txt         torch=0
py3.10-minimal-sdist-requirements-build.txt   torch=0
py3.10-requirements.txt                       torch=0

I confirmed the non-ci files are unaffected by regenerating py3.10-requirements.txt on this branch: zero pylance/lance-namespace lines appear.

One thing worth deciding before the refresh runs. I regenerated py3.10-requirements.txt from master's unmodified pyproject.toml as a control, and it already differs from the committed file by 1551 lines and 14 version pins — fastapi 0.141.1 → 0.142.2, mypy 2.3.1 → 2.4.0, plus markupsafe, mmh3, pyjwt, python-dotenv, pytz, charset-normalizer, librt, ast-serialize and a new opentelemetry-api transitive. That is upstream drift since the last refresh, not anything this PR introduces, but it means make lock-python-dependencies-all run today will bundle those bumps into this PR whoever runs it. Happy either way — I just did not want to hand you a dependency PR carrying a dozen unrelated upgrades without flagging it.

Why I cannot run it. pixi run needs conda.anaconda.org and the --torch-backend cpu compile needs download.pytorch.org, which then fans out to download-r2.pytorch.org; all three are blocked by my network egress policy, and add_cpu_torch_hashes.py invokes --torch-backend cpu a second time, so there is no way around the PyTorch index. Everything else resolves fine, so this is specifically the conda and PyTorch hosts.

Could you run make lock-python-dependencies-all and pixi lock on this branch, or would you prefer I close this and have someone fold the extra into a refresh you were going to do anyway? I am glad to do any other part of it.

haoxu0 commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up: I tried running both commands and make lock-python-dependencies-all cannot complete on master, independently of this PR.

ValueError: Missing CPU hashes for torch==2.14.1+cpu
make: *** [lock-python-dependencies-all] Error 1

add_cpu_torch_hashes.py looks each torch entry up by appending +cpu to its version, but a --universal --torch-backend cpu resolve emits two pins per package, and the second is already +cpu:

torch==2.14.0 ; sys_platform == 'darwin'
torch==2.14.0+cpu ; sys_platform != 'darwin'

so it asks for 2.14.0+cpu+cpu and raises. That shape is in all three committed py3.*-ci-requirements.txt today, so it is not something this branch introduced. The script and its tests landed together in #6188 and the requirements were last regenerated before that, and the existing test feeds in a bare torch==2.13.0, which is why it went unnoticed.

Fix is up as #6959, with regression tests for the split shape and a check against all three committed files. Once that merges I will rerun the lock target here and push the result.

One note for whoever regenerates: the target starts with rm -rf sdk/python/requirements/*, so a failed run leaves 14 of the 15 files deleted. Worth a git checkout -- sdk/python/requirements/ afterwards.

pixi lock itself works and is unaffected.

`feast` is an editable path dependency, so `pixi.lock` records its full
`requires_dist`, including the expansion of every extra. Adding the `lance`
extra therefore changes the lock even though no pixi feature consumes it.

The change is confined to that one block: the `ci` expansion gains `lance`,
and the two `lance` requirements are recorded. No package resolves
differently, which is what the diff size shows -- 3 insertions and 1
deletion, and the 1377 pypi and 211 conda artifact URLs are unchanged.

Signed-off-by: hao-xu5 <hxu44@apple.com>

haoxu0 commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

pixi lock is done and pushed as df2b0ec.

The change is confined to the one block I expected, because feast is an editable path dependency and so the lock records its full requires_dist including every extra's expansion:

+  - pylance>=12.0.0 ; extra == 'lance'
+  - lance-namespace>=0.11.1 ; extra == 'lance'
-  - feast[...,k8s,mcp,...] ; extra == 'ci'
+  - feast[...,k8s,lance,mcp,...] ; extra == 'ci'

3 insertions, 1 deletion. Nothing resolves differently — the 1377 pypi and 211 conda artifact URLs are byte-identical to before, which is expected since none of the three pixi features consumes the ci or lance extra.

make lock-python-dependencies-all is still blocked on #6959. I will rerun it and push the requirements here once that merges.

haoxu0 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Updated in afe2961:

  • synced the branch with current master and resolved the mcp-server/lance extra overlap
  • ran pixi lock; it reports the lockfile is up to date
  • regenerated all three requirements files affected by the ci extra: py3.10-ci-requirements.txt, py3.11-ci-requirements.txt, and py3.12-ci-requirements.txt
  • each now contains locked lance-namespace==0.11.1 and pylance==13.0.0 entries with hashes

I also retried make lock-python-dependencies-all. With the #6959 fix applied locally it gets past the CPU torch hash issue, but the target then fails in the unrelated minimal-sdist build step because pybuild-deps==0.5.0 calls pip-tools with an incompatible OutputWriter(generate_hashes=...) API. Since lance is only included through the ci extra, the 12 non-CI requirement files are unaffected; I restored those rather than committing unrelated resolver drift.

Local verification after the final master sync:

  • dependency test: 1 passed
  • Ruff: passed
  • pixi lock: already up to date
  • git diff --check origin/master: passed

This branch has not been deployed

No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants


Back | FazBrowse Home | New Git URL