ENH: Store polars DataFrames as Parquet and support pl.LazyFrame inputs - #966
Merged
basnijholt merged 6 commits intoJun 10, 2026
Merged
ENH: Store polars DataFrames as Parquet and support pl.LazyFrame inputs#966basnijholt merged 6 commits into
basnijholt merged 6 commits into
Conversation
Copy link
Copy Markdown
Collaborator
Sorry, something went wrong.
Implements the remaining part of #879: DataFrame outputs are serialized as Parquet files on disk (with cloudpickle fallback), and parameters annotated as pl.LazyFrame receive lazy frames - a true pl.scan_parquet when the upstream output is stored as Parquet, otherwise .lazy().
Comment thread
pipefunc/typing.py
|
|
||
| import numpy as np | ||
|
|
||
| from pipefunc._utils import is_imported |
Copy link
Copy Markdown
Codecov Report✅ All modified and coverable lines are covered by tests.
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
Sorry, something went wrong.
Avoids iterating all parameter annotations per map element in _convert_lazyframe_kwargs; the no-polars-params fast path is now a single cached attribute lookup (~46 ns/call).
Copy link
Copy Markdown
Merging this PR will not alter performance✅ 6 untouched benchmarks Comparing polars (7abd39f) with main (2eae216) |
Sorry, something went wrong.
…conversion onto PipeFunc - load() previously opened every file twice (magic-byte sniff + read); now peeks and rewinds a single handle. This path is hot for all users. - _convert_lazyframe_kwargs is now a PipeFunc method, living next to the _lazyframe_parameters cache it uses, so _pipeline/_base.py no longer imports a private helper from map/_run.py. - Use bytes.startswith for the magic check in _load_all.
The all-deps CI session imports polars at collection time, so the
is_imported('polars') early returns never executed there (codecov/patch
flagged them). Simulate via monkeypatch.delitem(sys.modules, 'polars').
…itive CodeQL mis-resolves the stdlib 'typing' import inside _utils.py as pipefunc.typing (the package has a module of that name), so importing is_imported from _utils here looked like a cyclic import. Use the sys.modules check directly, as cache.py already does.
…alse positive" This reverts commit ce7db6b.
Copy link
Copy Markdown
Contributor
✅ PR Title Formatted CorrectlyThe title of this PR has been updated to match the correct format. Thank you! |
Sorry, something went wrong.
This was referenced Jun 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
1 more reviewer2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the remaining part of #879 (see this comment): Parquet on-disk serialization for polars.DataFrame outputs and lazy (pl.LazyFrame) loading in downstream functions.
Deliberately out of scope: sinking pl.LazyFrame outputs to Parquet (load() couldn't distinguish them from DataFrame outputs), lazy scanning of individual mapspec elements, and zarr storage (keeps its pickle codec).
Testing
Closes #879