FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

BUG: Include a fingerprint of the function body in cache keys by basnijholt · Pull Request #965 · pipefunc/pipefunc · GitHub

BUG: Include a fingerprint of the function body in cache keys - #965

Merged
basnijholt merged 1 commit into
mainfrom
cache-key-function-hash
Jun 10, 2026
Merged

BUG: Include a fingerprint of the function body in cache keys#965
basnijholt merged 1 commit into
mainfrom
cache-key-function-hash

Conversation

Copy link
Copy Markdown
Collaborator

Closes #964.

Problem

The cache key contained only the output_name and the input values — never the function's implementation (PipeFunc._cache_id only included a hash when the callable defined __pipefunc_hash__). Consequences:

  • Redefine a function in a Jupyter notebook → the pipeline silently returns the stale result of the old implementation.
  • With a persistent DiskCache, stale results even survive kernel restarts and new sessions.

This was the remaining source of the caching confusion reported in #902 (the mutation half was fixed in #905).

Solution

New pipefunc.cache.compute_function_hash(func) fingerprints a callable's implementation, and PipeFunc._cache_id appends it to the key. Resolution order:

  1. __pipefunc_hash__() if defined — the existing explicit hook keeps highest priority (unchanged behavior).
  2. inspect.getsource(func) — works for ordinary functions including notebook cells (IPython registers cell source in linecache).
  3. Code-object fallback (co_code/co_consts/co_names + defaults, nested code objects expanded recursively since their repr contains memory addresses) for sourceless functions, e.g. exec-defined.
  4. Last resort: previous name-only behavior plus a UserWarning, so silent staleness is never the failure mode (e.g. builtins).

functools.partial unwraps with its bound arguments folded in; callable class instances fingerprint their class (instance state is intentionally excluded — that is what __pipefunc_hash__ is for); NestedPipeFunc combines its children's fingerprints.

Design notes: source-hash chosen over bytecode-primary (human-predictable invalidation, stable across Python versions, works in notebooks; same approach as joblib.Memory) and over cloudpickle-bytes hashing (module functions pickle by reference → no invalidation; pickled set ordering interacts with hash randomization → nondeterministic keys across sessions).

Performance

Computed once per PipeFunc instance (_cache_id is a cached_property), lazily on first cache use: ~0.6 ms cold (file read), zero marginal cost per cache entry — per-entry key computation just reuses the precomputed string.

Breaking-ish

Existing persistent DiskCache entries are invalidated once on upgrade (keys changed). Arguably the point of the fix.

Limitation (documented)

Editing a helper function called by the cached function does not invalidate — same limitation as joblib; __pipefunc_hash__ is the escape hatch. Documented in the caching concept page.

Verification

  • End-to-end repro: populate DiskCache with v1 of a function, redefine with a different body → recomputes; redefine with the identical body → still hits the cache.
  • Unit tests: identical source → equal hash, body change → different hash, partial bound-args, exec-defined (code-object path), builtins → None + warning path, callable instances, NestedPipeFunc child edits propagate.
  • Six existing exact-key test assertions updated to use _cache_id dynamically.
  • Full suite: 1417 passed; pre-commit run --all-files and doc-string-check clean.

Editing a cached function previously returned stale results because the
cache key only contained the output name: redefining a function in a
Jupyter notebook, or restarting after a code change with a persistent
DiskCache, silently served results of the old implementation.

- New `pipefunc.cache.compute_function_hash`: hashes the function's
  source (falling back to the code object for sourceless functions,
  recursing into `functools.partial` and callable instances' classes).
- `PipeFunc._cache_id` now appends this fingerprint; `__pipefunc_hash__`
  still takes precedence when defined. If no fingerprint can be computed
  (e.g. builtins), a warning is emitted and the old behavior is kept.
- `NestedPipeFunc` combines the fingerprints of its children.
- Existing persistent caches are invalidated once on upgrade.

Closes #964

Copy link
Copy Markdown
Contributor

✅ PR Title Formatted Correctly

The title of this PR has been updated to match the correct format. Thank you!

codecov Bot commented Jun 10, 2026
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
pipefunc/_pipefunc.py 100.00% <100.00%> (ø)
pipefunc/cache.py 100.00% <100.00%> (ø)

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codspeed-hq Bot commented Jun 10, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 6 untouched benchmarks


Comparing cache-key-function-hash (07fb913) with main (2eae216)

basnijholt merged commit cd7db17 into main Jun 10, 2026
21 checks passed
basnijholt deleted the cache-key-function-hash branch June 10, 2026 20:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache returns stale results after editing a function (function body not part of cache key)

1 participant


Back | FazBrowse Home | New Git URL