Codecov Report

Attention: Patch coverage is 96.00000% with 1 line in your changes missing coverage. Please review.

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

CodSpeed Performance Report

Merging #842 will not alter performance

Comparing continue-on-errors (ee9672e) with main (2d248ad)

Summary

✅ 6 untouched benchmarks

✅ PR Title Formatted Correctly

The title of this PR has been updated to match the correct format. Thank you!

Hello @basnijholt this feature would be extremely useful for a project I'm working on! I just wanted to check what is left on it and what it would take for it to be merged and released? I might be able to do some work on it if I know what's left to do.

@MitchellAcoustics, I actually did a completely new implementation in #854 that I much prefer. All tests pass but I am taking another look now on what is still needed!

Thanks for leaving a message! What is your overall experience with pipefunc? And have any painpoints or other improvement ideas?

Sounds good! Let me know if I can help with #854 to get it released.

I've experimented with pipefunc in a few academic projects, but now I'm going all out with it for the data science / simulation backend of a bigger software product. Things I love (especially compared to other DAG pipeline projects like Hamilton or Dagster):

• All the standard DAG benefits - transparency and organisation, caching, parallelization, skipping initial computation when interim results are available, etc.
• Pure Python and works entirely within my package/project. So glad it doesn't require a separate server or anything.
• Let's me write my functions essentially as I would anyway. Hamilton was great, but I didn't like that it forced me to change up function naming styles, etc. The choice to use a pipefunc wrapper or define a PipeFunc separately is brilliant. I also like explicitly specifying which PipeFuncs go into which Pipelines and the more advanced stuff you've added with NestedPipeFuncs, VariantPipelines, etc.
• pipeline_to_pydantic is killer for making sure things fail fast and passing data into our backend.
• Once I got used to mapspec I really appreciated how you've set up the parallelization bits.

Some minor pain points I've come across:

• pipeline.run requires an output_name, even if you have a unique leaf node. It'd be nice to be able to omit output_name like in the direct pipeline() call or when full_output=True and just run the full pipeline, even without unique leaf nodes.
• Inconsistencies in accepted types and parameter names for output_name - .map has output_names and doesn't accept a list. Directly calling pipeline() won't accept a list of output names (str | tuple), but pipeline.run() does.
• Using the @pipefunc wrapper prevents the documentation its wrapping from showing up in IDE hover callouts and completions. Maybe this would be solved by adding @functools.wraps into the pipefunc wrapper somehow?
• I don't think I've quite worked out caching quite right yet. Not sure if it works differently when running pipelines in a jupyter notebook, or how to get persistent disk caching working well. This could be a me thing though!

What I'm dealing with now is actually an interesting challenge - applying the pipeline at multiple levels of a nested data structure to analyse a network. I wrote a detailed description of the use case, but I'll move that off to a separate discussion! If you've had any thoughts on using pipefunc with nested data or some particularly clever way of applying/composing together a Pipeline DAG to operate on data which is itself a DAG, I'd love to hear it.

@MitchellAcoustics, I am about to go on vacation but I will definitely address all your points when I come back! Thanks a lot for taking the time to write it out 😄

For now #854 might be ready. It has 100% testing coverage and I am happy with it. However, I want to take some more time and go over it once more in detail. It is a large change and want to re-evaluate whether I made the right decisions. It would be great it you could try it out!