FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

MAINT: Deprecate `cleanup` parameter in favor of `resume` parameter by basnijholt · Pull Request #921 · pipefunc/pipefunc · GitHub

6 changes: 3 additions & 3 deletions docs/source/concepts/cli.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ In other words, you can:
- **Automatically generate a CLI** from your pipeline’s input parameters.
- **Validate and coerce user inputs** using a dynamically created Pydantic model.
- **Extract parameter descriptions** from your function docstrings (if packages like Griffe are installed) so that the CLI help text includes detailed information.
- **Configure mapping options** (e.g., parallel execution, storage method, and cleanup) via dedicated command‐line flags.
- **Configure mapping options** (e.g., parallel execution, storage method, and resume behavior) via dedicated command‐line flags.
- **Print pipeline documentation** directly via the `docs` subcommand.

```{note}
Expand Down Expand Up @@ -60,7 +60,7 @@ The CLI supports three modes:
python cli-example.py docs
```

In CLI and JSON modes, additional mapping options (prefixed with `--map-`) allow you to control how the pipeline executes, including settings like the run folder, parallel execution, storage backend, and cleanup behavior.
In CLI and JSON modes, additional mapping options (prefixed with `--map-`) allow you to control how the pipeline executes, including settings like the run folder, parallel execution, storage backend, and resume behavior.

---

Expand All @@ -81,7 +81,7 @@ When you invoke `pipeline.cli()`, the following steps occur:
- **`docs`**: Prints the pipeline documentation.

3. **Mapping Options:**
Mapping-related options (e.g., `--map-run_folder`, `--map-parallel`, `--map-storage`, and `--map-cleanup`) are added to the `cli` and `json` subcommands, letting you configure pipeline execution without modifying code.
Mapping-related options (e.g., `--map-run_folder`, `--map-parallel`, `--map-storage`, and `--map-resume`) are added to the `cli` and `json` subcommands, letting you configure pipeline execution without modifying code.

4. **Input Validation and Execution:**
For the `cli` and `json` subcommands, the CLI parses and validates the inputs using the generated Pydantic model.
Expand Down
63 changes: 49 additions & 14 deletions pipefunc/_pipeline/_base.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
from pipefunc.map._run import AsyncMap, run_map, run_map_async
from pipefunc.map._run_eager import run_map_eager
from pipefunc.map._run_eager_async import run_map_eager_async
from pipefunc.map._run_info import _handle_cleanup_deprecation
from pipefunc.resources import Resources

from ._autodoc import PipelineDocumentation, format_pipeline_docs
Expand Down Expand Up @@ -769,8 +770,9 @@ def map(
chunksizes: int | dict[OUTPUT_TYPE, int | Callable[[int], int] | None] | None = None,
storage: StorageType | None = None,
persist_memory: bool = True,
cleanup: bool = True,
reuse_validation: Literal["auto", "strict", "skip"] = "auto",
cleanup: bool | None = None,
resume: bool = False,
resume_validation: Literal["auto", "strict", "skip"] = "auto",
fixed_indices: dict[str, int | slice] | None = None,
auto_subpipeline: bool = False,
show_progress: bool | Literal["rich", "ipywidgets", "headless"] | None = None,
Expand Down Expand Up @@ -846,10 +848,23 @@ def map(
Whether to write results to disk when memory based storage is used.
Does not have any effect when file based storage is used.
cleanup
.. deprecated:: 0.89.0
Use `resume` parameter instead. Will be removed in version 1.0.0.

Whether to clean up the ``run_folder`` before running the pipeline.
reuse_validation
When set, takes priority over ``resume`` parameter.
``cleanup=True`` is equivalent to ``resume=False``.
``cleanup=False`` is equivalent to ``resume=True``.
resume
Whether to resume data from a previous run in the ``run_folder``.

- ``False`` (default): Clean up the ``run_folder`` before running (fresh start).
- ``True``: Attempt to load and resume results from a previous run.

Note: If ``cleanup`` is specified, it takes priority over this parameter.
resume_validation
Controls validation strictness when reusing data from a previous run
(only applies when ``cleanup=False``):
(only applies when ``resume=True``):

- ``"auto"`` (default): Validate that inputs/defaults match the previous run.
If equality comparison fails (returns ``None``), warn but proceed anyway.
Expand All @@ -860,7 +875,7 @@ def map(
You are responsible for ensuring inputs are actually identical.

Note: Shapes and MapSpecs are always validated regardless of this setting.
Ignored when ``cleanup=True``.
Ignored when ``resume=False``.
fixed_indices
A dictionary mapping axes names to indices that should be fixed for the run.
If not provided, all indices are iterated over.
Expand Down Expand Up @@ -910,6 +925,8 @@ def map(
use `Result.output` to get the actual result.

"""
resume = _handle_cleanup_deprecation(cleanup, resume, stacklevel=2)

if scheduling_strategy == "generation":
run_map_func = run_map
elif scheduling_strategy == "eager":
Expand All @@ -928,8 +945,9 @@ def map(
chunksizes=chunksizes,
storage=storage,
persist_memory=persist_memory,
cleanup=cleanup,
reuse_validation=reuse_validation,
cleanup=None, # Already handled deprecation above
resume=resume,
resume_validation=resume_validation,
fixed_indices=fixed_indices,
auto_subpipeline=auto_subpipeline,
show_progress=show_progress,
Expand All @@ -947,8 +965,9 @@ def map_async(
chunksizes: int | dict[OUTPUT_TYPE, int | Callable[[int], int] | None] | None = None,
storage: StorageType | None = None,
persist_memory: bool = True,
cleanup: bool = True,
reuse_validation: Literal["auto", "strict", "skip"] = "auto",
cleanup: bool | None = None,
resume: bool = False,
resume_validation: Literal["auto", "strict", "skip"] = "auto",
fixed_indices: dict[str, int | slice] | None = None,
auto_subpipeline: bool = False,
show_progress: bool | Literal["rich", "ipywidgets", "headless"] | None = None,
Expand Down Expand Up @@ -1024,10 +1043,23 @@ def map_async(
Whether to write results to disk when memory based storage is used.
Does not have any effect when file based storage is used.
cleanup
.. deprecated:: 0.89.0
Use `resume` parameter instead. Will be removed in version 1.0.0.

Whether to clean up the ``run_folder`` before running the pipeline.
reuse_validation
When set, takes priority over ``resume`` parameter.
``cleanup=True`` is equivalent to ``resume=False``.
``cleanup=False`` is equivalent to ``resume=True``.
resume
Whether to resume data from a previous run in the ``run_folder``.

- ``False`` (default): Clean up the ``run_folder`` before running (fresh start).
- ``True``: Attempt to load and resume results from a previous run.

Note: If ``cleanup`` is specified, it takes priority over this parameter.
resume_validation
Controls validation strictness when reusing data from a previous run
(only applies when ``cleanup=False``):
(only applies when ``resume=True``):

- ``"auto"`` (default): Validate that inputs/defaults match the previous run.
If equality comparison fails (returns ``None``), warn but proceed anyway.
Expand All @@ -1038,7 +1070,7 @@ def map_async(
You are responsible for ensuring inputs are actually identical.

Note: Shapes and MapSpecs are always validated regardless of this setting.
Ignored when ``cleanup=True``.
Ignored when ``resume=False``.
fixed_indices
A dictionary mapping axes names to indices that should be fixed for the run.
If not provided, all indices are iterated over.
Expand Down Expand Up @@ -1095,6 +1127,8 @@ def map_async(


"""
resume = _handle_cleanup_deprecation(cleanup, resume, stacklevel=2)

if scheduling_strategy == "generation":
run_map_func = run_map_async
elif scheduling_strategy == "eager":
Expand All @@ -1113,8 +1147,9 @@ def map_async(
chunksizes=chunksizes,
storage=storage,
persist_memory=persist_memory,
cleanup=cleanup,
reuse_validation=reuse_validation,
cleanup=None, # Already handled deprecation above
resume=resume,
resume_validation=resume_validation,
fixed_indices=fixed_indices,
auto_subpipeline=auto_subpipeline,
show_progress=show_progress,
Expand Down
13 changes: 7 additions & 6 deletions pipefunc/_pipeline/_cli.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
allowing you to supply parameters interactively (via the `cli` subcommand), load them from a JSON file
(via the `json` subcommand), or simply view the pipeline documentation (via the `docs` subcommand).
Mapping options (prefixed with `--map-`) allow you to configure parallel execution, storage method,
and cleanup behavior. In `cli` or `json` mode it runs `pipeline.map` with the provided inputs and
and resume behavior. In `cli` or `json` mode it runs `pipeline.map` with the provided inputs and
mapping options.

Usage Examples:
Expand All @@ -50,15 +50,15 @@ def cli(pipeline: Pipeline, description: str | None = None) -> None:
- ``docs``: Display the pipeline documentation (using `pipeline.print_documentation`).

Mapping options (prefixed with `--map-`) are available for the `cli` and `json` subcommands to control
parallel execution, storage method, and cleanup behavior.
parallel execution, storage method, and resume behavior.

Usage Examples:

**CLI mode:**
``python cli-example.py cli --x 2 --y 3 --map-parallel false --map-cleanup true``
``python cli-example.py cli --x 2 --y 3 --map-parallel false --map-resume true``

**JSON mode:**
``python cli-example.py json --json-file inputs.json --map-parallel false --map-cleanup true``
``python cli-example.py json --json-file inputs.json --map-parallel false --map-resume true``

**Docs mode:**
``python cli-example.py docs``
Expand Down Expand Up @@ -209,9 +209,10 @@ def _add_map_arguments(parser: argparse.ArgumentParser) -> None:
"run_folder": "run_folder",
"parallel": True,
"storage": "file_array",
"cleanup": True,
"cleanup": None,
"resume": False,
}
include_only = {"run_folder", "parallel", "storage", "cleanup"}
include_only = {"run_folder", "parallel", "storage", "cleanup", "resume"}
for arg, p in sig_map.parameters.items():
if arg not in include_only:
continue
Expand Down
8 changes: 5 additions & 3 deletions pipefunc/map/_prepare.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,9 @@ def prepare_run(
executor: Executor | dict[OUTPUT_TYPE, Executor] | None,
chunksizes: int | dict[OUTPUT_TYPE, int | Callable[[int], int] | None] | None,
storage: str | dict[OUTPUT_TYPE, str] | None,
cleanup: bool,
reuse_validation: Literal["auto", "strict", "skip"],
cleanup: bool | None,
resume: bool,
resume_validation: Literal["auto", "strict", "skip"],
fixed_indices: dict[str, int | slice] | None,
auto_subpipeline: bool,
show_progress: bool | Literal["rich", "ipywidgets", "headless"] | None,
Expand Down Expand Up @@ -81,7 +82,8 @@ def prepare_run(
executor=executor,
storage=_expand_output_name_in_storage(pipeline, storage),
cleanup=cleanup,
reuse_validation=reuse_validation,
resume=resume,
resume_validation=resume_validation,
)
outputs = ResultDict(_inputs_=inputs, _pipeline_=pipeline)
store = run_info.init_store()
Expand Down
59 changes: 47 additions & 12 deletions pipefunc/map/_run.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
from ._mapspec import MapSpec, _shape_to_key
from ._prepare import prepare_run
from ._result import DirectValue, Result, ResultDict
from ._run_info import _handle_cleanup_deprecation
Comment thread Dismissed
from ._shapes import external_shape_from_mask, internal_shape_from_mask, shape_is_resolved
from ._storage_array._base import StorageBase, iterate_shape_indices, select_by_mask

Expand Down Expand Up @@ -74,8 +75,9 @@ def run_map(
chunksizes: int | dict[OUTPUT_TYPE, int | Callable[[int], int] | None] | None = None,
storage: StorageType | None = None,
persist_memory: bool = True,
cleanup: bool = True,
reuse_validation: Literal["auto", "strict", "skip"] = "auto",
cleanup: bool | None = None,
resume: bool = False,
resume_validation: Literal["auto", "strict", "skip"] = "auto",
fixed_indices: dict[str, int | slice] | None = None,
auto_subpipeline: bool = False,
show_progress: bool | Literal["rich", "ipywidgets", "headless"] | None = None,
Expand Down Expand Up @@ -152,10 +154,23 @@ def run_map(
Whether to write results to disk when memory based storage is used.
Does not have any effect when file based storage is used.
cleanup
.. deprecated:: 0.89.0
Use `resume` parameter instead. Will be removed in version 1.0.0.

Whether to clean up the ``run_folder`` before running the pipeline.
reuse_validation
When set, takes priority over ``resume`` parameter.
``cleanup=True`` is equivalent to ``resume=False``.
``cleanup=False`` is equivalent to ``resume=True``.
resume
Whether to resume data from a previous run in the ``run_folder``.

- ``False`` (default): Clean up the ``run_folder`` before running (fresh start).
- ``True``: Attempt to load and resume results from a previous run.

Note: If ``cleanup`` is specified, it takes priority over this parameter.
resume_validation
Controls validation strictness when reusing data from a previous run
(only applies when ``cleanup=False``):
(only applies when ``resume=True``):

- ``"auto"`` (default): Validate that inputs/defaults match the previous run.
If equality comparison fails (returns ``None``), warn but proceed anyway.
Expand All @@ -166,7 +181,7 @@ def run_map(
You are responsible for ensuring inputs are actually identical.

Note: Shapes and MapSpecs are always validated regardless of this setting.
Ignored when ``cleanup=True``.
Ignored when ``resume=False``.
fixed_indices
A dictionary mapping axes names to indices that should be fixed for the run.
If not provided, all indices are iterated over.
Expand Down Expand Up @@ -195,6 +210,8 @@ def run_map(
``storage``. This is useful for very large pipelines where the results do not fit into memory.

"""
resume = _handle_cleanup_deprecation(cleanup, resume, stacklevel=2)

prep = prepare_run(
pipeline=pipeline,
inputs=inputs,
Expand All @@ -206,7 +223,8 @@ def run_map(
chunksizes=chunksizes,
storage=storage,
cleanup=cleanup,
reuse_validation=reuse_validation,
resume=resume,
resume_validation=resume_validation,
fixed_indices=fixed_indices,
auto_subpipeline=auto_subpipeline,
show_progress=show_progress,
Expand Down Expand Up @@ -364,8 +382,9 @@ def run_map_async(
chunksizes: int | dict[OUTPUT_TYPE, int | Callable[[int], int] | None] | None = None,
storage: StorageType | None = None,
persist_memory: bool = True,
cleanup: bool = True,
reuse_validation: Literal["auto", "strict", "skip"] = "auto",
cleanup: bool | None = None,
resume: bool = False,
resume_validation: Literal["auto", "strict", "skip"] = "auto",
fixed_indices: dict[str, int | slice] | None = None,
auto_subpipeline: bool = False,
show_progress: bool | Literal["rich", "ipywidgets", "headless"] | None = None,
Expand Down Expand Up @@ -442,10 +461,23 @@ def run_map_async(
Whether to write results to disk when memory based storage is used.
Does not have any effect when file based storage is used.
cleanup
.. deprecated:: 0.89.0
Use `resume` parameter instead. Will be removed in version 1.0.0.

Whether to clean up the ``run_folder`` before running the pipeline.
reuse_validation
When set, takes priority over ``resume`` parameter.
``cleanup=True`` is equivalent to ``resume=False``.
``cleanup=False`` is equivalent to ``resume=True``.
resume
Whether to resume data from a previous run in the ``run_folder``.

- ``False`` (default): Clean up the ``run_folder`` before running (fresh start).
- ``True``: Attempt to load and resume results from a previous run.

Note: If ``cleanup`` is specified, it takes priority over this parameter.
resume_validation
Controls validation strictness when reusing data from a previous run
(only applies when ``cleanup=False``):
(only applies when ``resume=True``):

- ``"auto"`` (default): Validate that inputs/defaults match the previous run.
If equality comparison fails (returns ``None``), warn but proceed anyway.
Expand All @@ -456,7 +488,7 @@ def run_map_async(
You are responsible for ensuring inputs are actually identical.

Note: Shapes and MapSpecs are always validated regardless of this setting.
Ignored when ``cleanup=True``.
Ignored when ``resume=False``.
fixed_indices
A dictionary mapping axes names to indices that should be fixed for the run.
If not provided, all indices are iterated over.
Expand Down Expand Up @@ -491,6 +523,8 @@ def run_map_async(
`start()` method on the `AsyncMap` instance is called.

"""
resume = _handle_cleanup_deprecation(cleanup, resume, stacklevel=2)

prep = prepare_run(
pipeline=pipeline,
inputs=inputs,
Expand All @@ -502,7 +536,8 @@ def run_map_async(
chunksizes=chunksizes,
storage=storage,
cleanup=cleanup,
reuse_validation=reuse_validation,
resume=resume,
resume_validation=resume_validation,
fixed_indices=fixed_indices,
auto_subpipeline=auto_subpipeline,
show_progress=show_progress,
Expand Down
Loading
Loading

Back | FazBrowse Home | New Git URL