FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Backport PR #2629: feat: Store dask arrays matching dask chunk size by ilan-gold · Pull Request #2669 · scverse/anndata · GitHub

  1. scverse
  2. anndata

Repository navigation

Backport PR #2629: feat: Store dask arrays matching dask chunk size - #2669

Open
ilan-gold wants to merge 3 commits into
0.13.xfrom
auto-backport-of-pr-2629-on-0.13.x
ConversationCommits (3)ChecksFiles changed
Open

Backport PR #2629: feat: Store dask arrays matching dask chunk size#2669
ilan-gold wants to merge 3 commits into
0.13.xfrom
auto-backport-of-pr-2629-on-0.13.x

Conversation

ilan-gold commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

@joshua-gould I just noticed this also disables sharing for on-disk concatenation, which despite being a feature I dislike, does exist. I'm curious, before this gets merged into a release line, what your experience has been with storing/reading the data. Generally, lots of large files create headaches. Are you not seeing this?

  • Closes #
  • Tests added
  • Release note not necessary because:

codecov Bot commented Sep 30, 2026 •
edited
Loading

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
7363 2 7361 2211
View the top 2 failed test(s) by shortest run time 
tests.test_io_utils::test_write_chunk_size[h5ad]
Stack Traces | 0.021s run time 
tmp_path = PosixPath('.../pytest-0/popen-gw1/test_write_chunk_size_h5ad_0')
output_format = 'h5ad'

    #x1B[0m#x1B[37m@pytest#x1B[39;49;00m.mark.parametrize(#x1B[33m"#x1B[39;49;00m#x1B[33moutput_format#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, [#x1B[33m"#x1B[39;49;00m#x1B[33mzarr#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33mh5ad#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m])#x1B[90m#x1B[39;49;00m
    #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_write_chunk_size#x1B[39;49;00m(tmp_path, output_format):#x1B[90m#x1B[39;49;00m
        pth = tmp_path / #x1B[33mf#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[33mtest.#x1B[39;49;00m#x1B[33m{#x1B[39;49;00moutput_format#x1B[33m}#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        adata = ad.AnnData(X=da.arange(#x1B[94m36#x1B[39;49;00m).reshape((#x1B[94m6#x1B[39;49;00m, #x1B[94m6#x1B[39;49;00m)).rechunk((#x1B[94m2#x1B[39;49;00m, #x1B[94m2#x1B[39;49;00m)))#x1B[90m#x1B[39;49;00m
        #x1B[94mif#x1B[39;49;00m output_format == #x1B[33m"#x1B[39;49;00m#x1B[33mzarr#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
            adata.write_zarr(pth)#x1B[90m#x1B[39;49;00m
            store = zarr.open(pth, mode=#x1B[33m"#x1B[39;49;00m#x1B[33mr#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            #x1B[94massert#x1B[39;49;00m store[#x1B[33m"#x1B[39;49;00m#x1B[33mX#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m].chunks == adata.X.chunksize#x1B[90m#x1B[39;49;00m
            np.testing.assert_array_equal(store[#x1B[33m"#x1B[39;49;00m#x1B[33mX#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m], adata.X)#x1B[90m#x1B[39;49;00m
        #x1B[94melse#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
            adata.write_h5ad(pth)#x1B[90m#x1B[39;49;00m
            #x1B[94mwith#x1B[39;49;00m h5py.File(pth, mode=#x1B[33m"#x1B[39;49;00m#x1B[33mr#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m) #x1B[94mas#x1B[39;49;00m store:#x1B[90m#x1B[39;49;00m
>               #x1B[94massert#x1B[39;49;00m store[#x1B[33m"#x1B[39;49;00m#x1B[33mX#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m].chunks == adata.X.chunksize#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE               assert None == (2, 2)#x1B[0m
#x1B[1m#x1B[31mE                +  where None = <HDF5 dataset "X": shape (6, 6), type "<i8">.chunks#x1B[0m
#x1B[1m#x1B[31mE                +  and   (2, 2) = dask.array<rechunk-merge, shape=(6, 6), dtype=int64, chunksize=(2, 2), chunktype=numpy.ndarray>.chunksize#x1B[0m
#x1B[1m#x1B[31mE                +    where dask.array<rechunk-merge, shape=(6, 6), dtype=int64, chunksize=(2, 2), chunktype=numpy.ndarray> = AnnData object with n_obs × n_vars = 6 × 6\n    layers: None (.X).X#x1B[0m

#x1B[1m#x1B[31mtests/test_io_utils.py#x1B[0m:247: AssertionError
tests.test_inplace_subset::test_inplace_subset_var[scipy_csc_array-slice_int_subset]
Stack Traces | 0.071s run time 
self = NumpyBlock: slice(3, 4, 1), 1 x 10, dtype: float64, deep = True

    #x1B[0m#x1B[37m@final#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mcopy#x1B[39;49;00m(#x1B[96mself#x1B[39;49;00m, *, deep: #x1B[96mbool#x1B[39;49;00m) -> Self:#x1B[90m#x1B[39;49;00m
    #x1B[90m    #x1B[39;49;00m#x1B[33m"""copy constructor"""#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        values = #x1B[96mself#x1B[39;49;00m.values#x1B[90m#x1B[39;49;00m
        refs: BlockValuesRefs | #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[94mif#x1B[39;49;00m deep:#x1B[90m#x1B[39;49;00m
            values = values.copy()#x1B[90m#x1B[39;49;00m
            refs = #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[94melse#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
            values = values.view()#x1B[90m#x1B[39;49;00m
            refs = #x1B[96mself#x1B[39;49;00m.refs#x1B[90m#x1B[39;49;00m
>       #x1B[94mreturn#x1B[39;49;00m #x1B[96mtype#x1B[39;49;00m(#x1B[96mself#x1B[39;49;00m)(values, placement=#x1B[96mself#x1B[39;49;00m._mgr_locs, ndim=#x1B[96mself#x1B[39;49;00m.ndim, refs=refs)#x1B[90m#x1B[39;49;00m
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE       ResourceWarning: unclosed file <_io.BufferedReader name='.../data/archives/v0.7.8/adata.zarr.zip'>#x1B[0m

#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14.../core/internals/blocks.py#x1B[0m:650: ResourceWarning

#x1B[33mThe above exception was the direct cause of the following exception:#x1B[0m

cls = <class '_pytest.runner.CallInfo'>
func = <function call_and_report.<locals>.<lambda> at 0x7f9758ece6c0>
when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    #x1B[0m#x1B[37m@classmethod#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mfrom_call#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m
        #x1B[96mcls#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
        func: Callable[[], TResult],#x1B[90m#x1B[39;49;00m
        when: Literal[#x1B[33m"#x1B[39;49;00m#x1B[33mcollect#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33msetup#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33mcall#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, #x1B[33m"#x1B[39;49;00m#x1B[33mteardown#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m],#x1B[90m#x1B[39;49;00m
        reraise: #x1B[96mtype#x1B[39;49;00m[#x1B[96mBaseException#x1B[39;49;00m] | #x1B[96mtuple#x1B[39;49;00m[#x1B[96mtype#x1B[39;49;00m[#x1B[96mBaseException#x1B[39;49;00m], ...] | #x1B[94mNone#x1B[39;49;00m = #x1B[94mNone#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
    ) -> CallInfo[TResult]:#x1B[90m#x1B[39;49;00m
    #x1B[90m    #x1B[39;49;00m#x1B[33m"""Call func, wrapping the result in a CallInfo.#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    :param func:#x1B[39;49;00m
    #x1B[33m        The function to call. Called without arguments.#x1B[39;49;00m
    #x1B[33m    :type func: Callable[[], _pytest.runner.TResult]#x1B[39;49;00m
    #x1B[33m    :param when:#x1B[39;49;00m
    #x1B[33m        The phase in which the function is called.#x1B[39;49;00m
    #x1B[33m    :param reraise:#x1B[39;49;00m
    #x1B[33m        Exception or exceptions that shall propagate if raised by the#x1B[39;49;00m
    #x1B[33m        function, instead of being wrapped in the CallInfo.#x1B[39;49;00m
    #x1B[33m    """#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        excinfo = #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        instant = timing.Instant()#x1B[90m#x1B[39;49;00m
        #x1B[94mtry#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
>           result: TResult | #x1B[94mNone#x1B[39;49;00m = func()#x1B[90m#x1B[39;49;00m
                                     ^^^^^^#x1B[90m#x1B[39;49;00m

#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14....../site-packages/_pytest/runner.py#x1B[0m:361: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14....../site-packages/_pytest/runner.py#x1B[0m:250: in <lambda>
    #x1B[0m#x1B[94mlambda#x1B[39;49;00m: runtest_hook(item=item, **kwds),#x1B[90m#x1B[39;49;00m
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14.../site-packages/pluggy/_hooks.py#x1B[0m:512: in __call__
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._hookexec(#x1B[96mself#x1B[39;49;00m.name, #x1B[96mself#x1B[39;49;00m._hookimpls.copy(), kwargs, firstresult)#x1B[90m#x1B[39;49;00m
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14.../site-packages/pluggy/_manager.py#x1B[0m:120: in _hookexec
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._inner_hookexec(hook_name, methods, kwargs, firstresult)#x1B[90m#x1B[39;49;00m
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14.../site-packages/_pytest/logging.py#x1B[0m:865: in pytest_runtest_call
    #x1B[0m#x1B[94myield#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14.../site-packages/_pytest/capture.py#x1B[0m:900: in pytest_runtest_call
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m (#x1B[94myield#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            ^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14.../site-packages/_pytest/skipping.py#x1B[0m:268: in pytest_runtest_call
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m (#x1B[94myield#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
            ^^^^^#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14........./site-packages/_pytest/unraisableexception.py#x1B[0m:183: in pytest_runtest_call
    #x1B[0mcollect_unraisable(item.config)#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14........./site-packages/_pytest/unraisableexception.py#x1B[0m:79: in collect_unraisable
    #x1B[0m#x1B[94mraise#x1B[39;49;00m errors[#x1B[94m0#x1B[39;49;00m]#x1B[90m#x1B[39;49;00m
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

config = <_pytest.config.Config object at 0x7f97d5d75010>

    #x1B[0m#x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mcollect_unraisable#x1B[39;49;00m(config: Config) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
        pop_unraisable = config.stash[unraisable_exceptions].pop#x1B[90m#x1B[39;49;00m
        errors: #x1B[96mlist#x1B[39;49;00m[pytest.PytestUnraisableExceptionWarning | #x1B[96mRuntimeError#x1B[39;49;00m] = []#x1B[90m#x1B[39;49;00m
        meta = #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        hook_error = #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[94mtry#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
            #x1B[94mwhile#x1B[39;49;00m #x1B[94mTrue#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                #x1B[94mtry#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                    meta = pop_unraisable()#x1B[90m#x1B[39;49;00m
                #x1B[94mexcept#x1B[39;49;00m #x1B[96mIndexError#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                    #x1B[94mbreak#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
                #x1B[94mif#x1B[39;49;00m #x1B[96misinstance#x1B[39;49;00m(meta, #x1B[96mBaseException#x1B[39;49;00m):#x1B[90m#x1B[39;49;00m
                    hook_error = #x1B[96mRuntimeError#x1B[39;49;00m(#x1B[33m"#x1B[39;49;00m#x1B[33mFailed to process unraisable exception#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
                    hook_error.__cause__ = meta#x1B[90m#x1B[39;49;00m
                    errors.append(hook_error)#x1B[90m#x1B[39;49;00m
                    #x1B[94mcontinue#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
                msg = meta.msg#x1B[90m#x1B[39;49;00m
                #x1B[94mtry#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
>                   warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE                   pytest.PytestUnraisableExceptionWarning: Exception ignored while finalizing file <_io.FileIO name='.../data/archives/v0.7.8/adata.zarr.zip' mode='rb' closefd=True>: None#x1B[0m

#x1B[1m#x1B[31m../../../..../anndata/88DSKUwu/hatch-test.stable/lib/python3.14........./site-packages/_pytest/unraisableexception.py#x1B[0m:67: PytestUnraisableExceptionWarning

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

joshua-gould commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Reading lots of files can definitely cause headaches. However, auto-sharding with dask made my jobs extremely slow (I typically work with datasets of ~22 million observations by 1000 variables).

ilan-gold commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

But write-once (slow) and read-many (fast) seems a worthy tradeoff?

joshua-gould commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

But write-once (slow) and read-many (fast) seems a worthy tradeoff?

It depends. In my case, sharding was causing my jobs to take an inordinate amount of time and my subsequent steps are CPU and not I/O bound.

ilan-gold commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

If you can't write the array, then you can't use the array afterwards. That seems strong enough.

This branch has not been deployed

No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

Back | FazBrowse Home | New Git URL