Add climada.util.interpolation.sparse_local_exceedance and sparse_local_frequency, which compute the exceedance curve of every matrix column from that column's nonzero entries, instead of calling preprocess_and_interpolate_ev once per column on a densified CSR column.
Call them from Impact.local_exceedance_impact, Impact.local_return_period, Hazard.local_exceedance_intensity and Hazard.local_return_period. supports_sparse_fast_path gates the switch; every other configuration keeps the existing code path unchanged.
getcol on a CSR matrix scans the whole index array, so calling it once per column made these methods O(n_columns × nnz).
Measured on a real US tropical cyclone dataset (LitPop USA 150 arcsec x historical TC hazard USA, both from the data API; 3,890 events x 643,099 exposure points) with pr_benchmark_exceedance.py, which times both paths in one process and compares them:
old new
Impact.local_exceedance_impact 62700 ms 869 ms 72x
Impact.local_return_period 63241 ms 882 ms 72x
Hazard.local_exceedance_intensity 109259 ms 1173 ms 93x
Hazard.local_return_period 109263 ms 1210 ms 90x
In a full risk assessment on that data the step goes from 83.6% of the run to 6.6%, and the run from 74.7 s to 13.1 s.
The script compares every pair with np.testing.assert_array_equal: no differences.
One thing worth flagging: the zeros can be dropped because they cannot influence a retained point. Sorted ascending, the zeros come first, and the cumulative frequency is a reverse cumulative sum, so the entry for the p-th largest value only ever sums the p largest frequencies. value_threshold=0 then discards exactly the entries the zeros sat in. This is why the helpers read data[start:end] rather than densifying; the reasoning depends on the threshold being 0, which is one of the conditions in supports_sparse_fast_path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes proposed in this PR:
getcol on a CSR matrix scans the whole index array, so calling it once per column made these methods O(n_columns × nnz).
Measured on a real US tropical cyclone dataset (LitPop USA 150 arcsec x historical TC hazard USA, both from the data API; 3,890 events x 643,099 exposure points) with pr_benchmark_exceedance.py, which times both paths in one process and compares them:
In a full risk assessment on that data the step goes from 83.6% of the run to 6.6%, and the run from 74.7 s to 13.1 s.
The script compares every pair with np.testing.assert_array_equal: no differences.
One thing worth flagging: the zeros can be dropped because they cannot influence a retained point. Sorted ascending, the zeros come first, and the cumulative frequency is a reverse cumulative sum, so the entry for the p-th largest value only ever sums the p largest frequencies. value_threshold=0 then discards exactly the entries the zeros sat in. This is why the helpers read data[start:end] rather than densifying; the reasoning depends on the threshold being 0, which is one of the conditions in supports_sparse_fast_path.
PR Author Checklist
PR Reviewer Checklist