| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
In a build of NumPy against libcxx, we found the following code
regressed from almost instantaneous with NumPy 2.3 to over 40s with
NumPy 2.4.
```
import numpy as np
full_list = np.array("2015-12-01T00:00:00.000000000", 'datetime64[ns]') + np.arange(162143) * np.timedelta64(4, 'h').astype('timedelta64[ns]')
sampled_dates = full_list[10:28]
np.isin(sampled_dates, full_list)
```
Our belief is the following:
* std::unordered_set in libcxx uses power of two hash buckets.
* std::hash is the identity function.
* in this particular example, we are hashing integers (datetime64
values) separated by multiples of a power of two.
* the net result is that all of the integers end up in the same hash
bucket.
We can make the code more robust simply by using the same npy_fnv1a hash
used elsewhere in the same file since it will do a better job of
distributing hash bits.
Sorry, something went wrong.
|
Nice catch! Using npy_fnv1a seems like a solid fix. |
Sorry, something went wrong.
Agreed, thanks for the detailed analysis and fix. I think it was probably an oversight not to use fnv1a in the first place. |
Sorry, something went wrong.
BUG: Fix weak hash function in np.isin(). (#30840)
| Back | FazBrowse Home | New Git URL |
In a build of NumPy against libcxx, we found the following code regressed from almost instantaneous with NumPy 2.3 to over 40s with NumPy 2.4.
import numpy as np full_list = np.array("2015-12-01T00:00:00.000000000", 'datetime64[ns]') + np.arange(162143) * np.timedelta64(4, 'h').astype('timedelta64[ns]') sampled_dates = full_list[10:28] np.isin(sampled_dates, full_list)Our belief is the following:
We can make the code more robust simply by using the same npy_fnv1a hash used elsewhere in the same file since it will do a better job of distributing hash bits.