Well that is smoking out a bunch of issues ...

I bet there's a whole bunch of projects building NumPy in CI without BLAS support, either by accident or because it doesn't matter to them. I think we need to have this default though, because it affects the pip install numpy from source and most users are going to want BLAS/LAPACK support.

LGTM. Just for confirmation

• 64 bit BLAS is not required unless specified.
• 32 bit OpenBLAS is not working?

64 bit BLAS is not required unless specified.

Indeed, that's how it has always been - you get 32-bit by default, and have to opt in to ILP64.

32 bit OpenBLAS is not working?

It's working fine and is used in a bunch of places. Only the PyPy build is changed here, it was silently failing to find OpenBLAS ILP64 before (so passing with lapack-lite), and when I fixed the detection logic it started failing with a "can't import _multiarray_umath error". So that needs investigating, but on a Windows machine - debugging that via Azure isn't really possible. Cc @mattip for visibility.

it started failing with a "can't import _multiarray_umath error". So that needs investigating

I am not sure what you meant by "it" in the context of PyPy failing. It seems all the CI is passing here?

@mattip I had to add DISABLE_BLAS and -Dallow-noblas to the PyPy job to make it pass. There never was a working CI job that linked against openblas64_, and the "fail instead of silently proceeding" change in this PR smoked that out.

Let's get this in. Thanks Ralf.

@charris I think we can backport all this to 1.26.x, but set the default for allow-noblas to true. That way it keeps the code between the two branches in sync (which is useful), and allows using the flag to test whether our wheel builds and CI jobs indeed pick up OpenBLAS. While at the same time not changes the behavior for 1.26.0.

This seems already done in 1.26 except main isn't building pp39.

This seems already done in 1.26 except main isn't building pp39.

Except the default is still false there, which is fine for now, but as I commented here, we should decide before the final release whether that is what we'd like to do.

Except the default is still false there, which is fine for now

I think we should allow building without OpenBLAS by default on 1.26. That has been the policy up to now. The time to change it is in the move to 2.0, not in a 1.26 interim release which is mainly meant to enable building with meson for python 3.12.

@mattip if it were only about the policy and there were no side effects, it would be a simple decision indeed and I'd agree we should only consider it for 2.0. Why I am hesitant (either way) is that the BLAS/LAPACK detection mechanisms changed. E.g. someone who had a ~/site.cfg file pointing to their library of choice will now, if the default is true, silently get a slow lapack-lite build. I'd expect even Linux distro packagers may get this wrong. E.g., they have OpenBLAS or ATLAS in the right place, but no pkg-config installed when they build a numpy 1.26.0 package. Usually they just look at their build automation that tells them a new release was uploaded to PyPI, trigger a build, and call it good if it builds and the test suite passes.

So the trade-off we deal with here is:

• allow-noblas=true: likely silent performance regressions for a lot of users
• allow-noblas=false: likely a lot more build failures early on, and making people who don't care about performance explicitly add -C-Dallow-noblas=true to their build/install command.

Given the above, would you still prefer allow-noblas=true?

Hmm. Crazy thought: could we port just enough of the system_info discovery system to give a clear error before using lapack-lite if

• meson does not find a blas/lapack
• the pre-meson discovery techniques would have found a blas/lapack (i.e. setup.cfg is a valid file, or the NPY_BLAS_LIBS/NPY_LAPACK_LIBS/NPY_CBLAS_LIBS env variables are set)

My first thought was "even more work, don't really wanna". Second thought: "yeah, I think this is actually feasible and may be appreciated by our users".

There's two ways of doing that:

1. stripping down system_info.py and running it as a separate script called from numpy/meson.build through a run_command call,
2. reimplementing a bunch of the logic here in Meson's DSL

Neither is going to be 100% complete without a lot of effort, but it's doable to catch most of the common cases with maybe half a day of work.

The trickiest situation is also the most common one: when the user did nothing special like setting an env var or creating a site.cfg file, but some library would have been auto-detected in one of the hardcoded paths in system_info.py. That case is easier to handle with (1), so I think I'd go for that approach.

I'd be happy just to be able to continue using something like .numpy-site.cfg. I read the Redhat thread on the missing pkgconf files for OpenBLAS, and I think they had a point about the difficulty in handling all the different OpenBLAS options, not to mention other BLAS versions. Which is to say, automatic detection may not be sufficient. IIRC, there was also a builtin priority in the way that was done before. Having a simple way to just specify which library to use would be helpful, along with platform specific documentation.

All this makes me appreciate Pearu's work way back in the beginning.

@charris you can use a custom file to specify the location, compile flags, etc. of BLAS/LAPACK. It has changed from a numpy-specific and ad-hoc format to a pkg-config file, as documented at http://scipy.github.io/devdocs/building/blas_lapack.html#using-pkg-config-to-detect-libraries-in-a-nonstandard-location.

IIRC, there was also a builtin priority in the way that was done before.

Yep, that part is still in the works (also for SciPy). I should have that done by the end of September; recreating all of system_info.py has turned out to be a little more challenging than expected.

All this makes me appreciate Pearu's work way back in the beginning.

definitely!

you can use a custom file to specify the location, compile flags

I ended up downloading a tarball from the OpenBLAS nightlies and installing it in /usr. Note that meson does not recognize PKG_CONFIG_PATH, so the install locations are limited to the defaults.

We could upgrade tools/openblas_support.py to make such an install, in Fedora 38 it currently installs in /var/tmp. That module could also use more documentation.

I ended up downloading a tarball from the OpenBLAS nightlies and installing it in /usr. Note that meson does not recognize PKG_CONFIG_PATH, so the install locations are limited to the defaults.

That's not how that works. You can put a .pc file anywhere, and then the PKG_CONFIG_PATH environment variable should be set to point to the location of the .pc file. The include and library directories are given inside the .pc files, and those can be any absolute path.

We could upgrade tools/openblas_support.py to make such an install, in Fedora 38 it currently installs in /var/tmp. That module could also use more documentation.

We will get rid of that file as soon as possible, so please don't spend more than the minimum needed effort on it. The plan is to change our custom OpenBLAS build from a .tar.gz to a wheel, and at that point the install command is pip install openblas and the install locations are then inside or relative to site-packages.