| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
|
I can't speak to the meson changes, or the default option, but I will to the CI job. |
Sorry, something went wrong.
|
This should not be needed. The official wheels use a modified OpenBLAS from scipy-openblas that mangles the exported function names, there should be no overlap with the system OpenBLAS. This is done by using the --with-scipy-openblas -- -Dallow-noblas=false -Dpkg_config_path=${PWD}/.openblas arguments to spin build.
Please submit an issue documenting the problem, preferably with a reproducer, so we can try to figure out which symbols are not being mangled. Be sure to specify how you installed NumPy. |
Sorry, something went wrong.
|
The reproducer in the R issue is LD_PRELOAD=/usr/lib/libopenblas.so.3 venv/bin/python -c 'import numpy; print(numpy)' I tried the equivalent on Ubuntu24.04 x86_64 (glibc) and it did not segfault $ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libopenblas.so.0 python -c 'import numpy; print(numpy.__version__)' 2.3.5 |
Sorry, something went wrong.
|
OK, I can reproduce in a docker with musllinux $ docker run --rm -it quay.io/pypa/musllinux_1_2_x86_64 /bin/bash
...
# /opt/python/cp312-cp312/bin/python -m venv /tmp/venv312
# source /tmp/venv312/bin/activate
# pip install numpy
# apk add openblas gdb # Edit: libopenblas -> openblas
# LD_PRELOAD=/usr/lib/libopenblas.so.3 gdb --args python -c 'import numpy; print(numpy)'
...
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x0000721ce6703f69 in scipy_cblas_sdot64_ () from \
/tmp/venv312/lib/python3.12/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-55393b74.so
(gdb) bt
#0 0x0000721ce6703f69 in scipy_cblas_sdot64_ () from /tmp/venv312/lib/python3.12/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-55393b74.so
#1 0x0000721ce7ec5bbd in FLOAT_dot () from /tmp/venv312/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-musl.so
#2 0x0000721ce811ea5a in cblas_matrixproduct () from /tmp/venv312/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-musl.so
#3 0x0000721ce805ef4a in PyArray_MatrixProduct2 () from /tmp/venv312/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-musl.so
#4 0x0000721ce8050c0c in array_dot () from /tmp/venv312/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-musl.so
I wonder how this differs from glibc? |
Sorry, something went wrong.
|
This seems like another case of #21643. I wonder why compilation for alpine is different. |
Sorry, something went wrong.
|
When I run under ltrace, I see a call to sdot_k_HASWELL, which is not being mangled. On glibc the call goes to the numpy openblas, even with LD_PRELOAD. $ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libopenblas.so.0 ltrace -x "*sdot*" -L python -c 'import numpy; print(numpy.__version__)' scipy_cblas_sdot64_@libscipy_openblas64_-fdde5778.so(2, 0x392344f0, 1, 0x392344f0 <unfinished ...> sdot_k_HASWELL@libscipy_openblas64_-fdde5778.so(2, 0x392344f0, 1, 0x392344f0) = 0x787724fc7d40 <... scipy_cblas_sdot64_ resumed> ) = 0x787724fc7d40 2.3.5 But on alpine linux it does seem to go to the preloaded libopenblas instead. |
Sorry, something went wrong.
|
Another difference I see, using ltrace -x "*openblas*" -L <program> is that even when <program> does not use openblas, i.e. LD_PRELOAD... ltrace -x "*openblas*" -L ls /tmp/abc, # LD_PRELOAD=/usr/lib/libopenblas.so.3 ltrace -x "@*openblas*" -L ls /tmp/noexistant _init@libopenblas.so.3(0x7ed358475860, 128, 0x7ed3564f7000, 0) = 0 gotoblas_init@libopenblas.so.3(0x7ed3583b3648, 0, 3, 0x7ed3583b36a0 <unfinished ...> openblas_fork_handler@libopenblas.so.3(0x7ed3583b3648, 0, 3, 0) = 0 openblas_read_env@libopenblas.so.3(0x7ed35847600c, 0x7ed3583d0c1c, 4, 0) = 0 gotoblas_dynamic_init@libopenblas.so.3(0, 0, 11, 1 <unfinished ...> support_avx@libopenblas.so.3(7, 8, 0x7ed3581ab510, 0x444d4163) = 1 gotoblas_corename@libopenblas.so.3(7, 0, 0x7ed3583adc40, 0x7ed3583adc40) = 0x7ed35819bd64 openblas_warning@libopenblas.so.3(2, 0x7ffc2dd59e50, 0xffffffff, 0 <unfinished ...> openblas_verbose@libopenblas.so.3(2, 0x7ffc2dd59e50, 0xffffffff, 0) = 0 <... openblas_warning resumed> ) = 0 <... gotoblas_dynamic_init resumed> ) = 0 blas_get_cpu_number@libopenblas.so.3(0x3fff, 0x7f00000, 0x7ef0, 0 <unfinished ...> get_num_procs@libopenblas.so.3(0x3fff, 0x7f00000, 0x7ef0, 0) = 24 openblas_num_threads_env@libopenblas.so.3(0x7ffc2dd59dc0, 0x7ffc2dd59dc0, 2, 127) = 0 openblas_goto_num_threads_env@libopenblas.so.3(0x7ffc2dd59dc0, 0x7ffc2dd59dc0, 2, 127) = 0 openblas_omp_num_threads_env@libopenblas.so.3(0x7ffc2dd59dc0, 0x7ffc2dd59dc0, 2, 127) = 0 <... blas_get_cpu_number resumed> ) = 24 blas_thread_init@libopenblas.so.3(0x7ffc2dd59dc0, 0x7ffc2dd59dc0, 2, 127 <unfinished ...> blas_memory_alloc@libopenblas.so.3(2, 0x7ffc2dd59dc0, 24, 127 <unfinished ...> gotoblas_dynamic_init@libopenblas.so.3(0x7ed3583be340, 0x7ffc2dd59dc0, 16, 127) = 0 <... blas_memory_alloc resumed> ) = 0x7ed34e000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed346000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed33e000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed336000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed32e000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed326000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed31e000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed316000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed30e000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed306000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2fe000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2f6000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2ee000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2e6000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2de000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2d6000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2ce000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2c6000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2be000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2b6000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2ae000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed2a6000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed29e000000 blas_memory_alloc@libopenblas.so.3(2, 128, 0, 0x7ed358426f77) = 0x7ed296000000 openblas_thread_timeout@libopenblas.so.3(0, 128, 0x7ed3583c9dc8, 0x7ed358426f77) = 0 <... blas_thread_init resumed> ) = 0 <... gotoblas_init resumed> ) = 3 ls: /tmp/noexistant: No such file or directory gotoblas_quit@libopenblas.so.3(0x5b939d59f1b0, 1, 16, 0 <unfinished ...> blas_shutdown@libopenblas.so.3(0x5b939d59f1b0, 1, 16, 0 <unfinished ...> blas_thread_shutdown_@libopenblas.so.3(0x5b939d59f1b0, 1, 16, 0 <unfinished ...> blas_memory_free@libopenblas.so.3(0x7ed34e000000, 1, 16, 0) = 0 blas_memory_free@libopenblas.so.3(0x7ed346000000, 128, 0x7ed3583b6320, 0x7ed34e000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed33e000000, 128, 0x7ed3583b6320, 0x7ed346000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed336000000, 128, 0x7ed3583b6320, 0x7ed33e000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed32e000000, 128, 0x7ed3583b6320, 0x7ed336000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed326000000, 128, 0x7ed3583b6320, 0x7ed32e000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed31e000000, 128, 0x7ed3583b6320, 0x7ed326000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed316000000, 128, 0x7ed3583b6320, 0x7ed31e000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed30e000000, 128, 0x7ed3583b6320, 0x7ed316000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed306000000, 128, 0x7ed3583b6320, 0x7ed30e000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2fe000000, 128, 0x7ed3583b6320, 0x7ed306000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2f6000000, 128, 0x7ed3583b6320, 0x7ed2fe000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2ee000000, 128, 0x7ed3583b6320, 0x7ed2f6000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2e6000000, 128, 0x7ed3583b6320, 0x7ed2ee000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2de000000, 128, 0x7ed3583b6320, 0x7ed2e6000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2d6000000, 128, 0x7ed3583b6320, 0x7ed2de000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2ce000000, 128, 0x7ed3583b6320, 0x7ed2d6000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2c6000000, 128, 0x7ed3583b6320, 0x7ed2ce000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2be000000, 128, 0x7ed3583b6320, 0x7ed2c6000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2b6000000, 128, 0x7ed3583b6320, 0x7ed2be000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2ae000000, 128, 0x7ed3583b6320, 0x7ed2b6000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed2a6000000, 128, 0x7ed3583b6320, 0x7ed2ae000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed29e000000, 128, 0x7ed3583b6320, 0x7ed2a6000000) = 0 blas_memory_free@libopenblas.so.3(0x7ed296000000, 128, 0x7ed3583b6320, 0x7ed29e000000) = 0 <... blas_thread_shutdown_ resumed> ) = 0 <... blas_shutdown resumed> ) = 0 gotoblas_dynamic_quit@libopenblas.so.3(0x7ed3583be340, 128, 512, 0x7ed3583b6320) = 0x7ed3583b62f0 <... gotoblas_quit resumed> ) = 0x7ed3583b62f0 _fini@libopenblas.so.3(0x7ed3560da9e0, 3, 0x7ed35819a855, 0x7ed3560daa38) = 0 +++ exited (status 1) +++ On glibc the _init is not called, but the cleanup functions are called tracing on glibc$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libopenblas.so.0 ltrace -x "@*openblas*" -L ls /tmp/noexistant ls: cannot access '/tmp/noexistant': No such file or directory gotoblas_quit@libopenblas.so.0(0x729250803170, 1, 1, 1 <unfinished ...> blas_shutdown@libopenblas.so.0(0x729250803170, 1, 1, 1 <unfinished ...> blas_thread_shutdown_@libopenblas.so.0(0x729250803170, 1, 1, 1) = 0 <... blas_shutdown resumed> ) = 0 gotoblas_dynamic_quit@libopenblas.so.0(0x7292507f3180, 0, 0, 0x7292507f1160) = 0x7292507f1130 <... gotoblas_quit resumed> ) = 0 +++ exited (status 2) +++ |
Sorry, something went wrong.
|
@rgommers pointed out (in private communication) that this has to do with the RTLD_* flags: >>> import ctypes, sys, os
>>> openblas = ctypes.CDLL("/usr/lib/libopenblas.so.3", os.RTLD_LOCAL)
>>> import numpy
# Success
vs. >>> import ctypes, sys, os
>>> openblas = ctypes.CDLL("/usr/lib/libopenblas.so.3", os.RTLD_GLOBAL)
>>> import numpy
Segmentation fault (core dumped)
On glibc both of these succeed. Symbols like sdot_k_HASWELL are exported and visible similarly from both the glibc and musl shared object, even though we use the -fvisibility=protected flag when compiling. Even more worrisome, when using ltrace, the blas_init functions are called from the libopenblas shared object and not the libscipy_openbas one, so all the buffers and threads are set up in the wrong shared object. |
Sorry, something went wrong.
|
This is a bit of a mess in OpenBLAS unfortunately. There was one aborted attempt at cleaning this up in OpenMathLib/OpenBLAS#3658. The last comment there seems on point to me. The other way to deal with this is with a linker script when building scipy-openblas. Passing -Wl,--version-script=openblas.version to GCC, with openblas.version something like: OPENBLAS {
global:
/* BLAS Level 1-3 functions */
s*_; d*_; c*_; z*_;
/* CBLAS interface */
cblas_*;
/* LAPACK functions */
*gesv_; *getrf_; *potrf_; *heev_; *syev_; *geqrf_; *gelqf_;
*gesdd_; *geev_; *posv_; *ppsv_; *pbsv_; *gbsv_; *sysv_; *hesv_;
...
/* OpenBLAS public configuration API only */
openblas_get_num_threads;
openblas_set_num_threads;
openblas_get_num_procs;
openblas_get_parallel;
openblas_get_config;
openblas_get_corename;
openblas_get_device_config;
local:
*;
};
With some tweaking of that and adding scipy_* and *_64_ it may be possible to fix this in a maintainable and low-effort way. |
Sorry, something went wrong.
|
I played with versioning but always get the preloaded symbols like gotoblas_init and blas_thread_init@libopenblas.so.3. Even when I version gotoblas_init, when called from the _init of libscipyopenblas the "wrong" one is called. I wonder if statically linking to libscipyopenblas.a (for our wheels, not for conda) will change anything |
Sorry, something went wrong.
|
Static linking isn't worth trying I think, it isn't well-supported by OpenBLAS and my previous attempt at statically linking libopenblas.a gave me weird issues and I gave up. |
Sorry, something went wrong.
|
yeah, I see. When the DLL is opened void gotoblas_init(void) is called to set stuff up. It would have to be called manually somehow. |
Sorry, something went wrong.
|
Let's go back to the start. @t-kalinowski could you expand on
How exactly did the process load the system OpenBLAS? Is there a way to change that to use the equivalent of ctypes.CDLL("/usr/lib/libopenblas.so.3", os.RTLD_LOCAL)? |
Sorry, something went wrong.
I don’t think so in the Alpine + R + reticulate case. Alpine’s R is built/linked against the system OpenBLAS, so libopenblas is already loaded by the time any R code runs and reticulate embeds Python. The RTLD_LOCAL/RTLD_GLOBAL distinction only applies when a library is loaded via dlopen(), not for a link-time dependency that’s present from process start.
So there isn’t a supported knob in R to “re-open” BLAS with RTLD_LOCAL. Avoiding the conflict would require that the globally loaded system OpenBLAS cannot take precedence for BLAS symbols used by NumPy. In practice that means either building R (or the host program) without linking to the system OpenBLAS, or modifying the NumPy/OpenBLAS build so that its BLAS symbols remain isolated from any globally loaded OpenBLAS. |
Sorry, something went wrong.
|
RTLD_LOCAL does not exist at all on musl-based distros. I don't think there are any other solutions possible here aside from ensuring scipy_openblas doesn't leak private symbols (with either a version script or by properly hiding symbols in OpenBLAS itself). This PR also doesn't do much to help - it may happen to work if R and NumPy both load the same version of LP64 OpenBLAS, but when they're using different versions and the public functions in one library end up calling routines from the other loaded libopenblas you can get weird bugs. |
Sorry, something went wrong.
|
I'm inclined to say we should aim for the proper solution (OpenMathLib/OpenBLAS#3658 (comment)), since that is just the right thing to do and in the end "just" plumbing. I will open a new OpenBLAS issue for this now. |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Hello,
We recently hit a segfault on Alpine when importing NumPy in a process that had already loaded the system OpenBLAS. After some investigation, it seems Alpine’s system OpenBLAS is LP64 (32-bit BLAS ints) while the musllinux wheels default to ILP64 (64-bit BLAS ints), and mixing the two in one process can crash (seen via a reticulate workflow, rstudio/reticulate#1858 (comment)). From what I can tell, most distro OpenBLAS packages (including Alpine) and MKL defaults are LP64, while manylinux wheels use ILP64 for large-array support, so this change tries to align musl builds with the system default, but please let me know if I’m missing a better precedent.
What changed
Testing
Happy to adjust anything. Thanks!