libnpymath and libnpyrandom #17498Description
We should get this done for 1.11.0. Using static libraries from numpy is unhealthy. Let me summarize the main problems and link some related issues.
From #17481 (comment):
The main issue is using a static library at all. This is bad, mainly for these two reasons:
- It is not guaranteed that
numpyandscipyare built with the same compilers. And that leads to potential issues that are hard to predict, as we've experienced multiple times already. Especially on Windows, where we build scipy with mingw and numpy with MSVC. That was the reason why using thevc142instead of thevc141toolchain in numpy 1.21.5-1.22.1 led to complex issues and that change had to be reverted. - Cross-compiling tends to break. We haven't officially supported that in the past, but even in the one case where we do it (macOS x64-64 to arm64) it goes pretty horribly wrong: we stuff the x86-64
libnpymath.ainto the macOS scipy wheel. This silently worked, because the required symbols were already loaded by numpy itself. Now that they're being removed fromnumpy, we're all of a sudden loading them fromlibnpymath.aand they have the wrong architecture.
For more details, see numpy/numpy#20880.
Here are some related issues where the use of libnpmath showed up as a problem:
- Build problem on macOS with numpy 1.19.4 #13323 (contains
libnpymath.a(npy_math.o), could not parse object file) - Cross-compiling issue: BUG: meson fails to properly detect numpy, pybind11 and (sort of) pythran when cross compiling #16783
- Another cross-compiling issue: Tracking issue for cross-compilation needs and issues #14812 (comment)
- Regression because scipy 1.8/1.9 wheels contain x86-64 instructions in macOS arm64 wheel: BUG:
scipy.specialimport broken with numpy 1.24.0rc1 numpy/numpy#22673
What we should do is described in numpy/numpy#20880 (comment). Basically, vendor the sources for the functions/typedefs we still need, and rebuild them as part of the SciPy build. This is probably best done by refactoring the numpy sources a bit so there's a standalone C99 compat library with only the complex typedefs, complex functions from C99 that MSVC doesn't provide, and the blocklisted functions. And then add a command-line switch to the numpy meson build to only build that library (use a meson disabler object for that). And include numpy as a meson subproject in the scipy source tree - this is kinda like a git submodule - so we can build it as needed, and don't actually need to vendor anything.
I don't have the bandwidth to push this forward now, but should be able to do so in Q1'23.