Repository navigation
[Bug]: Meson build fails due to qhull link issue. #27159
Description
Bug summary
Building matplotlib fails due to "Connection reset by peer" when meson is building qhull. Perhaps relevant: qhull/qhull#132
Code for reproduction
python -m pip install -ve .Actual outcome
[Errno 104] Connection reset by peer~/github/matplotlib$ python -m pip install -ve .
Using pip 22.0.2 from /home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pip (python 3.10)
Obtaining file:///home/xenophon/github/matplotlib
Running command pip subprocess to install build dependencies
Collecting meson-python>=0.13.1
Using cached meson_python-0.14.0-py3-none-any.whl (76 kB)
Collecting numpy>=1.25
Using cached numpy-1.26.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
Collecting pybind11>=2.6
Using cached pybind11-2.11.1-py3-none-any.whl (227 kB)
Collecting setuptools_scm>=7
Using cached setuptools_scm-8.0.4-py3-none-any.whl (42 kB)
Collecting pyproject-metadata>=0.7.1
Using cached pyproject_metadata-0.7.1-py3-none-any.whl (7.4 kB)
Collecting tomli>=1.0.0
Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting meson>=0.63.3
Using cached meson-1.2.2-py3-none-any.whl (964 kB)
Collecting packaging>=20
Using cached packaging-23.2-py3-none-any.whl (53 kB)
Collecting typing-extensions
Using cached typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Collecting setuptools
Using cached setuptools-68.2.2-py3-none-any.whl (807 kB)
Installing collected packages: typing-extensions, tomli, setuptools, pybind11, packaging, numpy, meson, setuptools_scm, pyproject-metadata, meson-python
Successfully installed meson-1.2.2 meson-python-0.14.0 numpy-1.26.1 packaging-23.2 pybind11-2.11.1 pyproject-metadata-0.7.1 setuptools-68.2.2 setuptools_scm-8.0.4 tomli-2.0.1 typing-extensions-4.8.0
Installing build dependencies ... done
Running command Checking if build backend supports build_editable
Checking if build backend supports build_editable ... done
Running command Getting requirements to build editable
Getting requirements to build editable ... done
Running command pip subprocess to install backend dependencies
Collecting ninja>=1.8.2
Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
Collecting patchelf>=0.11.0
Using cached patchelf-0.17.2.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.musllinux_1_1_x86_64.whl (425 kB)
Installing collected packages: patchelf, ninja
Successfully installed ninja-1.11.1.1 patchelf-0.17.2.1
Installing backend dependencies ... done
Running command Preparing editable metadata (pyproject.toml)
+ meson setup /home/xenophon/github/matplotlib /home/xenophon/github/matplotlib/build/cp310 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/xenophon/github/matplotlib/build/cp310/meson-python-native-file.ini
The Meson build system
Version: 1.2.2
Source dir: /home/xenophon/github/matplotlib
Build dir: /home/xenophon/github/matplotlib/build/cp310
Build type: native build
Project name: matplotlib
Project version: 3.9.0.dev0
C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc ld.bfd 2.38
C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C++ linker for the host machine: c++ ld.bfd 2.38
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python found: YES (/home/xenophon/github/matplotlib/env/bin/python)
Did not find pkg-config by name 'pkg-config'
Found Pkg-config: NO
Run-time dependency python found: YES 3.10
pybind11-config found: YES (/tmp/pip-build-env-8wxg444r/overlay/bin/pybind11-config) 2.11.1
Run-time dependency pybind11 found: YES 2.11.1
Executing subproject freetype-2.6.1
freetype-2.6.1| Project name: freetype2
freetype-2.6.1| Project version: 2.6.1
freetype-2.6.1| C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
freetype-2.6.1| C linker for the host machine: cc ld.bfd 2.38
freetype-2.6.1| Has header "unistd.h" : YES
freetype-2.6.1| Has header "fcntl.h" : YES
freetype-2.6.1| Has header "stdint.h" : YES
freetype-2.6.1| Configuring ftconfig.h using configuration
freetype-2.6.1| Configuring ftoption.h using configuration
freetype-2.6.1| Build targets in project: 3
freetype-2.6.1| Subproject freetype-2.6.1 finished.
Downloading qhull source from http://www.qhull.org/download/qhull-2020-src-8.0.2.tgz
WARNING: failed to download with error: [Errno 104] Connection reset by peer. Trying after a delay...
WARNING: failed to download with error: [Errno 104] Connection reset by peer. Trying after a delay...
WARNING: failed to download with error: [Errno 104] Connection reset by peer. Trying after a delay...
WARNING: failed to download with error: [Errno 104] Connection reset by peer. Trying after a delay...
WARNING: failed to download with error: [Errno 104] Connection reset by peer. Trying after a delay...
Traceback (most recent call last):
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/mesonmain.py", line 194, in run
return options.run_func(options)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/msetup.py", line 358, in run
app.generate()
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/msetup.py", line 183, in generate
return self._generate(env, capture, vslite_ctx)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/msetup.py", line 228, in _generate
intr.run()
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreter/interpreter.py", line 3002, in run
super().run()
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 164, in run
self.evaluate_codeblock(self.ast, start=1)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 190, in evaluate_codeblock
raise e
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 182, in evaluate_codeblock
self.evaluate_statement(cur)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 196, in evaluate_statement
return self.function_call(cur)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 521, in function_call
res = func(node, func_args, kwargs)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/decorators.py", line 277, in wrapper
return f(*nargs, **wrapped_kwargs)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/decorators.py", line 596, in wrapper
return f(*wrapped_args, **wrapped_kwargs)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreter/interpreter.py", line 2414, in func_subdir
self.evaluate_codeblock(codeblock)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 190, in evaluate_codeblock
raise e
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 182, in evaluate_codeblock
self.evaluate_statement(cur)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 206, in evaluate_statement
return self.evaluate_if(cur)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 306, in evaluate_if
self.evaluate_codeblock(node.elseblock)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 190, in evaluate_codeblock
raise e
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 182, in evaluate_codeblock
self.evaluate_statement(cur)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 198, in evaluate_statement
self.assignment(cur)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 635, in assignment
value = self.evaluate_statement(node.value)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 196, in evaluate_statement
return self.function_call(cur)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 521, in function_call
res = func(node, func_args, kwargs)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/decorators.py", line 277, in wrapper
return f(*nargs, **wrapped_kwargs)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/decorators.py", line 596, in wrapper
return f(*wrapped_args, **wrapped_kwargs)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreter/interpreter.py", line 879, in func_subproject
return self.do_subproject(args[0], 'meson', kw)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreter/interpreter.py", line 924, in do_subproject
subdir = r.resolve(subp_name, method)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 466, in resolve
self.get_file()
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 543, in get_file
path = self.get_file_internal('source')
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 763, in get_file_internal
self.download(what, cache_path)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 738, in download
dhash, tmpfile = self.get_data_with_backoff(srcurl)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 731, in get_data_with_backoff
return self.get_data(urlstring)
File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 702, in get_data
block = resp.read(blocksize)
File "/usr/lib/python3.10/http/client.py", line 466, in read
s = self.fp.read(amt)
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
ERROR: Unhandled python OSError. This is probably not a Meson bug, but an issue with your build environment.
error: subprocess-exited-with-error
× Preparing editable metadata (pyproject.toml) did not run successfully.
│ exit code: 104
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/xenophon/github/matplotlib/env/bin/python /home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_editable /tmp/tmp811wdin1
cwd: /home/xenophon/github/matplotlib
Preparing editable metadata (pyproject.toml) ... error
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Expected outcome
builds successfully
Additional information
It could be an http vs https issue: the link (and all of qhull's page) is http. Chaning the link/file/hash on qhull.wrap to the equivalent tarball on qhull's github page at least advances the error to
Downloading qhull source from https://github.com/qhull/qhull/archive/refs/tags/v8.0.2.tar.gz Downloading file of unknown size. Executing subproject qhull qhull| Project name: qhull qhull| Project version: 8.0.2 qhull| C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0") qhull| C linker for the host machine: cc ld.bfd 2.38 ../../subprojects/qhull-2020.2/meson.build:5:12: ERROR: Include dir src does not exist.
I also tried installing libqhull8.0 and using system libraries, but got the same issue
Operating system
Linux 6.2.0-34-generic; Ubuntu 22.04.1
Matplotlib Version
master from today: dfdb37a
Matplotlib Backend
N/A
Python version
3.10.12
Jupyter version
N/A
Installation
git checkout
Activity
rcomer commented on Oct 20, 2023
Hi @Jacob-Stevens-Haas, since the move to meson, we need a different command to install. This is shown in the development version of the guide:
https://matplotlib.org/devdocs/devel/development_setup.html#install-matplotlib-in-editable-mode
ksunden commented on Oct 20, 2023
Your initial error also looks like a (likely temporary) networking error. Perhaps the qhull download servers were offline for a bit.
Jacob-Stevens-Haas commented on Oct 23, 2023
Thanks both!
Correcting the Command
When using the new command from the development guide, I get the ERROR: Include dir src does not exist (The same error as previously when I pointed to the github.com tarball instead of qhull.org)
Error when building with: `python -m pip install --verbose --no-build-isolation --editable .[dev]`Using pip 22.0.2 from /home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pip (python 3.10) Obtaining file:///home/xenophon/github/matplotlib Running command Checking if build backend supports build_editable Checking if build backend supports build_editable ... done Running command Preparing editable metadata (pyproject.toml) + meson setup /home/xenophon/github/matplotlib /home/xenophon/github/matplotlib/build/cp310 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/xenophon/github/matplotlib/build/cp 310/meson-python-native-file.ini The Meson build system Version: 1.2.2 Source dir: /home/xenophon/github/matplotlib Build dir: /home/xenophon/github/matplotlib/build/cp310 Build type: native build Project name: matplotlib Project version: 3.9.0.dev0 C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0") C linker for the host machine: cc ld.bfd 2.38 C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0") C++ linker for the host machine: c++ ld.bfd 2.38 Host machine cpu family: x86_64 Host machine cpu: x86_64 Program python found: YES (/home/xenophon/github/matplotlib/env/bin/python) Did not find pkg-config by name 'pkg-config' Found Pkg-config: NO Run-time dependency python found: YES 3.10 pybind11-config found: YES (/home/xenophon/github/matplotlib/env/bin/pybind11-config) 2.11.1 Run-time dependency pybind11 found: YES 2.11.1 Executing subproject freetype-2.6.1 freetype-2.6.1| Project name: freetype2 freetype-2.6.1| Project version: 2.6.1 freetype-2.6.1| C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0") freetype-2.6.1| C linker for the host machine: cc ld.bfd 2.38 freetype-2.6.1| Has header "unistd.h" : YES freetype-2.6.1| Has header "fcntl.h" : YES freetype-2.6.1| Has header "stdint.h" : YES freetype-2.6.1| Configuring ftconfig.h using configuration freetype-2.6.1| Configuring ftoption.h using configuration freetype-2.6.1| Build targets in project: 3 freetype-2.6.1| Subproject freetype-2.6.1 finished. WARNING: Subproject qhull's revision may be out of date; its wrap file has changed since it was first configured Executing subproject qhull qhull| Project name: qhull qhull| Project version: 8.0.2 qhull| C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0") qhull| C linker for the host machine: cc ld.bfd 2.38 ../../subprojects/qhull-2020.2/meson.build:5:12: ERROR: Include dir src does not exist. A full log can be found at /home/xenophon/github/matplotlib/build/cp310/meson-logs/meson-log.txt error: subprocess-exited-with-error × Preparing editable metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. full command: /home/xenophon/github/matplotlib/env/bin/python /home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_ed itable /tmp/tmp3o0u5ac4 cwd: /home/xenophon/github/matplotlib Preparing editable metadata (pyproject.toml) ... error error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details.The same error occurs even if I replace the wrap file url/filename/hash with the one from github.
Using system libraries
Using the correct command with system Qhull works. Thanks! Details: upgraded from pip 22.2 to pip 23.3, apt install libqhull-dev libqhull8.0
Still not able to download qhull manually
They're not offline, as wget receives a partial download (see above linked issue), but they do seem to be having trouble (several days later). For sanity's sake, is anyone else able to download http://www.qhull.org/download/qhull-2020-src-8.0.2.tgz in any way? Even so, is it perhaps better to use github's link, which uses HTTPS, versus qhull's?
tacaswell commented on Oct 23, 2023
if you do git clean -xfd in your Matplotlib checkout and then try again does it work? My guess is that there may be a partial set of files which is enough to skip re-downloading / unpacking but not enough to work. When in doubt wipe it clean and try again 🔨 💣 .
ksunden commented on Oct 23, 2023
I'm not too concerned about https, because we do checksum the download (sha256) so having ssl during download doesn't provide much additional safety, actually.
I am able to download the file (though admittedly firefox gives me a warning about https when I click the link from github) (I was also able to do git clean -xfd and reinstall today).
Jacob-Stevens-Haas commented on Oct 24, 2023
Oh, that's a good point about the checksum. Today it did download (with the mentioned security risk). It might have just been an issue over the weekend.
tacaswell commented on Oct 24, 2023
QuLogic commented on Oct 24, 2023
Meson does the downloading now, and that directory is not used any more. Tarballs are cached in subprojects/packagecache.
tacaswell commented on Oct 24, 2023
oh, good to know!
Hid my incorrect comment to not confuse people in the future.
Jacob-Stevens-Haas commented on Dec 6, 2023
While I had been able to get around the issue in the past by using system libraries, I came back to work on a a new laptop at a new workplace and ran into the same issue of being unable to download from qhull servers. A desk neighbor mentioned that they too had had the same problem building matplotlib to make a PR in the past (#27158).
I had assumed I was the only developer trying to work on matplotlib that ran into the qhull issue. But randomly meeting someone else, in-person, only a month later makes me think that a lot of would-be developers on academic/corporate networks could be getting stopped or burdened by this issue.
My colleague believed it was because IT's firewall felt something qhull, perhaps because it was HTTP and not HTTPS. OTOH, the authors of qhull mentioned that the qhull server didn't support range requests, in case that had been the issue.
I have a branch moving the build to qhull's github-hosted 8.0.2 tarball. My feeling is that github is generally a more stable place for code than a project like qhull's website. Would that be a welcome PR?
tacaswell commented on Dec 7, 2023
I think moving to use the GH hosted tarball is reasonable (so we stop hammering the qhull server).
When we were doing the downloading I think the tarballs were included in the directories that we had CI restore from cache.