Repository navigation
Cross-compiling matplotlib and dependencies for OpenWRT fails on import matplotlib.pyplot #9244
Description
Bug report
Bug summary
When cross-compiling matplotlib and dependencies (numpy, six, pytz, etc) for OpenWRT in Ubuntu 16.04LTS, all code properly compiles. However, when testing matplotlib on the target, when importing matplotlib.pyplot I get the failure "terminate called after throwing an instance of 'char const*'". Specifically, after a font is imported in font_manager, the failure comes up when attempting to add a second font (regardless of the first font). I've attempted multiple versions of Python3 (3.6.2, 3.6.0, 3.5.4), as well as numerous combinations of dependent packages. Not that I am only compiling freetype as a library for the shared object "libfreetype.so.6*", as well as the ft2font header file, instead of the full freetype package.
Code for reproduction
import matplotlib
print(matplotlib.__version__)
import matplotlib.pyplot as plt
print("Success!")Actual outcome
root@LEDE:~# python3 test.py --verbose-debug $HOME=/root matplotlib data path /usr/lib/python3.6/site-packages/matplotlib/mpl-data ***************************************************************** You have the following UNSUPPORTED LaTeX preamble customizations: Please do not ask for support with these customizations active. ***************************************************************** loaded rc file /usr/lib/python3.6/site-packages/matplotlib/mpl-data/matplotlibrc matplotlib version 2.0.2 verbose.level debug interactive is False platform is linux loaded modules: <dict_keyiterator object at 0x17ebb00> CACHEDIR=/root/.cache/matplotlib font search path ['/usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf', '/usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/afm', '/usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/pdfcorefonts'] trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/cmb10.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/cmr10.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXSizOneSymReg.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXSizThreeSymBol.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono-Oblique.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif-Bold.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneral.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerifDisplay.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniBolIta.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/cmex10.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/cmmi10.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/cmss10.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXSizTwoSymBol.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/cmsy10.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/cmtt10.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUni.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-BoldOblique.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralItalic.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralBolIta.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-Bold.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFiveSymReg.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-Oblique.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralBol.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXSizThreeSymReg.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFourSymBol.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFourSymReg.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniBol.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniIta.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif-BoldItalic.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansDisplay.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono-Bold.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXSizTwoSymReg.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXSizOneSymBol.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono-BoldOblique.ttf trying fontname /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif-Italic.ttf createFontDict: /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/cmb10.ttf createFontDict: /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/cmr10.ttf terminate called after throwing an instance of 'char const*' Aborted
Expected outcome
root@LEDE:~# python3 test.py 2.0.2 Success!
Matplotlib version
- Operating System: OpenWRT / LEDE V17.01.02
- Matplotlib Version: 2.0.2
- Python Version: 3.6.2
- Other Libraries:
- numpy: 1.12.1
- six: 1.10.0
- dateutil: 2.6.1
- pytz: 2017.2
- cycler: 0.10.0
- pyparsing: 2.2.0
- libfreetype: 2.5.5
- libpng: 1.2.57
BUILDING MATPLOTLIB
matplotlib: yes [2.0.2]
python: yes [3.6.2 (default, Jun 8 2017, 21:02:37) [GCC
5.4.0 20160609]]
platform: yes [linux]
REQUIRED DEPENDENCIES AND EXTENSIONS
numpy: yes [version 1.12.1]
six: yes [using six version 1.10.0]
dateutil: yes [using dateutil version 2.6.1]
functools32: yes [Not required]
subprocess32: yes [Not required]
pytz: yes [using pytz version 2017.2]
cycler: yes [using cycler version 0.10.0]
tornado: yes [tornado was not found. It is required for the
WebAgg backend. pip/easy_install may attempt to
install it after matplotlib.]
pyparsing: yes [using pyparsing version 2.2.0]
libagg: yes [pkg-config information for 'libagg' could not
be found. Using local copy.]
freetype: yes [version 2.5.5]
png: yes [version 1.2.57]
qhull: yes [pkg-config information for 'qhull' could not be
found. Using local copy.]
Activity
jwbrack commented on Sep 27, 2017
Additionally, libfreetype was compile with the following options:
CONFIGURE_ARGS += \ --enable-shared \ --enable-static \ --with-bzip2=yes \ --with-zlib=yes \ --with-png=no \tacaswell commented on Sep 27, 2017
Can you try against either master or v2.1.x branches? I have a vague memory of a similar issue being fixed.
tacaswell commented on Sep 27, 2017
you need to apply this patch locally I think #9074
jwbrack commented on Sep 27, 2017
Same results.
Patch 9074 + V2.0.2
Patch 9074 + V2.1.x @ f2c8fae
anntzer commented on Oct 3, 2017
Do you have instructions on how to reproduce your setup, possibly using docker? (would something like https://wiki.openwrt.org/doc/howto/docker_openwrt_image work?) (e.g., how did you install python etc.)
jwbrack commented on Oct 4, 2017
Python / matplotlib / etc are all compiled into the firmware image. I had to add a missing package to LEDE 17.01.02 (numpy), and add in python3 support for others (pytz, pyparsing, dateutil, cycler, and six), but everything else is stock in the build environment (including both libfreetype and libpng). This is all based on the commit tag "v17.01.2" for the LEDE project (last major release).
I attempted to build for docker (x86 architecture), but testing here shows as success in loading matplotlib.pyplot (no errors in freetype). Note that the original target is the BRCM 2708 (Raspberry Pi 3). This seems to point that there is some possible issue in cross-compiling for ARM architectures, and passing down the flags appropriately (should happen transparently in buildroot systems). See below for the makefile:
Added patch to modify the destdirs to point to the arm compiled packages:
I'm attempting a QEMU ARM image now, and will upload with success. Else, I have a Raspberrry Pi 3 image ready to go at any time. Am I able to directly upload the firmware images here?
anntzer commented on Oct 4, 2017
tbh I'm not super familiar (actually, not at all) with the general docker ecosystem, but I guess you can use https://quay.io/ similarly to https://github.com/pypa/manylinux (which is the only image I've interacted with so far :p).
jwbrack commented on Oct 4, 2017
Issue is not with the docker. A docker image (compiled against x86, which is the native build environment) architecture works just fine for loading in matplotlib,pyplot, which makes it seem that this is an issue with cross-compiling to an ARM architecture. Docker is not a real virtual machine, that is the architecture in the docked image must match that of the host. Thus why I am trying to compile against a QEMU arm virtual machine to see if I have any success there in re-creating the error.
For more on armvirt:
https://lede-project.org/docs/guide-developer/test-virtual-image-using-armvirt
anntzer commented on Oct 4, 2017
Just to be clear, I don't think I have the knowledge to test a full VM.
jwbrack commented on Oct 4, 2017
How about I put my Raspi somewhere publicly accessible with the build on it? That way you can just ssh into the box for access, and do all your testing there.
anntzer commented on Oct 4, 2017
Go for it. email me to firstname.lastname@berkeley.edu
jwbrack commented on Oct 4, 2017
Sent an email.
jwbrack commented on Oct 6, 2017
Determined the root issue after instrumenting the code a bit. The font list cache file (fontList.json) is not being generated. See failure below. It seems a similar issue was found in #5406, but no resolution was really added there.
createFontDict: /usr/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif-BoldItalic.ttf mpl_PyFile_DupClose(): Call to PyObject_CallMethod() failed mpl_PyFile_DupClose(): Python error occured - printing traceback Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/matplotlib/font_manager.py", line 1444, in <module> fontManager = json_load(_fmcache) File "/usr/lib/python3.6/site-packages/matplotlib/font_manager.py", line 973, in json_load with open(filename, 'r') as fh: FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/matplotlib/fontList.json' During handling of the above exception, another exception occurred: ValueError: whence value 217555 unsupported terminate called after throwing an instance of 'std::runtime_error' what(): Couldn't close file Abortedanntzer commented on Oct 6, 2017
Actually the error occurs while trying to open the not-yet-existent cache (which should normally trigger a normal exception that leads to the file being generated).
I assume the error occurs at the line
?
On the CPython side the error seems to occur in bufferedio.c: _io__Buffered_seek_impl:
but looking at the mpl code it looks like whence should be zero. Can you check that this is indeed where the error happens? That somehow an invalid value of 217555 reaches _io__Buffered_seek_impl? (Yes that probably means recompiling cpython, if you can't get gdb to work...)
zokl commented on Oct 30, 2017
Hello,
i'm trying to compile the matte run on LEDE and the x86-64 platform, version 1.4.3 is seamless and everything works from 1.5.3 I have a similar problem in compilation. Can you expose your compilation packages to matplotlib?
Thank you, zbynek Kocur
anntzer commented on Dec 3, 2017
@jwbrack Perhaps #9763 could help? It is a complete rewrite of ft2font, which in particular doesn't rely on the old file API (but needs a very recent compiler).
zokl commented on Dec 3, 2017
Thank you @jwbrack , i will try it.
github-actions commented on Apr 19, 2023
This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!
anntzer commented on Apr 19, 2023
Especially with the move to pybind11 the situation likely changed; let's close this and track this on a new issue if needed.
In particular, in 97477d7 the line that throws the exception mentioned originally was removed; now we call PyErr_WriteUnraisable if an error occurs in the file close.