chore: matplotlib 3.5.1, yt 4 by henryiii · Pull Request #2234 · pyodide/pyodide · GitHub

Description

Bumping MPL version - it turns out that Hist (which uses boost-histogram) depends on mplhep, which in turn requires matplotlib 3.4+. So I migrated the patches and bumped MPL.

I noticed micropip doesn't seem to respect extras, that is, await micropip.install("hist[plot]") doesn't seem to work. (will investigate later)

Updated matplotlib to 3.5
- Added fonttools
Updated yt to 4
- Added cmyt and unyt, which I am pretty sure are yt specific. At least by looking at the name. 😆
- Also colorspacious
- Also tdqm
- Also tomli and tomli-w (backported from the next yt release to avoid adding a dead dependency toml)

Closes #2088 too. Also 3.5 was the first version to officially support 3.10, I think.

Thanks @henryiii!

I noticed micropip doesn't seem to respect extras

We at least tried to respect extras:

pyodide/packages/micropip/src/micropip/_micropip.py

Line 315 in e3760e8

for recurs_req in dist.requires(extras):

We even have an integration test for this. It isn't very discerning but I believe at least the tiny attempt towards supporting extras that we have is required to pass this test.

pyodide/packages/micropip/test_micropip.py

Line 218 in e3760e8

_micropip.PACKAGE_MANAGER.gather_requirements(

But I don't understand this stuff very well so there's some guesswork involved.

It's possible it's just hiding the failure since it can't resolve matplotlib>=3.4.

Hmm. Well in any case we would certainly appreciate any improvements you can make to micropip. Failing that, at least it's good if we can get a better sense for the existing bugs =)

Same permission denied issue in CI.

FYI, this was the bug, I believe. We can actually drop the freetype patch with this and it still builds (not sure if that's better, but it does build without the patch).

If it builds without the patch, I think we should drop it.

Could someone restart the failed build? Wrapt flake unrelated.

Dumping the errors here to look at them later. fonttools is an "easy" one, as that's a (I guess new) dependency.

ValueError: Trying to re-register the builtin cmap 'cubehelix'

conftest.JavascriptException: PythonError: Traceback (most recent call last):
  File "/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/lib/python3.9/site-packages/_pyodide/_base.py", line 494, in eval_code_async
    await CodeRunner(
  File "/lib/python3.9/site-packages/_pyodide/_base.py", line 345, in run_async
    coroutine = eval(self.code, globals, locals)
  File "<exec>", line 1, in <module>
  File "/lib/python3.9/site-packages/yt/__init__.py", line 63, in <module>
    from yt.fields.api import \
  File "/lib/python3.9/site-packages/yt/fields/api.py", line 19, in <module>
    from . import angular_momentum
  File "/lib/python3.9/site-packages/yt/fields/angular_momentum.py", line 26, in <module>
    from .vector_operations import \
  File "/lib/python3.9/site-packages/yt/fields/vector_operations.py", line 32, in <module>
    from yt.geometry.geometry_handler import \
  File "/lib/python3.9/site-packages/yt/geometry/geometry_handler.py", line 29, in <module>
    from yt.utilities.parallel_tools.parallel_analysis_interface import \
  File "/lib/python3.9/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 30, in <module>
    from yt.data_objects.image_array import ImageArray
  File "/lib/python3.9/site-packages/yt/data_objects/image_array.py", line 18, in <module>
    from yt.visualization.image_writer import write_bitmap, write_image
  File "/lib/python3.9/site-packages/yt/visualization/image_writer.py", line 26, in <module>
    from .color_maps import mcm
  File "/lib/python3.9/site-packages/yt/visualization/color_maps.py", line 164, in <module>
    add_colormap("cubehelix", _cubehelix_data)
  File "/lib/python3.9/site-packages/yt/visualization/color_maps.py", line 52, in add_colormap
    mcm.register_cmap(name, yt_colormaps[name])
  File "/lib/python3.9/site-packages/matplotlib/cm.py", line 245, in register_cmap
    raise ValueError(msg)
ValueError: Trying to re-register the builtin cmap 'cubehelix'.

ModuleNotFoundError: No module named 'fontTools'

conftest.JavascriptException: PythonError: Traceback (most recent call last):
  File "/lib/python3.9/site-packages/_pyodide/_base.py", line 415, in eval_code
    CodeRunner(
  File "/lib/python3.9/site-packages/_pyodide/_base.py", line 296, in run
    coroutine = eval(self.code, globals, locals)
  File "<exec>", line 1, in <module>
  File "/lib/python3.9/site-packages/matplotlib/pyplot.py", line 958, in savefig
    res = fig.savefig(*args, **kwargs)
  File "/lib/python3.9/site-packages/matplotlib/figure.py", line 3019, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "/lib/python3.9/site-packages/matplotlib/backend_bases.py", line 2259, in print_figure
    canvas = self._get_output_canvas(backend, format)
  File "/lib/python3.9/site-packages/matplotlib/backend_bases.py", line 2184, in _get_output_canvas
    canvas_class = get_registered_canvas_class(fmt)
  File "/lib/python3.9/site-packages/matplotlib/backend_bases.py", line 144, in get_registered_canvas_class
    backend_class = importlib.import_module(backend_class).FigureCanvas
  File "/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/lib/python3.9/site-packages/matplotlib/backends/backend_pdf.py", line 47, in <module>
    from . import _backend_pdf_ps
  File "/lib/python3.9/site-packages/matplotlib/backends/_backend_pdf_ps.py", line 8, in <module>
    from fontTools import subset
ModuleNotFoundError: No module named 'fontTools'

assert len(content) == 16283

    @pytest.mark.skip_refcount_check
    @pytest.mark.skip_pyproxy_check
    def test_svg(selenium):
        if selenium.browser == "node":
            pytest.xfail("No supported matplotlib backends on node")
        selenium.load_package("matplotlib")
        selenium.run("from matplotlib import pyplot as plt")
        selenium.run("plt.figure(); pass")
        selenium.run("x = plt.plot([1,2,3])")
        selenium.run("import io")
        selenium.run("fd = io.BytesIO()")
        selenium.run("plt.savefig(fd, format='svg')")
        content = selenium.run("fd.getvalue().decode('utf8')")
>       assert len(content) == 16283
E       assert 14998 == 16283
E         +14998
E         -16283

packages/matplotlib/test_matplotlib.py:86: AssertionError

YT has broken out a new dependency, unyt, but there are actually a lot of dependencies not included here:

https://github.com/yt-project/yt/blob/ff5e808ec4dbebad812cab82c75297d2cfcd0519/setup.cfg#L40-L52

Including the dead "toml" library instead of tomli. 🤦 Guessing something here is triggering the "yt not found" issue.

Edit: lots of new dependencies, actually. :(

Any idea why yt is still reporting "Module not found"? I've tried to add anything it might depend on, but can't tell if it's actually failing because a dependency is missing, as it only reports yt failed. Is it possibly not loading the chained exceptions and there's still a missing dependency?

It looks like the built wheel is missing the folders yt (with the Python code) and yt.libs (with the .so file) which are present in correct wheels. My guess is it might be --skip-build troubles, since these are sort of familiar symptoms. Might be good to see what happens if you merge #2238 here.

As a bandaid I think you can add:

build:
  unvendor-tests: false
  post:
    cd build/yt-4.0.2/
    cp -r build/temp.emscripten_wasm32-3.9/yt dist/yt-4.0.2/yt

this is good enough until we actually fix our build system. I think I know the right fix, #2238 is moving in the right direction but we actually need to run the normal build command to make a wheel and intercept the compilation commands as they are issued and replace them on the spot. This is the only thing that can work.

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

See https://matplotlib.org/3.3.0/api/prev_api_changes/api_changes_3.2.0/deprecations.html#fonts

See https://matplotlib.org/3.3.0/api/prev_api_changes/api_changes_3.2.0/deprecations.html#fonts. This should have been producing a FutureWarning in 3.3 already, are those not getting displayed? Ideally warnings should be errors in tests.

I wonder if this is showing up in #2194, actually…

  File "/lib/python3.9/site-packages/matplotlib/backends/browser_backend.py", line 5, in <module>
    from js import document, window
ImportError: cannot import name 'document' from 'js' (unknown location)

Do you know where js is coming from? (and maybe even why upgrading matplotlib is causing js to not be there now when it was before...) (Node only)

I think the right solution is to xfail the test on node. I think matplotlib has never functioned on node and it was only by chance that we weren't hitting problems in the tests before.

Ahh, this is the (new yt split package) import cymt test. How do I disable that on node only?

For the import tests, you need to add it here:

pyodide/packages/test_packages_common.py

Line 36 in fff0f0e

"node": [],

It might be good to add comments to all the new yt dependencies saying that they are yt dependencies?

Why do we need this patch? Surely sys.platform != 'darwin'?

"This comes from #541. We let it as is in case it helps building on Mac."

I don't know if it helps on a Mac, I'm running in docker. Probably should try running native at some point. What does this return when cross compiling?

I've run it native before. I guess that's the issue that it returns Darwin on macs. The Python ecosystem is really bad at cross compiling =(

Thanks @henryiii!

It might be good to add comments to all the new yt dependencies saying that they are yt dependencies?

That could pretty quickly become obsolete if anything else gets added. tomli is very likely to be a build time dependency for some package eventually (before 3.11, anyway, when it becomes a stdlib package tomllib). The others can be pretty quickly checked via:

import yaml
from pathlib import Path

DIR = Path(__file__).parent.parent.resolve()
PACKAGES = DIR / "packages"

all_packages = {}

for package in PACKAGES.glob("*/meta.yaml"):
    with package.open(encoding="utf-8") as f:
        info = yaml.safe_load(f)
        all_packages[info["package"]["name"]] = info

deps = {name: val.get("requirements", {}).get("run", []) for name, val in all_packages.items()}
ideps = {name: set() for name in deps}
ideps["distutils"] = set()
for package, val in deps.items():
    for dep in val:
        ideps[dep].add(package)

for package in sorted(ideps, key=lambda x:(-len(ideps[x]), x)):
    print(f"{package} ({len(ideps[package])}):", *ideps[package])

Output:

numpy (25): scikit-learn scikit-image imageio pyb2d statsmodels pywavelets cmyt scipy bokeh mne patsy colorspacious autograd pyerfa numcodecs biopython boost-histogram pandas nlopt unyt astropy zarr networkx matplotlib yt
distutils (13): micropip logbook matplotlib mne scikit-image sympy bokeh joblib setuptools statsmodels pandas pywavelets astropy
six (11): cycler optlang python-dateutil retrying attrs bleach pytest bokeh html5lib patsy python-sat
setuptools (8): networkx nose logbook pytest mne pyb2d pandas yt
packaging (6): micropip bleach matplotlib scikit-image yt astropy
matplotlib (5): networkx scikit-image pywavelets yt cmyt
scipy (5): scikit-learn mne scikit-image statsmodels pywavelets
pillow (4): matplotlib bokeh scikit-image imageio
pyparsing (4): micropip matplotlib setuptools packaging
python-dateutil (3): matplotlib pandas bokeh
sympy (3): optlang unyt yt
colorspacious (2): yt cmyt
future (2): uncertainties autograd
libiconv (2): lxml libxml
libxml (2): lxml libxslt
more-itertools (2): pytest cmyt
pytz (2): matplotlib pandas
pyyaml (2): bokeh astropy
typing-extensions (2): pydantic bokeh
webencodings (2): bleach html5lib
zlib (2): lxml libxml
CLAPACK (1): scipy
Jinja2 (1): bokeh
MarkupSafe (1): Jinja2
asciitree (1): zarr
atomicwrites (1): pytest
attrs (1): pytest
beautifulsoup4 (1): lxml
cffi (1): cffi_example
cmyt (1): yt
cssselect (1): lxml
cycler (1): matplotlib
decorator (1): networkx
fonttools (1): matplotlib
glpk (1): swiglpk
html5lib (1): lxml
imageio (1): scikit-image
iniconfig (1): pytest
joblib (1): scikit-learn
kiwisolver (1): matplotlib
libxslt (1): lxml
libyaml (1): pyyaml
mpmath (1): sympy
msgpack (1): numcodecs
networkx (1): scikit-image
nose (1): cytoolz
numcodecs (1): zarr
pandas (1): statsmodels
parso (1): jedi
patsy (1): statsmodels
pluggy (1): pytest
py (1): pytest
pycparser (1): cffi
pydantic (1): pyb2d
pyerfa (1): astropy
pywavelets (1): scikit-image
regex (1): nltk
sharedlib-test (1): sharedlib-test-py
soupsieve (1): beautifulsoup4
swiglpk (1): optlang
threadpoolctl (1): scikit-learn
tomli (1): yt
tomli-w (1): yt
toolz (1): cytoolz
tqdm (1): yt
unyt (1): yt
Pygments (0):
astropy (0):
autograd (0):
biopython (0):
bleach (0):
bokeh (0):
boost-histogram (0):
cffi_example (0):
cloudpickle (0):
cpp-exceptions-test (0):
cytoolz (0):
distlib (0):
docutils (0):
fpcast-test (0):
freesasa (0):
jedi (0):
logbook (0):
lxml (0):
micropip (0):
mne (0):
nlopt (0):
nltk (0):
optlang (0):
pyb2d (0):
pyodide-interrupts (0):
pyrsistent (0):
pytest (0):
python-sat (0):
retrying (0):
scikit-image (0):
scikit-learn (0):
sharedlib-test-py (0):
sqlalchemy (0):
statsmodels (0):
traits (0):
uncertainties (0):
wrapt (0):
xlrd (0):
yt (0):
zarr (0):

Ahh, I think the problem with extras is that there is no warning / error if one or more is unsolvable, it just returns happily. If you do micropip.install("hist[plot]"), I think it's now getting past matplotlib, then hitting and failing on iminuit, and then ignoring things after it that it could have installed (mplhep), and not showing an error. keep_going doesn't seem to change anything (at least after it is installed).

henryiii force-pushed the henryiii/chore/bumps2 branch from efe8d1d to 81596d6 Compare March 1, 2022 23:05

henryiii commented Mar 1, 2022

View reviewed changes

This was referenced Mar 2, 2022

Convert to using tomli yt-project/yt#3830

Closed

fix: better error messages, fewer tracebacks #2242

Merged

henryiii force-pushed the henryiii/chore/bumps2 branch from a614060 to a15faaf Compare March 3, 2022 15:39

henryiii and others added 8 commits March 5, 2022 12:24

chore: updated matplotlib to 3.5.1

3dfc217

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

fix: drop freetype patch

2f9e8fe

fix: add fonttools and bump yt

ed8910d

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

fix: more tests, include unyt

88b9728

feat: add more packages for yt

0b0cdfc

fix: add deps, include upstream tomli patch

5664e6f

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

Add post script fix

25f99f4

fix: remove deprecated usage of get_width/height

6f2a8de

See https://matplotlib.org/3.3.0/api/prev_api_changes/api_changes_3.2.0/deprecations.html#fonts

henryiii force-pushed the henryiii/chore/bumps2 branch from a9449e9 to 6f2a8de Compare March 5, 2022 17:39

henryiii commented Mar 5, 2022

View reviewed changes

Update test_packages_common.py

b63af8d

henryiii changed the title chore: updated matplotlib to 3.5.1 chore: matplotlib 3.5.1, yt 4 Mar 6, 2022

hoodmane reviewed Mar 6, 2022

View reviewed changes

Update test_packages_common.py

456a95e

hoodmane approved these changes Mar 7, 2022

View reviewed changes

hoodmane merged commit 138c7a7 into pyodide:main Mar 7, 2022

henryiii deleted the henryiii/chore/bumps2 branch March 7, 2022 05:26

                     - lib/matplotlib/backends/html5_canvas_backend.py
                   - - src/setup.cfg
-                    - ./setup.cfg
+                    - ./mplsetup.cfg

-                          w, h = image.get_width(), image.get_height()
+                          image_arr = np.asarray(image)
+                          h, w = image_arr.shape

		@@ -12,4 +13,5 @@ diff --git a/setupext.py b/setupext.py
		raise Skipped("Mac OS-X only")

chore: matplotlib 3.5.1, yt 4 by henryiii · Pull Request #2234 · pyodide/pyodide · GitHub

Repository navigation

chore: matplotlib 3.5.1, yt 4 - #2234

chore: matplotlib 3.5.1, yt 4#2234hoodmane merged 10 commits intopyodide:mainpyodide/pyodide:mainfrom henryiii:henryiii/chore/bumps2henryiii/pyodide:henryiii/chore/bumps2Copy head branch name to clipboard

Conversation

henryiii commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

hoodmane commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hoodmane commented Mar 1, 2022

Uh oh!

henryiii commented Mar 1, 2022

Uh oh!

henryiii Mar 1, 2022

Choose a reason for hiding this comment

Uh oh!

hoodmane Mar 1, 2022

Choose a reason for hiding this comment

Uh oh!

henryiii commented Mar 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented Mar 2, 2022

Uh oh!

henryiii commented Mar 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented Mar 5, 2022

Uh oh!

hoodmane commented Mar 5, 2022

Uh oh!

hoodmane commented Mar 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii Mar 5, 2022

Choose a reason for hiding this comment

Uh oh!

henryiii Mar 5, 2022

Choose a reason for hiding this comment

Uh oh!

henryiii commented Mar 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hoodmane commented Mar 6, 2022

Uh oh!

henryiii commented Mar 6, 2022

Uh oh!

hoodmane commented Mar 6, 2022

Uh oh!

hoodmane commented Mar 6, 2022

Uh oh!

hoodmane Mar 6, 2022

Choose a reason for hiding this comment

Uh oh!

henryiii Mar 7, 2022

Choose a reason for hiding this comment

Uh oh!

hoodmane Mar 7, 2022

Choose a reason for hiding this comment

Uh oh!

hoodmane left a comment

Choose a reason for hiding this comment

Uh oh!

henryiii commented Mar 7, 2022

Uh oh!

henryiii commented Mar 8, 2022

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

chore: matplotlib 3.5.1, yt 4#2234
hoodmane merged 10 commits into
pyodide:mainfrom
henryiii:henryiii/chore/bumps2

henryiii commented Mar 1, 2022 •
edited
Loading

hoodmane commented Mar 1, 2022 •
edited
Loading

henryiii commented Mar 1, 2022 •
edited
Loading

henryiii commented Mar 2, 2022 •
edited
Loading

henryiii commented Mar 2, 2022 •
edited
Loading

hoodmane commented Mar 5, 2022 •
edited
Loading

henryiii commented Mar 6, 2022 •
edited
Loading