[Bug]: ImportError when using Matplotlib v3.8.0 in Python package tests · Issue #26827 · matplotlib/matplotlib · GitHub

Bug summary

I am encountering an ImportError when attempting to use Matplotlib v3.8.0 in my Python package's tests, specifically when running them with pytest. This issue does not occur with older versions of Matplotlib, such as v3.7.3.

Code for reproduction

>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/pyplot.py", line 66, in <module>
    from matplotlib.figure import Figure, FigureBase, figaspect
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/figure.py", line 43, in <module>
    from matplotlib import _blocking_input, backend_bases, _docstring, projections
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/projections/__init__.py", line 58, in <module>
    from mpl_toolkits.mplot3d import Axes3D
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in <module>
    from .axes3d import Axes3D
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py", line 23, in <module>
    from matplotlib import _api, cbook, docstring, _preprocess_data
ImportError: cannot import name 'docstring' from 'matplotlib' (/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py)

Actual outcome

When running the tests, I receive the following ImportError:

tests/integration/test_cli.py:263: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.local/lib/python3.10/site-packages/pyKVFinder/main.py:214: in cli
    plot_frequencies(frequencies, output_plot)
../../.local/lib/python3.10/site-packages/pyKVFinder/utils.py:952: in plot_frequencies
    import matplotlib.pyplot as plt
../../.local/lib/python3.10/site-packages/matplotlib/pyplot.py:66: in <module>
    from matplotlib.figure import Figure, FigureBase, figaspect
../../.local/lib/python3.10/site-packages/matplotlib/figure.py:43: in <module>
    from matplotlib import _blocking_input, backend_bases, _docstring, projections
../../.local/lib/python3.10/site-packages/matplotlib/projections/__init__.py:58: in <module>
    from mpl_toolkits.mplot3d import Axes3D
/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py:1: in <module>
    from .axes3d import Axes3D
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    axes3d.py, original mplot3d version by John Porter
    Created: 23 Sep 2005
    
    Parts fixed by Reinier Heeres <reinier@heeres.eu>
    Minor additions by Ben Axelrod <baxelrod@coroware.com>
    Significant updates and revisions by Ben Root <ben.v.root@gmail.com>
    
    Module containing Axes3D, an object which can plot 3D objects on a
    2D matplotlib figure.
    """
    
    from collections import defaultdict
    import functools
    import inspect
    import itertools
    import math
    from numbers import Integral
    import textwrap
    
    import numpy as np
    
>   from matplotlib import _api, cbook, docstring, _preprocess_data
E   ImportError: cannot import name 'docstring' from 'matplotlib' (/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py)

/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py:23: ImportError

Expected outcome

I expected my tests to run successfully with Matplotlib v3.8.0, just as they did with v3.7.3.

Additional information

Has this worked in earlier versions?
It is working in previous versions (eg, v3.7.3).
Do you know why this bug is happening?
When running the following command to get the Matplotlib Backend:

import matplotlib; print(matplotlib.get_backend())

I got the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py", line 1275, in get_backend
    return rcParams['backend']
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py", line 759, in __getitem__
    from matplotlib import pyplot as plt
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/pyplot.py", line 66, in <module>
    from matplotlib.figure import Figure, FigureBase, figaspect
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/figure.py", line 43, in <module>
    from matplotlib import _blocking_input, backend_bases, _docstring, projections
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/projections/__init__.py", line 58, in <module>
    from mpl_toolkits.mplot3d import Axes3D
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in <module>
    from .axes3d import Axes3D
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py", line 23, in <module>
    from matplotlib import _api, cbook, docstring, _preprocess_data
ImportError: cannot import name 'docstring' from 'matplotlib' (/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py)

Operating system

Pop!_OS 22.04 LTS

Matplotlib Version

3.8.0

Matplotlib Backend

ImportError: cannot import name 'docstring' from 'matplotlib' (/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/init.py)

Python version

3.10.12

Jupyter version

6.5.1 (Not using Jupyter notebook)

Installation

pip

breaks my CI too

works around matplotlib/matplotlib#26827

Your installs are crossed. You are picking up mpl_toolkits from a different distribution.

On the other hand this happens pretty commonly - I forget the explanation and maybe indicates an issue with how we install the wheels?

This looks like an installation issue?

It is certainly true that docstring was removed after a deprecation cycle of 2 releases and changed to be private (_docstring)

However, the line that is erroring in mplot3d/axes3d.py was updated #22148, indicating that you have a version of mpl that it is picking up that is <3.6

52df591#diff-a67656224bfb26dc6de0a0f89e398cd6999377d1482aca3e867ee550ff05be57

I will also point out that matplotlib is coming from site-packages, but mpl_toolkits is coming from dist-packages

@jklymak - "You are ..." sound like we did something actively. I assume we did not. The nightly CI build in jenkins fail. This seems to be just dependency hell as I have seen quite a few times in the last few months.

Avoiding 3.8.0 fixes the jenkins problem and does not harm the github CI. The github CI also worked with the 3.8.0 version so there might be indeed be a sandboxing problem and some crossover between packages. I donot even know what mpl_toolkits is and why it would have anything to do with the way I am using matplotlib. My code is quite short and it surprises me that it should have such hidden dependencies

'''
Created on 2020-07-05

@author: wf
'''
import matplotlib.pyplot as plt
from collections import Counter
import numpy as np
import os

class Plot(object):
    '''
    create Plot based on counters
    see https://stackoverflow.com/questions/19198920/using-counter-in-python-to-build-histogram
    '''
    def __init__(self, valueList,title,xlabel=None,ylabel=None,gformat='.png',fontsize=12,plotdir=None,debug=False):
        '''
        Constructor
        '''
        self.counter=Counter(valueList)
        self.valueList=valueList
        self.title=title
        self.xlabel=xlabel
        self.ylabel=ylabel
        self.fontsize=fontsize
        self.gformat=gformat
        self.debug=debug
        path=os.path.dirname(__file__)
        if plotdir is not None:
            self.plotdir=plotdir
        else:
            self.plotdir=path+"/../plots/"
            os.makedirs(self.plotdir,exist_ok=True)
            
    def titleMe(self):   
        ''' set my title and labels '''     
        plt.title(self.title, fontsize=self.fontsize)
        if self.xlabel is not None:
            plt.xlabel(self.xlabel)
        if self.ylabel is not None:    
            plt.ylabel(self.ylabel)
            
    def showMe(self,mode='show',close=True):
        ''' show me in the given mode '''
        if mode=="show":
            plt.show() 
        else:
            plt.savefig(self.plotdir+self.title+self.gformat)
        if close:    
            plt.close()    
            
    def barchart(self,mode='show'):
        ''' barchart based histogram for the given counter '''
        labels, values = zip(*self.counter.items())
        indexes = np.arange(len(labels))
        width = 1
        self.titleMe()
        plt.bar(indexes, values, width)
        plt.xticks(indexes + width * 0.5, labels)
        plt.yticks(np.arange(1,max(values)+1,step=1))
        self.showMe(mode)
        
    def showDebug(self):    
        print("   value  list: ",self.valueList)
        print("counter  items: ",self.counter.items())
        print("counter values: ",self.counter.values())
        print("counter   keys: ",self.counter.keys())
        
    def hist(self,mode="show"):
        ''' create histogram for the given counter '''
        if self.debug:
            self.showDebug()
        self.titleMe()
        # see https://stackoverflow.com/a/2162045/1497139
        plt.hist(self.valueList,bins=len(self.counter.keys()))
        self.showMe(mode)
        pass

mpl_toolkits is distributed with matplotlib, in the matplotlib pypi (and conda, etc) package. It is a separate python package, where some more experimental features (most notably 3D plotting support), though it is imported (for 3D) by default.

As to why you have multiple and why python is choosing different locations for matplotlib and mpl_toolkits, I don't honestly know. Is it possible you edit sys.path in process (perhaps in test setup)?

Even with Matplotlib 3.7, I'd have expected this to at least be warning, as that was deprecated in 3.6.

mpl_toolkits is a namespace package that we ship with Matplotlib (that is where the 3d and axis artist code live).

For 3.8 we did change to using the "new-style" version of doing namespace packages (see #25244 #25381) because upstream started to actively warn about using the old way.

It looks like all of these issues are in cases where pip install --local xxx is being used to shadow an existing Matplotlib installation from the system package manager with a version from pip. Although this works most of the time, I think that this is one of the corner cases where it does not.

What is the version of setuptools on the cases where it is broken? I strongly suspect that is going to be the determining factor of when this works or not.

Ah, yeah, the namespace package definition did also change, though I think that the changes to packaging are likely orthogonal here... setuptools has been stripping out the changes silently for quite a while, the fact that it is a namespace package is probably important though, since namespace packages can have multiple locations on a filesystem. I'd have thought they would resolve in sys.path order, though, and it seems like it may not be?

When I made the change, I found that setuptools had already been stripping out __init__.py from mpl_toolkits for wheels already, so I was pretty sure there would be no change.

I think that much is true, and that even from wheels/recent setuptools installs from previous versions, the OS version would still be selected for mpl_toolkits.

It is extremely important that every distribution that uses the namespace package omits the init.py or uses a pkgutil-style init.py. If any distribution does not, it will cause the namespace logic to fail and the other sub-packages will not be importable.

I had been under the impression that it was okay to have a mix of "present" and "absent" __init__.py, but that it was very much a problem to have non-matching __init__.py. But rereading that sentence from the Namespace packaging docs does make me question it, though it is slightly ambiguous usage of the word "or". I did do a variety of tests installing basemap and matplotlib together to make sure that things did not clash, though I don't think I did much with --local (I was using separate virtualenvs)

That said, setuptools is kind of forcing our hand here. Even if we did not make the change to be explicit about how we package it, setuptools was stripping it from the wheels, and they are deprecating the old style entirely. And the upgrade path is just not good... I suspect that people who install via --local have been getting old 3D code for years at this point...

I resolved the ImportError issue by following these steps:

Uninstalled the system-installed python3-matplotlib package:

sudo apt remove python3-matplotlib

Uninstalled the previously installed Matplotlib using pip:

pip uninstall matplotlib

Reinstalled Matplotlib via pip:

pip install matplotlib

After completing these steps, I can now import Matplotlib in Python without encountering any errors:

>>> import matplotlib.pyplot as plt

It's worth noting that while I resolved the issue, I don't remember the initial installation method, but I have been updating Matplotlib via pip since then.

I suspect that people who install via --local have been getting old 3D code for years at this point...

That would be my guess as well, and just that making doscstring private is now making old 3D incompatible with new matplotlib.

@reticulatus #26827 (comment)

How does mpl_toolkits get added to sys.modules

by /usr/lib/python3/dist-packages/matplotlib*nspkg.pth invoked from site.py.

To remove nspkg.pth, we need to install python3-matplotlib 3.8 debian package, but it is not available yet, so I created one.

https://github.com/cielavenir/salsa-matplotlib/releases/tag/ciel%2F3.8.3_1 (edit 20240312: simplified build process a bit)

@QuLogic #26827 (comment)

I can reproduce this in an Ubuntu container using:

$ podman run --rm -it ubuntu:22.04
# apt update -yy
# apt install python3-matplotlib python3-pip
# pip install --user -U matplotlib
# python3
>>> import matplotlib
>>> import mpl_toolkits.mplot3d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in <module>
    from .axes3d import Axes3D
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py", line 23, in <module>
    from matplotlib import _api, cbook, docstring, _preprocess_data
ImportError: cannot import name 'docstring' from 'matplotlib' (/root/.local/lib/python3.10/site-packages/matplotlib/__init__.py)

$ docker run --rm -it ubuntu:22.04
# apt update -yy
# # (copy) jammy directory
# cd jammy
# apt install ./python3-matplotlib_3.8.3-1_amd64.deb ./python-matplotlib-data_3.8.3-1_all.deb ./python3-numpy_1.26.3-1_amd64.deb ./python3-contourpy_1.0.7-1*_amd64.deb python3-pip
# pip install --user -U matplotlib
# python3
>>> import matplotlib
>>> mpl_toolkits.mplot3d

It does import.

This issue is blocking building docs for the debian packaging of matplotlib (3.8.0). The problem is as described by #26827 (comment) or #26827 (comment).

The issue is that mpl_toolkits from matplotlib 3.6.3 is installed as a system package (python3-matplotlib) in /usr/lib/python3/dist-packages/mpl_toolkits. The old version gets installed while building the new version because of a circular dependency via python3-sphinx-gallery

I've tried many variations to solve it: adding an empty mpl_toolkits/__init__.py; creating a local sitecustomize.py, or mpl_toolkits.pth to ensure the local path is set in sys.path before /usr/lib/python3/dist-packages/; or using virtualenv (with --system-site-packages). With sitecustomize.py, for instance, I can see that sys.path got modified as intended. The local path for the new build is placed before the system path. Indeed I can successfully import matplotlib 3.8.0 itself from the local path.

But every attempt to import the new mpl_toolkits (or mpl_toolkits.mplot3d) from the local path fails.
In every case it imports from /usr/lib/python3/dist-packages/mpl_toolkits instead of the local path.

Do you need python3-sphinx-gallery to build matplotlib? I dont think so

The docs, man, the docs.

@jaraco we do seem to be having some regression that may or may not be related to Matplotlib moving to new-style namespace packages. Is there a better fix for this?

Sorry for the long delay in responding. I'd fully intended to follow up. I don't remember why I did not. The recent activity revived my awareness of the issue.

Following the repro instructions, I'm able to replicate the failure with this Dockerfile:

from ubuntu:22.04
ENV DEBIAN_FRONTEND=noninteractive
run apt update -yy
run apt install -y python3-matplotlib python3-pip
run pip install --user -U matplotlib
cmd python3 -i -c 'import matplotlib; import mpl_toolkits.mplot3d'

I can see now that the debian-installed matplotlib is using the pkg_resources-style namespace package for mpl_toolkits but the 3.8 pip-installed version is using the native namespace packages. As you've discovered, these two forms of namespace packages are incompatible within the same environment. It doesn't matter whether mpl_toolkits is installed by an older install in dist-packages or created by some other installer elsewhere on PYTHONPATH/sys.path, if one of the installs is using pkg_resources-style namespace packages, it will take precedence over other installs, even if they appear earlier on the path.

pkg_resources-style packages have two ways of declaring themselves, once early with the -nspkg.pth file and again with the declaration in the __init__.py of the namespace.

The reason adding the __init__.py to mpl_toolkits works around the issue is because it's turning that namespace package into a normal package and causing it to take precedence over any mpl_toolkits in dist-packages (later in the path).

Since matplotlib is migrating, it means that dual-installations will be incompatible across the 3.7/3.8 boundary, so users will need to somehow keep these versions isolated. Probably the two easiest ways to achieve this isolation is to use a virtualenv for the newer installation or to uninstall the older version from dist-packages. Upgrading the package-installed version also looks to be a viable alternative. Deleting the -nspth.pkg and mpl_toolkits/init.py from dist-packages should also work, though I'd feel uneasy about that, given that those files are managed by the package manager.

I wish I had a better suggestion. If we could go back in time, one way to ease the transition is to use pkgutil-style namespace packages. If mpl_toolkits could have transitioned to that in the late 2010s, it has better compatibility with both pkg_resources-style and native namespace packages. Now that mpl_toolkits is using native packages, it's probably too late to consider a transitional period.

What is the version of setuptools on the cases where it is broken? I strongly suspect that is going to be the determining factor of when this works or not.

There's no version of setuptools that fixes this issue. It's an inherent limitation in the design.

As far as the doc build goes, the blocking problem is specifically with the autosummary generated for doc/api/toolkits/mplot3d/axes3d.rst. That is, the warning about the version mismatch with mpl_toolkits.mplot3d is emitted for other pages, but sphinx can ignore the warning and carry on. With axes3d.rst however, sphinx gets an unrecoverable exception and halts.

I can get a "successful" doc built by removing the autoclass:: Axes3D and autosummary:: sections from axes3d.rst. Loses linking for axes3d.Axes3D, but other pages are fine. Also needed to skip 3d, axes, random_walk examples from the gallery, replacing l.235 in doc/conf.py with

    'filename_pattern': '^((?!sgskip|3d|3D|axes|random_walk).)*$',

I'll apply that workaround to get the docs building for Debian's python-matplotlib-doc.

For me the solution was to sudo apt remove python3-matplotlib. You can indicate that matplotlib, and python3-matplotlib different packages in the error stack, because I did not know it, and many others probably won't too.

yeah, removing python3-matplotlib works, but in my case freecad had dependency, so installing python3-matplotlib 3.8 was the only solution for me.

I resolved the ImportError issue by following these steps:

Uninstalled the system-installed python3-matplotlib package:
sudo apt remove python3-matplotlib
Uninstalled the previously installed Matplotlib using pip:
pip uninstall matplotlib
Reinstalled Matplotlib via pip:
pip install matplotlib
After completing these steps, I can now import Matplotlib in Python without encountering any errors:
>>> import matplotlib.pyplot as plt
It's worth noting that while I resolved the issue, I don't remember the initial installation method, but I have been updating Matplotlib via pip since then.

Works!!! Thanks a lot!!!

I think I tracked down the cause of this on my local machine.

When debugging in VSCode on WSL, there are multiple paths that point towards the same file. One is the local linux filepath (./matplotlib), and one is the file on the windows filesystem as mounted on the linux subsystem (/mnt/c/.../matplotlib). If you set a breakpoint in the former, then the debugger will sometimes open the latter to step through the code. But if you then set a breakpoint in the latter, then something gets confused and thinks there are multiple installations, leading to the error here.

The solution (if I'm right about this) was to go through my breakpoints and make sure you aren't splitting them across files on the separate paths.

DOC: document the issues with overlaying new mpl on old mpl

Per #26827 (comment) we went across a hard compatibly edge with mpl 3.7 -> 3.8 and having overlapping installs of anything < 3.8 and anything >= 3.8 is going to have this issue.

I've opened a PR ( #29673 ) add this to the API changes for 3.8 to close this issue. I don't think there is anything we can do at this point that would not be as or more disruptive than that status quo.

[test] add tests for m2v, mesh2vol, mesh2mask, mask has bias

[Bug]: ImportError when using Matplotlib v3.8.0 in Python package tests · Issue #26827 · matplotlib/matplotlib · GitHub

Repository navigation

[Bug]: ImportError when using Matplotlib v3.8.0 in Python package tests #26827

Bug summary

Code for reproduction

Actual outcome

Expected outcome

Additional information

Operating system

Matplotlib Version

Matplotlib Backend

Python version

Jupyter version

Installation

26 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: ImportError when using Matplotlib v3.8.0 in Python package tests · Issue #26827 · matplotlib/matplotlib · GitHub

Repository navigation

[Bug]: ImportError when using Matplotlib v3.8.0 in Python package tests #26827

Description

Bug summary

Code for reproduction

Actual outcome

Expected outcome

Additional information

Operating system

Matplotlib Version

Matplotlib Backend

Python version

Jupyter version

Installation

Activity

WolfgangFahl commented on Sep 19, 2023

jklymak commented on Sep 19, 2023

ksunden commented on Sep 19, 2023

ksunden commented on Sep 19, 2023

WolfgangFahl commented on Sep 19, 2023

WolfgangFahl commented on Sep 19, 2023

ksunden commented on Sep 19, 2023

tacaswell commented on Sep 19, 2023

ksunden commented on Sep 19, 2023

ksunden commented on Sep 19, 2023

jvsguerra commented on Sep 19, 2023

jklymak commented on Sep 19, 2023

26 remaining items

cielavenir commented on Feb 29, 2024

drew-parsons commented on Jun 23, 2024

cielavenir commented on Jun 24, 2024

drew-parsons commented on Jun 24, 2024

jaraco commented on Jun 24, 2024

drew-parsons commented on Jun 24, 2024

MRiabov commented on Jul 13, 2024

cielavenir commented on Jul 15, 2024

DimitriBolt commented on Aug 3, 2024

scottshambaugh commented on Jan 2, 2025

tacaswell commented on Feb 24, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions