<divid="unreleased-message"> You are reading an old version of the documentation (v3.4.1). For the latest version see <ahref="https://matplotlib.org/stable/devel/testing.html">https://matplotlib.org/stable/devel/testing.html</a></div>
<spanid="id1"></span><h1>Testing<aclass="headerlink" href="#testing" title="Permalink to this headline">¶</a></h1>
<p>Matplotlib uses the <aclass="reference external" href="http://doc.pytest.org/en/latest/">pytest</a> framework.</p>
<p>The tests are in <codeclass="file docutils literal notranslate"><spanclass="pre">lib/matplotlib/tests</span></code>, and customizations to the pytest
testing infrastructure are in <aclass="reference internal" href="../api/testing_api.html#module-matplotlib.testing" title="matplotlib.testing"><codeclass="xref py py-mod docutils literal notranslate"><spanclass="pre">matplotlib.testing</span></code></a>.</p>
<divclass="section" id="requirements">
<spanid="testing-requirements"></span><h2>Requirements<aclass="headerlink" href="#requirements" title="Permalink to this headline">¶</a></h2>
<p>To run the tests you will need to
<aclass="reference internal" href="development_setup.html#installing-for-devs"><spanclass="std std-ref">set up Matplotlib for development</span></a>. Note in
particular the <aclass="reference internal" href="development_setup.html#test-dependencies"><spanclass="std std-ref">additional dependencies</span></a> for testing.</p>
<divclass="admonition note">
<pclass="first admonition-title">Note</p>
<p>We will assume that you want to run the tests in a development setup.</p>
<pclass="last">While you can run the tests against a regular installed version of
Matplotlib, this is a far less common use case. You still need the
<aclass="reference internal" href="development_setup.html#test-dependencies"><spanclass="std std-ref">additional dependencies</span></a> for testing.
You have to additionally get the reference images from the repository,
because they are not distributed with pre-built Matplotlib packages.</p>
</div>
</div>
<divclass="section" id="running-the-tests">
<h2>Running the tests<aclass="headerlink" href="#running-the-tests" title="Permalink to this headline">¶</a></h2>
<p>In the root directory of your development repository run:</p>
<p>pytest can be configured via a lot of <aclass="reference external" href="http://doc.pytest.org/en/latest/usage.html">command-line parameters</a>. Some
particularly useful ones are:</p>
<tableborder="1" class="docutils align-default">
<colgroup>
<colwidth="46%" />
<colwidth="54%" />
</colgroup>
<tbodyvalign="top">
<trclass="row-odd"><td><codeclass="docutils literal notranslate"><spanclass="pre">-v</span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">--verbose</span></code></td>
<p>Pytest determines which functions are tests by searching for files whose names
begin with <codeclass="docutils literal notranslate"><spanclass="pre">"test_"</span></code> and then within those files for functions beginning with
<codeclass="docutils literal notranslate"><spanclass="pre">"test"</span></code> or classes beginning with <codeclass="docutils literal notranslate"><spanclass="pre">"Test"</span></code>.</p>
<p>Some tests have internal side effects that need to be cleaned up after their
execution (such as created figures or modified <aclass="reference internal" href="../api/matplotlib_configuration_api.html#matplotlib.rcParams" title="matplotlib.rcParams"><codeclass="xref py py-obj docutils literal notranslate"><spanclass="pre">rcParams</span></code></a>). The pytest fixture
<codeclass="docutils literal notranslate"><spanclass="pre">matplotlib.testing.conftest.mpl_test_settings</span></code> will automatically clean
these up; there is no need to do anything further.</p>
</div>
<divclass="section" id="random-data-in-tests">
<h2>Random data in tests<aclass="headerlink" href="#random-data-in-tests" title="Permalink to this headline">¶</a></h2>
<p>Random data is a very convenient way to generate data for examples,
however the randomness is problematic for testing (as the tests
must be deterministic!). To work around this set the seed in each test.
<p>The first time this test is run, there will be no baseline image to compare
against, so the test will fail. Copy the output images (in this case
<codeclass="file docutils literal notranslate"><spanclass="pre">result_images/test_lines/test_line_dashes.png</span></code>) to the correct
subdirectory of <codeclass="file docutils literal notranslate"><spanclass="pre">baseline_images</span></code> tree in the source directory (in this
case <codeclass="file docutils literal notranslate"><spanclass="pre">lib/matplotlib/tests/baseline_images/test_lines</span></code>). Put this new
file under source code revision control (with <codeclass="docutils literal notranslate"><spanclass="pre">git</span><spanclass="pre">add</span></code>). When rerunning
the tests, they should now pass.</p>
<p>Baseline images take a lot of space in the Matplotlib repository.
An alternative approach for image comparison tests is to use the
<aclass="reference internal" href="../api/testing_api.html#matplotlib.testing.decorators.check_figures_equal" title="matplotlib.testing.decorators.check_figures_equal"><codeclass="xref py py-obj docutils literal notranslate"><spanclass="pre">check_figures_equal</span></code></a> decorator, which should be
used to decorate a function taking two <aclass="reference internal" href="../api/figure_api.html#matplotlib.figure.Figure" title="matplotlib.figure.Figure"><codeclass="xref py py-obj docutils literal notranslate"><spanclass="pre">Figure</span></code></a> parameters and draws the same
images on the figures using two different methods (the tested method and the
baseline method). The decorator will arrange for setting up the figures and
then collect the drawn results and compare them.</p>
<p>See the documentation of <aclass="reference internal" href="../api/testing_api.html#matplotlib.testing.decorators.image_comparison" title="matplotlib.testing.decorators.image_comparison"><codeclass="xref py py-obj docutils literal notranslate"><spanclass="pre">image_comparison</span></code></a> and
<aclass="reference internal" href="../api/testing_api.html#matplotlib.testing.decorators.check_figures_equal" title="matplotlib.testing.decorators.check_figures_equal"><codeclass="xref py py-obj docutils literal notranslate"><spanclass="pre">check_figures_equal</span></code></a> for additional information
<h2>Creating a new module in matplotlib.tests<aclass="headerlink" href="#creating-a-new-module-in-matplotlib-tests" title="Permalink to this headline">¶</a></h2>
<p>We try to keep the tests categorized by the primary module they are
testing. For example, the tests related to the <codeclass="docutils literal notranslate"><spanclass="pre">mathtext.py</span></code> module
are in <codeclass="docutils literal notranslate"><spanclass="pre">test_mathtext.py</span></code>.</p>
<h2>Using GitHub Actions for CI<aclass="headerlink" href="#using-github-actions-for-ci" title="Permalink to this headline">¶</a></h2>
<p><aclass="reference external" href="https://docs.github.com/en/actions">GitHub Actions</a> is a hosted CI system
"in the cloud".</p>
<p>GitHub Actions is configured to receive notifications of new commits to GitHub
repos and to run builds or tests when it sees these new commits. It looks for a
YAML files in <codeclass="docutils literal notranslate"><spanclass="pre">.github/workflows</span></code> to see how to test the project.</p>
<p>GitHub Actions is already enabled for the <aclass="reference external" href="https://github.com/matplotlib/matplotlib/">main Matplotlib GitHub repository</a> -- for example, see <aclass="reference external" href="https://github.com/matplotlib/matplotlib/actions?query=workflow%3ATests">the Tests
workflows</a>.</p>
<p>GitHub Actions should be automatically enabled for your personal Matplotlib
fork once the YAML workflow files are in it. It generally isn't necessary to
look at these workflows, since any pull request submitted against the main
<p>Tox is configured using a file called <codeclass="docutils literal notranslate"><spanclass="pre">tox.ini</span></code>. You may need to
edit this file if you want to add new environments to test (e.g.,
<codeclass="docutils literal notranslate"><spanclass="pre">py33</span></code>) or if you want to tweak the dependencies or the way the
tests are run. For more info on the <codeclass="docutils literal notranslate"><spanclass="pre">tox.ini</span></code> file, see the <aclass="reference external" href="https://tox.readthedocs.io/en/latest/config.html">Tox
<h2>Building old versions of Matplotlib<aclass="headerlink" href="#building-old-versions-of-matplotlib" title="Permalink to this headline">¶</a></h2>
<p>When running a <codeclass="docutils literal notranslate"><spanclass="pre">git</span><spanclass="pre">bisect</span></code> to see which commit introduced a certain bug,
you may (rarely) need to build very old versions of Matplotlib. The following
<h2>Testing released versions of Matplotlib<aclass="headerlink" href="#testing-released-versions-of-matplotlib" title="Permalink to this headline">¶</a></h2>
<p>Running the tests on an installation of a released version (e.g. PyPI package
or conda package) also requires additional setup.</p>
<divclass="admonition note">
<pclass="first admonition-title">Note</p>
<pclass="last">For an end-user, there is usually no need to run the tests on released
versions of Matplotlib. Official releases are tested before publishing.</p>
<h3>Install additional dependencies<aclass="headerlink" href="#install-additional-dependencies" title="Permalink to this headline">¶</a></h3>
<p>Install the <aclass="reference internal" href="development_setup.html#test-dependencies"><spanclass="std std-ref">additional dependencies for testing</span></a>.</p>
<h3>Obtain the reference images<aclass="headerlink" href="#obtain-the-reference-images" title="Permalink to this headline">¶</a></h3>
<p>Many tests compare the plot result against reference images. The reference
images are not part of the regular packaged versions (pip wheels or conda
packages). If you want to run tests with reference images, you need to obtain
the reference images matching the version of Matplotlib you want to test.</p>
<p>To do so, either download the matching source distribution
<codeclass="docutils literal notranslate"><spanclass="pre">matplotlib-X.Y.Z.tar.gz</span></code> from <aclass="reference external" href="https://pypi.org/project/matplotlib/">PyPI</a>
or alternatively, clone the git repository and <codeclass="docutils literal notranslate"><spanclass="pre">git</span><spanclass="pre">checkout</span><spanclass="pre">vX.Y.Z</span></code>. Copy
the folder <codeclass="file docutils literal notranslate"><spanclass="pre">lib/matplotlib/tests/baseline_images</span></code> to the folder
<codeclass="file docutils literal notranslate"><spanclass="pre">matplotlib/tests</span></code> of your the matplotlib installation to test.