<divid="unreleased-message"> You are reading an old version of the documentation (v3.3.2). For the latest version see <ahref="https://matplotlib.org/stable/devel/testing.html">https://matplotlib.org/stable/devel/testing.html</a></div>
<spanid="testing"></span><h1>Developer's tips for testing<aclass="headerlink" href="#developer-s-tips-for-testing" title="Permalink to this headline">¶</a></h1>
<p>Matplotlib's testing infrastructure depends on <aclass="reference external" href="http://doc.pytest.org/en/latest/">pytest</a>. The tests are in
<codeclass="file docutils literal notranslate"><spanclass="pre">lib/matplotlib/tests</span></code>, and customizations to the pytest testing
infrastructure are in <aclass="reference internal" href="../api/testing_api.html#module-matplotlib.testing" title="matplotlib.testing"><codeclass="xref py py-mod docutils literal notranslate"><spanclass="pre">matplotlib.testing</span></code></a>.</p>
<divclass="section" id="requirements">
<h2>Requirements<aclass="headerlink" href="#requirements" title="Permalink to this headline">¶</a></h2>
<p>Install the latest version of Matplotlib as documented in
<aclass="reference internal" href="contributing.html#installing-for-devs"><spanclass="std std-ref">Retrieving and installing the latest version of the code</span></a>.</p>
<p>The following software is required to run the tests:</p>
<li><aclass="reference external" href="https://pytest-cov.readthedocs.io/en/latest/">pytest-cov</a> (>=2.3.1) to collect coverage information</li>
<li><aclass="reference external" href="https://pypi.org/project/pytest-flake8/">pytest-flake8</a> to test coding standards using <aclass="reference external" href="https://pypi.org/project/flake8/">flake8</a></li>
<li><aclass="reference external" href="https://pypi.org/project/pytest-timeout/">pytest-timeout</a> to limit runtime in case of stuck tests</li>
<li><aclass="reference external" href="https://pypi.org/project/pytest-xdist/">pytest-xdist</a> to run tests in parallel</li>
</ul>
</div>
<divclass="section" id="running-the-tests">
<h2>Running the tests<aclass="headerlink" href="#running-the-tests" title="Permalink to this headline">¶</a></h2>
<p>Running the tests is simple. Make sure you have pytest installed and run:</p>
<p>pytest can be configured via a lot of <aclass="reference external" href="http://doc.pytest.org/en/latest/usage.html">command-line parameters</a>. Some
particularly useful ones are:</p>
<tableborder="1" class="docutils align-default">
<colgroup>
<colwidth="43%" />
<colwidth="57%" />
</colgroup>
<tbodyvalign="top">
<trclass="row-odd"><td><codeclass="docutils literal notranslate"><spanclass="pre">-v</span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">--verbose</span></code></td>
<p>Pytest determines which functions are tests by searching for files whose names
begin with <codeclass="docutils literal notranslate"><spanclass="pre">"test_"</span></code> and then within those files for functions beginning with
<codeclass="docutils literal notranslate"><spanclass="pre">"test"</span></code> or classes beginning with <codeclass="docutils literal notranslate"><spanclass="pre">"Test"</span></code>.</p>
<p>Some tests have internal side effects that need to be cleaned up after their
execution (such as created figures or modified <aclass="reference internal" href="../api/matplotlib_configuration_api.html#matplotlib.rcParams" title="matplotlib.rcParams"><codeclass="xref py py-obj docutils literal notranslate"><spanclass="pre">rcParams</span></code></a>). The pytest fixture
<codeclass="xref py py-func docutils literal notranslate"><spanclass="pre">mpl_test_settings()</span></code> will automatically clean
these up; there is no need to do anything further.</p>
</div>
<divclass="section" id="random-data-in-tests">
<h2>Random data in tests<aclass="headerlink" href="#random-data-in-tests" title="Permalink to this headline">¶</a></h2>
<p>Random data is a very convenient way to generate data for examples,
however the randomness is problematic for testing (as the tests
must be deterministic!). To work around this set the seed in each test.
<p>The first time this test is run, there will be no baseline image to compare
against, so the test will fail. Copy the output images (in this case
<codeclass="file docutils literal notranslate"><spanclass="pre">result_images/test_lines/test_line_dashes.png</span></code>) to the correct
subdirectory of <codeclass="file docutils literal notranslate"><spanclass="pre">baseline_images</span></code> tree in the source directory (in this
case <codeclass="file docutils literal notranslate"><spanclass="pre">lib/matplotlib/tests/baseline_images/test_lines</span></code>). Put this new
file under source code revision control (with <codeclass="docutils literal notranslate"><spanclass="pre">git</span><spanclass="pre">add</span></code>). When rerunning
the tests, they should now pass.</p>
<p>Baseline images take a lot of space in the Matplotlib repository.
An alternative approach for image comparison tests is to use the
<aclass="reference internal" href="../api/testing_api.html#matplotlib.testing.decorators.check_figures_equal" title="matplotlib.testing.decorators.check_figures_equal"><codeclass="xref py py-obj docutils literal notranslate"><spanclass="pre">check_figures_equal</span></code></a> decorator, which should be
used to decorate a function taking two <aclass="reference internal" href="../api/_as_gen/matplotlib.figure.Figure.html#matplotlib.figure.Figure" title="matplotlib.figure.Figure"><codeclass="xref py py-obj docutils literal notranslate"><spanclass="pre">Figure</span></code></a> parameters and draws the same
images on the figures using two different methods (the tested method and the
baseline method). The decorator will arrange for setting up the figures and
then collect the drawn results and compare them.</p>
<p>See the documentation of <aclass="reference internal" href="../api/testing_api.html#matplotlib.testing.decorators.image_comparison" title="matplotlib.testing.decorators.image_comparison"><codeclass="xref py py-obj docutils literal notranslate"><spanclass="pre">image_comparison</span></code></a> and
<aclass="reference internal" href="../api/testing_api.html#matplotlib.testing.decorators.check_figures_equal" title="matplotlib.testing.decorators.check_figures_equal"><codeclass="xref py py-obj docutils literal notranslate"><spanclass="pre">check_figures_equal</span></code></a> for additional information
about their use.</p>
</div>
<divclass="section" id="known-failing-tests">
<h2>Known failing tests<aclass="headerlink" href="#known-failing-tests" title="Permalink to this headline">¶</a></h2>
<p>If you're writing a test, you may mark it as a known failing test with the
<aclass="reference external" href="https://pytest.org/en/stable/reference.html#pytest.mark.xfail" title="(in pytest v6.0.2)"><codeclass="xref py py-func docutils literal notranslate"><spanclass="pre">pytest.mark.xfail()</span></code></a> decorator. This allows the test to be added to the
test suite and run on the buildbots without causing undue alarm. For example,
although the following test will fail, it is an expected failure:</p>
<p>Note that the first argument to the <aclass="reference external" href="https://pytest.org/en/stable/reference.html#pytest.mark.xfail" title="(in pytest v6.0.2)"><codeclass="xref py py-func docutils literal notranslate"><spanclass="pre">xfail()</span></code></a> decorator is a
fail condition, which can be a value such as True, False, or may be a
dynamically evaluated expression. If a condition is supplied, then a reason
must also be supplied with the <codeclass="docutils literal notranslate"><spanclass="pre">reason='message'</span></code> keyword argument.</p>
<h2>Creating a new module in matplotlib.tests<aclass="headerlink" href="#creating-a-new-module-in-matplotlib-tests" title="Permalink to this headline">¶</a></h2>
<p>We try to keep the tests categorized by the primary module they are
testing. For example, the tests related to the <codeclass="docutils literal notranslate"><spanclass="pre">mathtext.py</span></code> module
are in <codeclass="docutils literal notranslate"><spanclass="pre">test_mathtext.py</span></code>.</p>
</div>
<divclass="section" id="using-travis-ci">
<h2>Using Travis CI<aclass="headerlink" href="#using-travis-ci" title="Permalink to this headline">¶</a></h2>
<p><aclass="reference external" href="https://travis-ci.com/">Travis CI</a> is a hosted CI system "in the
cloud".</p>
<p>Travis is configured to receive notifications of new commits to GitHub
repos (via GitHub "service hooks") and to run builds or tests when it
sees these new commits. It looks for a YAML file called
<codeclass="docutils literal notranslate"><spanclass="pre">.travis.yml</span></code> in the root of the repository to see how to test the
project.</p>
<p>Travis CI is already enabled for the <aclass="reference external" href="https://github.com/matplotlib/matplotlib/">main Matplotlib GitHub
repository</a> -- for
example, see <aclass="reference external" href="https://travis-ci.com/matplotlib/matplotlib">its Travis page</a>.</p>
<p>If you want to enable Travis CI for your personal Matplotlib GitHub
repo, simply enable the repo to use Travis CI in either the Travis CI
UI or the GitHub UI (Admin | Service Hooks). For details, see <aclass="reference external" href="https://docs.travis-ci.com/user/getting-started/">the
Travis CI Getting Started page</a>. This
generally isn't necessary, since any pull request submitted against
the main Matplotlib repository will be tested.</p>
<p>Once this is configured, you can see the Travis CI results at
<p>Tox is configured using a file called <codeclass="docutils literal notranslate"><spanclass="pre">tox.ini</span></code>. You may need to
edit this file if you want to add new environments to test (e.g.,
<codeclass="docutils literal notranslate"><spanclass="pre">py33</span></code>) or if you want to tweak the dependencies or the way the
tests are run. For more info on the <codeclass="docutils literal notranslate"><spanclass="pre">tox.ini</span></code> file, see the <aclass="reference external" href="https://tox.readthedocs.io/en/latest/config.html">Tox
<h2>Building old versions of Matplotlib<aclass="headerlink" href="#building-old-versions-of-matplotlib" title="Permalink to this headline">¶</a></h2>
<p>When running a <codeclass="docutils literal notranslate"><spanclass="pre">git</span><spanclass="pre">bisect</span></code> to see which commit introduced a certain bug,
you may (rarely) need to build very old versions of Matplotlib. The following