<divid="unreleased-message"> You are reading an old version of the documentation (v2.1.0). For the latest version see <ahref="https://matplotlib.org/stable/devel/testing.html">https://matplotlib.org/stable/devel/testing.html</a></div>
<spanid="testing"></span><h1>Developer’s tips for testing<aclass="headerlink" href="#developer-s-tips-for-testing" title="Permalink to this headline">¶</a></h1>
<p>Matplotlib’s testing infrastructure depends on <aclass="reference external" href="http://doc.pytest.org/en/latest/">pytest</a>. The tests are in
<codeclass="file docutils literal"><spanclass="pre">lib/matplotlib/tests</span></code>, and customizations to the pytest testing
infrastructure are in <codeclass="xref py py-mod docutils literal"><spanclass="pre">matplotlib.testing</span></code>.</p>
<divclass="section" id="requirements">
<h2>Requirements<aclass="headerlink" href="#requirements" title="Permalink to this headline">¶</a></h2>
<p>The following software is required to run the tests:</p>
<blockquote>
<div><ulclass="simple">
<li><aclass="reference external" href="http://doc.pytest.org/en/latest/">pytest</a>, version 3.0.0 or later</li>
<li><aclass="reference external" href="https://docs.python.org/3/library/unittest.mock.html>">mock</a>, when running Python versions < 3.3</li>
<li><aclass="reference external" href="https://www.ghostscript.com/">Ghostscript</a> (to render PDF files)</li>
<h2>Building matplotlib for image comparison tests<aclass="headerlink" href="#building-matplotlib-for-image-comparison-tests" title="Permalink to this headline">¶</a></h2>
<p>matplotlib’s test suite makes heavy use of image comparison tests,
meaning the result of a plot is compared against a known good result.
Unfortunately, different versions of FreeType produce differently
formed characters, causing these image comparisons to fail. To make
them reproducible, matplotlib can be built with a special local copy
of FreeType. This is recommended for all matplotlib developers.</p>
<p>Add the following content to a <codeclass="docutils literal"><spanclass="pre">setup.cfg</span></code> file at the root of the
<td>Disable tests that require network access</td>
</tr>
</tbody>
</table>
<p>Additional arguments are passed on to pytest. See the pytest documentation for
<aclass="reference external" href="http://doc.pytest.org/en/latest/usage.html">supported arguments</a>. Some of the more important ones are given here:</p>
process (requires <aclass="reference external" href="https://pypi.python.org/pypi/pytest-timeout">pytest-timeout</a>)</td>
</tr>
<trclass="row-even"><td><codeclass="docutils literal"><spanclass="pre">--capture=no</span></code> or <codeclass="docutils literal"><spanclass="pre">-s</span></code></td>
<td>Do not capture stdout</td>
</tr>
</tbody>
</table>
<p>To run a single test from the command line, you can provide a file path,
optionally followed by the function separated by two colons, e.g., (tests do
not need to be installed, but Matplotlib should be):</p>
<p>Pytest determines which functions are tests by searching for files whose names
begin with <codeclass="docutils literal"><spanclass="pre">"test_"</span></code> and then within those files for functions beginning with
<codeclass="docutils literal"><spanclass="pre">"test"</span></code> or classes beginning with <codeclass="docutils literal"><spanclass="pre">"Test"</span></code>.</p>
<p>Some tests have internal side effects that need to be cleaned up after their
execution (such as created figures or modified rc params). The pytest fixture
<codeclass="xref py py-func docutils literal"><spanclass="pre">mpl_test_settings()</span></code> will automatically clean
these up; there is no need to do anything further.</p>
</div>
<divclass="section" id="random-data-in-tests">
<h2>Random data in tests<aclass="headerlink" href="#random-data-in-tests" title="Permalink to this headline">¶</a></h2>
<p>Random data can is a very convenient way to generate data for examples,
however the randomness is problematic for testing (as the tests
must be deterministic!). To work around this set the seed in each test.
<p>The first time this test is run, there will be no baseline image to
compare against, so the test will fail. Copy the output images (in
this case <codeclass="xref py py-obj docutils literal"><spanclass="pre">result_images/test_category/spines_axes_positions.png</span></code>) to
the correct subdirectory of <codeclass="xref py py-obj docutils literal"><spanclass="pre">baseline_images</span></code> tree in the source
directory (in this case
<codeclass="xref py py-obj docutils literal"><spanclass="pre">lib/matplotlib/tests/baseline_images/test_category</span></code>). Put this new
file under source code revision control (with <codeclass="xref py py-obj docutils literal"><spanclass="pre">git</span><spanclass="pre">add</span></code>). When
defaults to generating <codeclass="docutils literal"><spanclass="pre">png</span></code>, <codeclass="docutils literal"><spanclass="pre">pdf</span></code> and <codeclass="docutils literal"><spanclass="pre">svg</span></code> output, but in
interest of keeping the size of the library from ballooning we should only
include the <codeclass="docutils literal"><spanclass="pre">svg</span></code> or <codeclass="docutils literal"><spanclass="pre">pdf</span></code> outputs if the test is explicitly exercising
a feature dependent on that backend.</p>
<p>There are two optional keyword arguments to the <codeclass="xref py py-obj docutils literal"><spanclass="pre">image_comparison</span></code>
decorator:</p>
<blockquote>
<div><ulclass="simple">
<li><codeclass="xref py py-obj docutils literal"><spanclass="pre">extensions</span></code>: If you only wish to test additional image formats
(rather than just <codeclass="xref py py-obj docutils literal"><spanclass="pre">png</span></code>), pass any additional file types in the
list of the extensions to test. When copying the new
baseline files be sure to only copy the output files, not their
conversions to <codeclass="docutils literal"><spanclass="pre">png</span></code>. For example only copy the files
ending in <codeclass="docutils literal"><spanclass="pre">pdf</span></code>, not in <codeclass="docutils literal"><spanclass="pre">_pdf.png</span></code>.</li>
<li><codeclass="xref py py-obj docutils literal"><spanclass="pre">tol</span></code>: This is the image matching tolerance, the default <codeclass="xref py py-obj docutils literal"><spanclass="pre">1e-3</span></code>.
If some variation is expected in the image between runs, this
value may be adjusted.</li>
</ul>
</div></blockquote>
</div>
<divclass="section" id="known-failing-tests">
<h2>Known failing tests<aclass="headerlink" href="#known-failing-tests" title="Permalink to this headline">¶</a></h2>
<p>If you’re writing a test, you may mark it as a known failing test with the
<codeclass="xref py py-func docutils literal"><spanclass="pre">pytest.mark.xfail()</span></code> decorator. This allows the test to be added to the
test suite and run on the buildbots without causing undue alarm. For example,
although the following test will fail, it is an expected failure:</p>
<h2>Creating a new module in matplotlib.tests<aclass="headerlink" href="#creating-a-new-module-in-matplotlib-tests" title="Permalink to this headline">¶</a></h2>
<p>We try to keep the tests categorized by the primary module they are
testing. For example, the tests related to the <codeclass="docutils literal"><spanclass="pre">mathtext.py</span></code> module
are in <codeclass="docutils literal"><spanclass="pre">test_mathtext.py</span></code>.</p>
</div>
<divclass="section" id="using-travis-ci">
<h2>Using Travis CI<aclass="headerlink" href="#using-travis-ci" title="Permalink to this headline">¶</a></h2>
<p><aclass="reference external" href="https://travis-ci.org/">Travis CI</a> is a hosted CI system “in the
cloud”.</p>
<p>Travis is configured to receive notifications of new commits to GitHub
repos (via GitHub “service hooks”) and to run builds or tests when it
sees these new commits. It looks for a YAML file called
<codeclass="docutils literal"><spanclass="pre">.travis.yml</span></code> in the root of the repository to see how to test the
project.</p>
<p>Travis CI is already enabled for the <aclass="reference external" href="https://github.com/matplotlib/matplotlib/">main matplotlib GitHub
repository</a> – for
example, see <aclass="reference external" href="https://travis-ci.org/matplotlib/matplotlib">its Travis page</a>.</p>
<p>If you want to enable Travis CI for your personal matplotlib GitHub
repo, simply enable the repo to use Travis CI in either the Travis CI
UI or the GitHub UI (Admin | Service Hooks). For details, see <aclass="reference external" href="https://docs.travis-ci.com/user/getting-started/">the
Travis CI Getting Started page</a>. This
generally isn’t necessary, since any pull request submitted against
the main matplotlib repository will be tested.</p>
<p>Once this is configured, you can see the Travis CI results at
<p>Tox is configured using a file called <codeclass="docutils literal"><spanclass="pre">tox.ini</span></code>. You may need to
edit this file if you want to add new environments to test (e.g.,
<codeclass="docutils literal"><spanclass="pre">py33</span></code>) or if you want to tweak the dependencies or the way the
tests are run. For more info on the <codeclass="docutils literal"><spanclass="pre">tox.ini</span></code> file, see the <aclass="reference external" href="https://tox.readthedocs.io/en/latest/config.html">Tox