MAINT: Deterministic SVG and PDF tests by jkseppan · Pull Request #7748 · matplotlib/matplotlib · GitHub

There has been plenty of work (at least #4434, #5671, #5766, #6597) to make it possible to have deterministic svg and pdf output, or at least closer to deterministic than before. This change adds one related feature: callers can remove the bits of pdf metadata that contain the date and matplotlib version, so that new releases don't necessarily change the result of every test. Furthermore, this change enables all the optional pdf and svg determinism features in tests.

I am developing some improvements to test-result caching on another branch, and in my tests only the following test files still change from run to run after this patch (although in my environment I get KNOWNFAIL=27, SKIP=8 so some more are possible):

clip_path_clipping.svg
hatching.svg
pgf_bbox_inches.pdf
pgf_mixedmode.pdf
pgf_pdflatex.pdf
pgf_rcupdate1.pdf
pgf_rcupdate2.pdf
pgf_xelatex.pdf
bbox_inches_tight_clipping.svg
image_clip.svg
scaled_lines.svg
offsetbox_clipping.svg
skew_axes.pdf
skew_axes.svg

And document the standard keys in PdfPages. Also insert the matplotlib version in the default value of Producer as well as Creator, to retain a debugging clue in case the user overrides one of them.

Remove the metadata entries that depend on the date and the exact version.

Current coverage is 62.12% (diff: 100%)

Merging #7748 into master will increase coverage by <.01%

@@             master      #7748   diff @@
==========================================
  Files           174        174          
  Lines         56028      56028          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          34805      34806     +1   
+ Misses        21223      21222     -1   
  Partials          0          0

Powered by Codecov. Last update 1fa4dd7...eab297c

Doesn't this break the test of SOURCE_DATE_EPOCH ?

No, that test starts a separate subprocess to write the output and doesn't use the image_comparison decorator at all.

Ah, 🐑

the pgf tests roll their own image comparison code.

attn @astrofrog this may need to be ported to pytest-mpl?

This will also break, in a way, the determinism test.

Can you explain in more detail? The _test_determinism_save function in test_backend_svg.py file also sets this rc parameter to a constant value.

Probably along the same lines as @tacaswell; I was thinking it wouldn't fail if svg.hashsalt was broken. However, I see this test doesn't use the decorator anyway, so it is likely to still fail correctly.

There is a cache of png files keyed by the MD5 hashes of corresponding svg and pdf files, which helps reduce test suite running times for svg and pdf files that stay exactly the same from one run to the next. This patch enables caching of test results, not only expected results, which is only useful if the tests are mostly deterministic (see matplotlib#7748). It adds reporting of cache misses, which can be helpful in getting tests to stay deterministic, and expiration since the test results are going to change more often than the expected results.

Some minor corrections, but otherwise fine.

In -> It

svg -> SVG

pdf -> PDF
svg -> SVG

This dict literal is a bit broken; should be dict(creationDate=None) or {'creationDate': None}.

Do we even need the version here? It's in the Creator tag.

I thought including the version in both would help with debugging when the user has overridden just one of these. If you use Matplotlib as a component of some larger application, you might want to override Creator and leave Producer pointing to the PDF backend.

Fair enough.

I don't understand the following sentence:
"It can be used to change some random id values in the output to be deterministic, at the cost that including multiple such SVG files in one document can lead to collisions."
Isn't there an grammar problem?

I'll try to rephrase.

Thanks! This looks good to me.
👍

using with -> using.

There is a cache of png files keyed by the MD5 hashes of corresponding svg and pdf files, which helps reduce test suite running times for svg and pdf files that stay exactly the same from one run to the next. This patch enables caching of test results, not only expected results, which is only useful if the tests are mostly deterministic (see matplotlib#7748). It adds reporting of cache misses, which can be helpful in getting tests to stay deterministic, and expiration since the test results are going to change more often than the expected results.

jkseppan added 4 commits January 5, 2017 19:27

Allow deleting pdf metadata keys

529afdc

And document the standard keys in PdfPages. Also insert the matplotlib version in the default value of Producer as well as Creator, to retain a debugging clue in case the user overrides one of them.

Make pdf test results (more) deterministic

8c7adbb

Remove the metadata entries that depend on the date and the exact version.

Reproducible svg

93b2eea

Document the changes in whats_new

eab297c

jkseppan added the topic: testing label Jan 5, 2017

tacaswell added this to the 2.1 (next point release) milestone Jan 5, 2017

tacaswell reviewed Jan 5, 2017

View reviewed changes

QuLogic reviewed Jan 5, 2017

View reviewed changes

jkseppan mentioned this pull request Jan 7, 2017

[MRG+2] MAINT: Refactor the converted-image cache #7764

Closed

QuLogic reviewed Jan 20, 2017

View reviewed changes

Fix documentation mistakes

ec941b4

QuLogic approved these changes Jan 20, 2017

View reviewed changes

QuLogic changed the title MAINT: Deterministic SVG and PDF tests [MRG+1] MAINT: Deterministic SVG and PDF tests Jan 20, 2017

NelleV reviewed Jan 20, 2017

View reviewed changes

Try to improve wording

749e81f

NelleV changed the title [MRG+1] MAINT: Deterministic SVG and PDF tests [MRG+2] MAINT: Deterministic SVG and PDF tests Jan 20, 2017

QuLogic reviewed Jan 20, 2017

View reviewed changes

Another wording fix

6736f7d

QuLogic changed the title [MRG+2] MAINT: Deterministic SVG and PDF tests MAINT: Deterministic SVG and PDF tests Jan 21, 2017

QuLogic merged commit b6f13f1 into matplotlib:master Jan 21, 2017

jkseppan deleted the deterministic-tests branch January 22, 2017 07:37

astrofrog mentioned this pull request Jan 26, 2017

Check whether any updates are needed following Matplotlib changes matplotlib/pytest-mpl#41

Closed

                       actual_fname = os.path.join(self.result_dir, baseline) + '.' + extension
-                      fig.savefig(actual_fname, **self.savefig_kwargs)
+                      kwargs = self.savefig_kwargs.copy()
+                      if extension == 'pdf':

+              For SVG output, the ``svg.hashsalt`` rc parameter has been added in an
+              earlier release. In can be used to change some random id values in the
+              output to be deterministic, at the cost that including multiple such
+              svg files in one document can lead to collisions.

                       self.infoDict = {
                           'Creator': 'matplotlib %s, http://matplotlib.org' % __version__,
-                          'Producer': 'matplotlib pdf backend%s' % revision,
+                          'Producer': 'matplotlib pdf backend %s' % __version__,

-              SVG files in one document can lead to collisions.
+              earlier release. This parameter changes some random identifiers in the
+              SVG file to be deterministic. The downside of this setting is that if
+              more than one file is generated using with deterministic identifiers



		def set_reproducibility_for_testing():
		rcParams['svg.hashsalt'] = 'matplotlib'

               the timestamp value in the PS and PDF outputs. See
               https://reproducible-builds.org/specs/source-date-epoch/
+              Alternatively, calling ``savefig`` with ``metadata={creationDate=None}``

               add sources of nondeterminism outside the control of matplotlib.
+              For SVG output, the ``svg.hashsalt`` rc parameter has been added in an
+              earlier release. It can be used to change some random id values in the

MAINT: Deterministic SVG and PDF tests by jkseppan · Pull Request #7748 · matplotlib/matplotlib · GitHub

Repository navigation

MAINT: Deterministic SVG and PDF tests - #7748

MAINT: Deterministic SVG and PDF tests#7748QuLogic merged 7 commits intomatplotlib:mastermatplotlib/matplotlib:masterfrom jkseppan:deterministic-testsjkseppan/matplotlib:deterministic-testsCopy head branch name to clipboard

Conversation

jkseppan commented Jan 5, 2017

Uh oh!

codecov-io commented Jan 5, 2017

Current coverage is 62.12% (diff: 100%)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tacaswell commented Jan 5, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

QuLogic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NelleV commented Jan 20, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MAINT: Deterministic SVG and PDF tests#7748
QuLogic merged 7 commits into
matplotlib:masterfrom
jkseppan:deterministic-tests