@cjmayo Let me know if this works with python 3. I have no reason to believe it shouldn't.

Edit: Er, I mean let me know if it doesn't work with python 3.

+1. This works for me. Thanks for taking care of this -- building the docs under Python 3 is something that fell through the cracks during the Python 3 port.

Thanks goes to @cjmayo for providing the original patches.

df93eca and 19b1a05 work with Python 3 for me and I can get the docs to build (I can't formally verify but I have lots of html, images and pdfs and all the links I have tried in the html work). But it is a bit more complicated.

I can only get this to work with Python 3.3.0 and not with Python 3.2.3 (specifically Gentoo Linux python-3.2.3). 3.2 fails with:

sphinx 1.1.3-r3
reading sources... [ 80%] examples/pylab_examples/unicode_demo   
Sphinx error:
'ascii' codec can't decode byte 0xc3 in position 134: ordinal not
Building HTML failed.

and I haven't figured that out.

My recipe:

Dependencies needed for Python 3.3 compatibility:

• docutils 0.9.1 plus patch combining:
  http://repo.or.cz/w/docutils.git/patch/12e5aa7b94e0a0ec37e754527e06707005d28d1c
  http://repo.or.cz/w/docutils.git/patch/46a3a9f46ebda4cb49e73d2edf5cdb5851d98a9f
• sphinx tip (6bb9dfbdf66e)

Matplotlib library patches (most of these probably break Python 2 use so need work):

• I have to revert the manual reference count patches to ft2font.cpp (for Python 2.7.3 also), and it seems others maybe have the same problem in Issue matplotlib.tests.test_mathtext.mathfont_cm_23_test.test makes python debug to crash #1309. But for yet others it fixed it stix_fonts_demo.py fails with bad refcount #1054???
• To cope with the seeming 2to3 import exception bug http://bugs.python.org/issue16214

--- lib/matplotlib/sphinxext/plot_directive.py.orig
+++ lib/matplotlib/sphinxext/plot_directive.py
@@ -125,8 +125,7 @@
 """
 from __future__ import print_function

-import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
-       traceback, exceptions
+import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, traceback

 from docutils.parsers.rst import directives
 from docutils import nodes

• hash functions in Python 3 take bytes not strings:

--- lib/matplotlib/sphinxext/mathmpl.py.orig
+++ lib/matplotlib/sphinxext/mathmpl.py
@@ -65,7 +65,7 @@
 def latex2html(node, source):
     inline = isinstance(node.parent, nodes.TextElement)
     latex = node['latex']
-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

     destdir = os.path.join(setup.app.builder.outdir, '_images', 'mathmpl')
     if not os.path.exists(destdir):

Matplotlib documentation build system patches:

• make.py, not sure what the final proposal was from Make make.py build docs under python 3 #1361 I just do:

--- doc/make.py.orig
+++ doc/make.py
@@ -5,108 +5,9 @@
 import glob
 import os
 import shutil
+from shutil import ignore_patterns, copytree
 import sys

-### Begin compatibility block for pre-v2.6: ###
-#
-# ignore_patterns and copytree funtions are copies of what is included
-# in shutil.copytree of python v2.6 and later.
-#
....

• df93eca
• 19b1a05
• Simple 2to3 on gen_gallery.py:

--- doc/sphinxext/gen_gallery.py.orig
+++ doc/sphinxext/gen_gallery.py
@@ -121,18 +121,18 @@

     gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
     if os.path.exists(gallery_path):
-        fh = file(gallery_path, 'r')
+        fh = open(gallery_path, 'r')
         regenerate = fh.read() != content
         fh.close()
     else:
         regenerate = True
     if regenerate:
-        fh = file(gallery_path, 'w')
+        fh = open(gallery_path, 'w')
         fh.write(content)
         fh.close()

     for key in app.builder.status_iterator(
-        thumbnails.iterkeys(), "generating thumbnails... ",
+        iter(thumbnails.keys()), "generating thumbnails... ",
         length=len(thumbnails)):
         image.thumbnail(key, thumbnails[key], 0.3)

Sorry for the noise. I was wrong about this "working for me". I was inadvertently using the Python 2.x installation of Sphinx. I'm working through some remaining Python 3 issues and will have a PR against this soon.

@cjmayo Have those docutils patches been merged upstream? We might not be able to get this in for 1.2 if we have to wait for upstream changes.

Yes they are direct from upstream:
http://docutils.sourceforge.net/docs/dev/repository.html#id5
It's just the git interface is a bit easier to browse.

But they are only needed for Python 3.3. All the other the other matplotlib fixes will be needed for Python 3.2 too and there is something else needed for 3.2 for that 'ascii' codec error.

Having major headaches with python2/3 unicode differences. Bleh. I'm trying to use the utilities from six.py but it claims six is not a module...

@cjmayo I have an idea about that ascii error on python 3.2. I'm pretty sure in python 3.2 things like u'asd' are invalid syntax, but in python 3.3 u'asd' is allowed.

I'm moving the commits from #1361 into this branch.

@mdboom Are you saying running python3.2 make.py is not sufficient, and that I need a separate python3 build of sphinx installed?

@dmcdougall: Yes -- you'll need to install sphinx in Python 3 (using pip-3.2 install sphinx or similar, or from a package), but then you also need to make sure that the Python 3 version of sphinx-build is the default on the path, probably by manually overriding the PATH environment variable.

@dmcdougall: You're farther along than I ever got at this point. That traceback is too promising -- it suggests that the inheritance_diagram hasn't yet been updated for Python 3 (I guess). I wonder if disabling that plugin in our conf.py let's it go further, or whether there are just other problems hiding behind that one?

@mdboom I'm using all of the changes in this PR. I added some commits earlier today. I also merged the ones from #1361 to keep everything together.

MacPorts doesn't seem to sport a py31/py32 version of sphinx. I have no experience with sphinx. My guess is that support for python 3 is minimal. I'm not sure where to go from here.

@mdboom Commenting out the inheritance diagram in conf.py it mostly works. I get the following error, but the doc build still completes!

<autodoc>:0: ERROR: Unknown interpreted text role "function".
/Users/damon/git/github/matplotlib/doc/devel/documenting_mpl.rst:384: ERROR: Unknown directive type "inheritance-diagram".

.. inheritance-diagram:: matplotlib.patches matplotlib.lines matplotlib.text
   :parts: 2
/Users/damon/git/github/matplotlib/doc/devel/transformations.rst:5: ERROR: Unknown directive type "inheritance-diagram".

.. inheritance-diagram:: matplotlib.transforms matplotlib.path
   :parts: 1
/Users/damon/git/github/matplotlib/doc/examples/pylab_examples/multipage_pdf.rst:8: WARNING: Exception occurred in plotting multipage_pdf
 from /Users/damon/git/github/matplotlib/doc/mpl_examples/pylab_examples/multipage_pdf.py:
Traceback (most recent call last):
  File "/Users/damon/python/lib/matplotlib/sphinxext/plot_directive.py", line 515, in run_code
    exec(code, ns)
  File "<string>", line 36
    d['Author'] = u'Jouni K. Sepp\xe4nen'

One of them is a unicode issue, because u'asd' is not valid syntax in py3.2.

It's great that it completes. That unicode issue is probably fixable -- something along the lines of

'Jouni K. Sepp\xe4nen'.decode('latin-1')

If that's the only problem, we're in pretty good shape.

I'm also getting this: loading pickled environment... failed: unsupported pickle protocol: 3.

Edit: but the build still completes.

Er, nevermind. Turns out the pickling thing gets done to save time on subsequent runs? Since I keep switching between python2 and python3 the pickled environment gets corrupted but it gets regenerated correctly. Disregard my previous comment.

I can get past the ascii error with Python 3.2 with an updated version of the gen_gallery.py patch above:

--- doc/sphinxext/gen_gallery.py.orig
+++ doc/sphinxext/gen_gallery.py
@@ -20,7 +20,7 @@
 {%% endblock %%}
 """

-import os, glob, re, sys, warnings
+import io, os, glob, re, sys, warnings
 import matplotlib.image as image

 multiimage = re.compile('(.*?)(_\d\d){1,2}')
@@ -121,18 +121,18 @@

     gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
     if os.path.exists(gallery_path):
-        fh = file(gallery_path, 'r')
+        fh = open(gallery_path, 'r')
         regenerate = fh.read() != content
         fh.close()
     else:
         regenerate = True
     if regenerate:
-        fh = file(gallery_path, 'w')
+        fh = io.open(gallery_path, 'w', encoding='utf8')
         fh.write(content)
         fh.close()

     for key in app.builder.status_iterator(
-        thumbnails.iterkeys(), "generating thumbnails... ",
+        iter(thumbnails.keys()), "generating thumbnails... ",
         length=len(thumbnails)):
         image.thumbnail(key, thumbnails[key], 0.3)

And a new patch to plot_directive.py:

--- lib/matplotlib/sphinxext/plot_directive.py.orig
+++ lib/matplotlib/sphinxext/plot_directive.py
@@ -125,6 +125,7 @@
 """
 from __future__ import print_function

+import io
 import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
        traceback, exceptions

@@ -667,7 +668,7 @@
         else:
             function_name = None

-        with open(source_file_name, 'r') as fd:
+        with io.open(source_file_name, 'r', encoding='utf8') as fd:
             code = fd.read()
         output_base = os.path.basename(source_file_name)
     else:
@@ -809,7 +810,7 @@

     # copy script (if necessary)
     target_name = os.path.join(dest_dir, output_base + source_ext)
-    with open(target_name, 'w') as f:
+    with io.open(target_name, 'w', encoding='utf8') as f:
         if source_file_name == rst_file:
             code_escaped = unescape_doctest(code)
         else:

This now gets me through the html generation and also png creation. Fails now in using graphviz.

Couple of comments on the recent comments.

On the 'Jouni K. Sepp\xe4nen' would from __future__ import unicode_literals work as used in examples/pylab_examples/unicode_demo.py?

Have you tested:

-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

on Python 2?

@cjmayo Cheers for the patch updates. For some reason I don't have a problem with the current ones, but I'll add your updates in since they work for you.

On the 'Jouni K. Sepp\xe4nen' would from __future__ import unicode_literals work as used in examples/pylab_examples/unicode_demo.py?

Probably. I didn't know this existed. I have never worked with python 3, so this whole PR/frustration is a combination of me learning and getting headaches.

Have you tested:

-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

on Python 2?

Nope. All the testing I did was the generate the documentation under python2, and the build succeeds. I have not checked the output.

I am getting a new error building the docs under python 3:

Running Sphinx v1.1.3

Exception occurred:
  File "/Users/damon/python/lib/matplotlib/dates.py", line 122, in <module>
    from dateutil.rrule import rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY, \
  File "/Users/damon/python/lib/dateutil/rrule.py", line 55
    raise ValueError, "Can't create weekday with n == 0"
                    ^
SyntaxError: invalid syntax

I have no idea why this is happening. 2to3 should have fixed this. Argh! I think it's time for me to call it a day and look at this tomorrow.

With the last two patches I can now get to the end of the doc building with Python 3.2.3, but I have to use sphinx tip (6bb9dfbdf66e) - sphinx 1.1.3 does not work for me.

@cjmayo Progress! Interesting that you need current Sphinx HEAD. What errors do you get with sphinx 1.1.3?

We have two options on proceeding depending on your response to my above question:

1. We may be able to patch our sphinxext files to work with sphinx v1.1.3.
2. If 1) is not possible, then this pull request will fester until the problem is solved upstream.

At the end of the day, the most important thing is to be able to produce the docs under some version of Python (and Python 2.x is currently working). Having them build under 3.x is of course very desirable, but if that can't be done with a stable version of Sphinx, than there's not much point to putting this is our release either. IMHO, of course.

@mdboom Agreed; there are more important v1.2.x bugs currently milestoned. Furthermore, I think if we are waiting in either case the v1.2.x milestone is unnecessary.

I will keep this PR open and remove the milestone. If anybody disagrees then adding it back on is fine.

Even with these changes, I'm encountering an import error related to this bug: RuntimeError: dictionary changed size during iteration from colors.py #1401

Thanks @ariddell - the error you are describing is a matplotlib issue relating to python3.3 to try and keep the issues separate, this build process fix is for the documentation building with a currently supported python (lets say python3.2 for sake of argument). So whilst I think what you highlight is important, it should be addressed as a completely independent issue (#1401).

Cheers,

@dmcdougall - do you remember the status of this? Is it worth re-visiting and getting this PR merged?

I'm getting the error with 3.2.3.

I've setup a clean virtualenv, pip install'd only numpy and git://github.com/dmcdougall/matplotlib.git@py3_docs_2#egg=matplotlib and I get this:

Python 3.2.3 (default, Oct 19 2012, 19:53:16) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.sphinxext.plot_directive
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/__init__.py", line 151, in <module>
    from matplotlib.rcsetup import (defaultParams,
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/rcsetup.py", line 20, in <module>
    from matplotlib.colors import is_color_like
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/colors.py", line 205, in <module>
    for k, v in cnames.items():
RuntimeError: dictionary changed size during iteration

Python path is: ['', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages/distribute-0.6.24-py3.2.egg', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages/pip-1.1-py3.2.egg', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/plat-linux2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/lib-dynload', '/usr/lib/python3.2', '/usr/lib/python3.2/plat-linux2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages']

Hope this helps. Thanks for all your work on this.

I'm getting the error with 3.2.3.

Ah... Ok - thanks for pointing that out. In that case I think it is very pertinent for this PR.

Cheers,

@dmcdougall - I'd be happy to merge this. Is there more work to be done before I do, or shall I go ahead and press the green button?

@pelson It appears that @cjmayo is currently operational with these changes, and that @ariddell is having issues with that dictionary error. That error, however, appears to have been resolved in the comments in #1401. So, I think we should merge this. If there are further dictionary errors, we can resolve those separately.

Cheers @cjmayo for the help.