Repository navigation
Unicode with usetex=True and pgf backend #5234
Description
The documentation says that registering the backend should work as well (or better) than use('pgf').
This breaks when using unicode because the TeX strings are handled by the default matplotlib texmanager instead of the pgf backend.
If there is no way to disable the standard texmanager when rendering using the pgf backend, the documentation should be changed to make this problem clear.
import matplotlib as mpl
#mpl.use("pgf") # works
import matplotlib.pyplot as plt
from matplotlib.backends.backend_pgf import FigureCanvasPgf # does not work
mpl.backend_bases.register_backend('pdf', FigureCanvasPgf) #
mpl.rcParams.update({
'font.family': 'serif',
'text.usetex': True,
'pgf.rcfonts': False,
'pgf.preamble': r'\usepackage{unicode-math}',
})
plt.plot([1, 2], [3, 4])
plt.xlabel(r'$α$')
plt.savefig('foo.pdf')Traceback (most recent call last):
File "baz.py", line 19, in <module>
plt.savefig('baz.pdf')
File "/usr/lib/python3.5/site-packages/matplotlib/pyplot.py", line 577, in savefig
res = fig.savefig(*args, **kwargs)
File "/usr/lib/python3.5/site-packages/matplotlib/figure.py", line 1476, in savefig
self.canvas.print_figure(*args, **kwargs)
File "/usr/lib/python3.5/site-packages/matplotlib/backends/backend_qt5agg.py", line 162, in print_figure
self.draw()
File "/usr/lib/python3.5/site-packages/matplotlib/backends/backend_qt5agg.py", line 148, in draw
FigureCanvasAgg.draw(self)
File "/usr/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 469, in draw
self.figure.draw(self.renderer)
File "/usr/lib/python3.5/site-packages/matplotlib/artist.py", line 59, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/usr/lib/python3.5/site-packages/matplotlib/figure.py", line 1085, in draw
func(*args)
File "/usr/lib/python3.5/site-packages/matplotlib/artist.py", line 59, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/usr/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 2110, in draw
a.draw(renderer)
File "/usr/lib/python3.5/site-packages/matplotlib/artist.py", line 59, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/usr/lib/python3.5/site-packages/matplotlib/axis.py", line 1128, in draw
self.label.draw(renderer)
File "/usr/lib/python3.5/site-packages/matplotlib/artist.py", line 59, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/usr/lib/python3.5/site-packages/matplotlib/text.py", line 595, in draw
bbox, info, descent = self._get_layout(renderer)
File "/usr/lib/python3.5/site-packages/matplotlib/text.py", line 320, in _get_layout
ismath=ismath)
File "/usr/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 223, in get_text_width_height_descent
renderer=self)
File "/usr/lib/python3.5/site-packages/matplotlib/texmanager.py", line 670, in get_text_width_height_descent
dvifile = self.make_dvi(tex, fontsize)
File "/usr/lib/python3.5/site-packages/matplotlib/texmanager.py", line 394, in make_dvi
texfile = self.make_tex(tex, fontsize)
File "/usr/lib/python3.5/site-packages/matplotlib/texmanager.py", line 309, in make_tex
fh.write(s.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\u03b1' in position 298: ordinal not in range(128)
Activity
mdboom commented on Oct 13, 2015
Cc: @pwuertz
pwuertz commented on Oct 13, 2015
I am pretty sure that it did work at some point, but it's true that usetex true + Unicode cannot be handled by other backends that are using pdflatex. So maybe it should be made clear that this combination of parameters is only possible with xe or lualatex.
What puzzles me is the error you are seeing. Your savefig is calling into the qt5 backend instead of pgf. I will try to find out why.
QuLogic commented on Oct 13, 2015
Why's that? pdflatex mostly will work with UTF-8 if you set the input encoding, though it somewhat depends on what version of everything is installed.
pwuertz commented on Oct 13, 2015
AFAIK pdflatex has no support for unicode fonts and generally doesn't know how to handle non-ascii characters. By including inputenc and a few other packages you can add some hardcoded character-to-glyph mappings so you can get away with common language specific characters. But this isn't exactly what you would call "unicode support".
pwuertz commented on Oct 13, 2015
Ok, this is a bit confusing, and I'm not sure how to wrap this up in the documentation.
The savefig command actually works, although you see an exception later. The reason for this is that the Qt5 / Agg backend is (and stays) the default backend and after saving the figure it additionally tries to draw its own canvas once. This then fails when the agg backend uses pdflatex to process your unicode label.
One could argue that the additional draw() call after print_figure() isn't required, so if you are not using the GUI the GUI shouldn't complain about the usetex configuration. In case you are using the GUI and backend_pgf for PDF output you'd have to change the rc params to something compatible to qt5 / agg though. This usecase is actually my motivation for preferring the register_backend solution - being able to interact with the figure and to save PGF->PDF figures within the same run.
itoijala commented on Oct 14, 2015
So basically using the register_backend approach I have to make sure that the labels are compatible with pdflatex. For me that means I will have to use the mpl.use approach. Otherwise using lualatex doesn't have any advantages.
pwuertz commented on Oct 14, 2015
Not really, the problem is the usetex flag. It isn't really required for backend pgf since all text is processed by tex anyways, so you could just omit it. The GUI will try to process your labels without pdflatex and probably won't be able to interpret more complex latex macros, which looks weird but it won't crash on it. Unfortunately matplotlib internally misuses(?) the usetex flag for some other decisions as well, but if you don't care you can just set usetex=False.
itoijala commented on Oct 14, 2015
I am using usetex to get the tick labels in the math font.
I guess that's not so important.
pwuertz commented on Oct 14, 2015
Ah yes, this is one of the things I meant by "other decisions". According to the documentation enabling "usetex" means that the non-latex backends should delegate text rendering to the latex manager and include the result, instead of using the internal matplotlib latex formatter. Unfortunately this flag is also being used for changing the style of the tick labels (putting them in math mode) instead of using a dedicated switch.
By the way, I just realized that there is a setting for enabling unicode input encoding in pdflatex/latexmanager. Maybe this is of interest to you?
http://matplotlib.org/users/usetex.html#usetex-with-unicode
Anyway, if you are not planning on using the GUI and a non-GUI-savefig within the same workflow just go with use("pgf") if this turns out to be more trouble than its worth.
ArchangeGabriel commented on Jun 26, 2016
I’m not sure whether this is what I’m facing, because your guys talk about pdflatex while I definitively use xelatex for pgf rendering. But still, needing both pdf export and interactive plots, I’m trying the “registering” method, and when saving a pdf:
--------------------------------------------------------------------------- UnicodeEncodeError Traceback (most recent call last) <ipython-input-7-7a82c592d709> in <module>() 39 #ax.set_title("Mode $m = 1$ ($\mathcal{M}_1 = \infty$)") 40 ---> 41 plt.savefig("out.pdf",transparent=True,dpi=300,bbox_inches='tight',pad_inches=0.05) 42 plt.close("all") /usr/lib/python3.5/site-packages/matplotlib/pyplot.py in savefig(*args, **kwargs) 686 def savefig(*args, **kwargs): 687 fig = gcf() --> 688 res = fig.savefig(*args, **kwargs) 689 fig.canvas.draw_idle() # need this if 'transparent=True' to reset colors 690 return res /usr/lib/python3.5/site-packages/matplotlib/figure.py in savefig(self, *args, **kwargs) 1563 self.set_frameon(frameon) 1564 -> 1565 self.canvas.print_figure(*args, **kwargs) 1566 1567 if frameon: /usr/lib/python3.5/site-packages/matplotlib/backends/backend_qt5agg.py in print_figure(self, *args, **kwargs) 195 def print_figure(self, *args, **kwargs): 196 FigureCanvasAgg.print_figure(self, *args, **kwargs) --> 197 self.draw() 198 199 /usr/lib/python3.5/site-packages/matplotlib/backends/backend_qt5agg.py in draw(self) 156 # The Agg draw is done here; delaying causes problems with code that 157 # uses the result of the draw() to update plot elements. --> 158 FigureCanvasAgg.draw(self) 159 self.update() 160 /usr/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py in draw(self) 472 473 try: --> 474 self.figure.draw(self.renderer) 475 finally: 476 RendererAgg.lock.release() /usr/lib/python3.5/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs) 59 def draw_wrapper(artist, renderer, *args, **kwargs): 60 before(artist, renderer) ---> 61 draw(artist, renderer, *args, **kwargs) 62 after(artist, renderer) 63 /usr/lib/python3.5/site-packages/matplotlib/figure.py in draw(self, renderer) 1157 dsu.sort(key=itemgetter(0)) 1158 for zorder, a, func, args in dsu: -> 1159 func(*args) 1160 1161 renderer.close_group('figure') /usr/lib/python3.5/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs) 59 def draw_wrapper(artist, renderer, *args, **kwargs): 60 before(artist, renderer) ---> 61 draw(artist, renderer, *args, **kwargs) 62 after(artist, renderer) 63 /usr/lib/python3.5/site-packages/matplotlib/axes/_base.py in draw(self, renderer, inframe) 2322 2323 for zorder, a in dsu: -> 2324 a.draw(renderer) 2325 2326 renderer.close_group('axes') /usr/lib/python3.5/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs) 59 def draw_wrapper(artist, renderer, *args, **kwargs): 60 before(artist, renderer) ---> 61 draw(artist, renderer, *args, **kwargs) 62 after(artist, renderer) 63 /usr/lib/python3.5/site-packages/matplotlib/axis.py in draw(self, renderer, *args, **kwargs) 1118 self._update_label_position(ticklabelBoxes, ticklabelBoxes2) 1119 -> 1120 self.label.draw(renderer) 1121 1122 self._update_offset_text_position(ticklabelBoxes, ticklabelBoxes2) /usr/lib/python3.5/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs) 59 def draw_wrapper(artist, renderer, *args, **kwargs): 60 before(artist, renderer) ---> 61 draw(artist, renderer, *args, **kwargs) 62 after(artist, renderer) 63 /usr/lib/python3.5/site-packages/matplotlib/text.py in draw(self, renderer) 747 748 with _wrap_text(self) as textobj: --> 749 bbox, info, descent = textobj._get_layout(renderer) 750 trans = textobj.get_transform() 751 /usr/lib/python3.5/site-packages/matplotlib/text.py in _get_layout(self, renderer) 359 w, h, d = renderer.get_text_width_height_descent(clean_line, 360 self._fontproperties, --> 361 ismath=ismath) 362 else: 363 w, h, d = 0, 0, 0 /usr/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py in get_text_width_height_descent(self, s, prop, ismath) 227 fontsize = prop.get_size_in_points() 228 w, h, d = texmanager.get_text_width_height_descent(s, fontsize, --> 229 renderer=self) 230 return w, h, d 231 /usr/lib/python3.5/site-packages/matplotlib/texmanager.py in get_text_width_height_descent(self, tex, fontsize, renderer) 673 else: 674 # use dviread. It sometimes returns a wrong descent. --> 675 dvifile = self.make_dvi(tex, fontsize) 676 dvi = dviread.Dvi(dvifile, 72 * dpi_fraction) 677 try: /usr/lib/python3.5/site-packages/matplotlib/texmanager.py in make_dvi(self, tex, fontsize) 397 398 if DEBUG or not os.path.exists(dvifile): --> 399 texfile = self.make_tex(tex, fontsize) 400 outfile = basefile + '.output' 401 command = self._get_shell_cmd( /usr/lib/python3.5/site-packages/matplotlib/texmanager.py in make_tex(self, tex, fontsize) 312 else: 313 try: --> 314 fh.write(s.encode('ascii')) 315 except UnicodeEncodeError as err: 316 mpl.verbose.report("You are using unicode and latex, but " UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 307: ordinal not in range(128)ArchangeGabriel commented on Jun 26, 2016
Oh, indeed the figure is saved (which is the most important point to me), and I also rely on ticks to be in math mode. ;)
ArchangeGabriel commented on Jun 26, 2016
And finally, I was indeed able to work around this by setting text.latex.unicode = True and also adding text.latex.preamble config with the relevant packages.
ArchangeGabriel commented on Feb 9, 2017
Just for the record, this is obviously still an issue with 2.0.
6 remaining items
github-actions commented on Mar 17, 2023
This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!
QuLogic commented on Apr 18, 2023
The traceback is still happening, but it's a little different now:
Traceback (most recent call last): File "mpl-tests/issue5234.py", line 16, in <module> plt.savefig('foo.pdf') File "lib/matplotlib/pyplot.py", line 1082, in savefig res = fig.savefig(*args, **kwargs) # type: ignore[func-returns-value] File "lib/matplotlib/figure.py", line 3362, in savefig self.canvas.print_figure(fname, **kwargs) File "lib/matplotlib/backends/backend_qtagg.py", line 76, in print_figure self.draw() File "lib/matplotlib/backends/backend_agg.py", line 401, in draw self.figure.draw(self.renderer) File "lib/matplotlib/artist.py", line 95, in draw_wrapper result = draw(artist, renderer, *args, **kwargs) File "lib/matplotlib/artist.py", line 72, in draw_wrapper return draw(artist, renderer) File "lib/matplotlib/figure.py", line 3126, in draw mimage._draw_list_compositing_images( File "lib/matplotlib/image.py", line 131, in _draw_list_compositing_images a.draw(renderer) File "lib/matplotlib/artist.py", line 72, in draw_wrapper return draw(artist, renderer) File "lib/matplotlib/axes/_base.py", line 3061, in draw mimage._draw_list_compositing_images( File "lib/matplotlib/image.py", line 131, in _draw_list_compositing_images a.draw(renderer) File "lib/matplotlib/artist.py", line 72, in draw_wrapper return draw(artist, renderer) File "lib/matplotlib/axis.py", line 1368, in draw self.label.draw(renderer) File "lib/matplotlib/artist.py", line 72, in draw_wrapper return draw(artist, renderer) File "lib/matplotlib/text.py", line 716, in draw bbox, info, descent = self._get_layout(renderer) File "lib/matplotlib/text.py", line 350, in _get_layout w, h, d = _get_text_metrics_with_cache( File "lib/matplotlib/text.py", line 69, in _get_text_metrics_with_cache return _get_text_metrics_with_cache_impl( File "lib/matplotlib/text.py", line 77, in _get_text_metrics_with_cache_impl return renderer_ref().get_text_width_height_descent(text, fontprop, ismath) File "lib/matplotlib/backends/backend_agg.py", line 226, in get_text_width_height_descent return super().get_text_width_height_descent(s, prop, ismath) File "lib/matplotlib/backend_bases.py", line 647, in get_text_width_height_descent return self.get_texmanager().get_text_width_height_descent( File "lib/matplotlib/texmanager.py", line 359, in get_text_width_height_descent dvifile = cls.make_dvi(tex, fontsize) File "lib/matplotlib/texmanager.py", line 291, in make_dvi cls._run_checked_subprocess( File "lib/matplotlib/texmanager.py", line 254, in _run_checked_subprocess raise RuntimeError( RuntimeError: latex was not able to process the following string: b'$\\u03b1$' Here is the full command invocation and its output: latex -interaction=nonstopmode --halt-on-error --output-directory=tmppnau5tte 9ea043e51f5cc6de5d5eeeb5b2484a5c.tex This is pdfTeX, Version 3.141592653-2.6-1.40.22 (TeX Live 2021) (preloaded format=latex) restricted \write18 enabled. entering extended mode (./9ea043e51f5cc6de5d5eeeb5b2484a5c.tex LaTeX2e <2020-10-01> patch level 4 L3 programming layer <2021-05-07> (/usr/share/texlive/texmf-dist/tex/latex/base/article.cls Document Class: article 2020/04/10 v1.4m Standard LaTeX document class (/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo)) (/usr/share/texlive/texmf-dist/tex/latex/type1cm/type1cm.sty) (/usr/share/texlive/texmf-dist/tex/latex/cm-super/type1ec.sty (/usr/share/texlive/texmf-dist/tex/latex/base/t1cmr.fd)) (/usr/share/texlive/texmf-dist/tex/latex/base/inputenc.sty) (/usr/share/texlive/texmf-dist/tex/latex/geometry/geometry.sty (/usr/share/texlive/texmf-dist/tex/latex/graphics/keyval.sty) (/usr/share/texlive/texmf-dist/tex/generic/iftex/ifvtex.sty (/usr/share/texlive/texmf-dist/tex/generic/iftex/iftex.sty))) (/usr/share/texlive/texmf-dist/tex/latex/underscore/underscore.sty) (/usr/share/texlive/texmf-dist/tex/latex/base/textcomp.sty) (/usr/share/texlive/texmf-dist/tex/latex/l3backend/l3backend-dvips.def) No file 9ea043e51f5cc6de5d5eeeb5b2484a5c.aux. *geometry* driver: auto-detecting *geometry* detected driver: dvips ! Package inputenc Error: Unicode character α (U+03B1) (inputenc) not set up for use with LaTeX. See the inputenc package documentation for explanation. Type H <return> for immediate help. ... l.29 {\rmfamily $α $}% No pages of output. Transcript written on tmppnau5tte/9ea043e51f5cc6de5d5eeeb5b2484a5c.log.This one is because pgf.preamble doesn't get applied to the latex drawing in Qt.
But we go through a lot of trouble to use the right canvas when saving, so it's weird that Qt is drawing here. That's because the QtAgg backend overrides print_figure to call draw after. This also occurs in the Wx backend, but not GTK or Tk.
Remove print_figure overrides in backend subclasses
Remove print_figure overrides in backend subclasses
Remove print_figure overrides in backend subclasses
Remove print_figure overrides in backend subclasses
Remove print_figure overrides in backend subclasses