Reinstate the manual reference counting as discovered in #1054
mdboom authored 4 months ago 1 parent cbb11da commit ca678a4

This is starting to look like a merry go round.

lssue # /1054 submitter by a fellow gentooer. Now you have in source versions of both pycxx and pyparsing.
the above issue was 'fixed' by a patch to the file src/ft2font.cpp. Well it suggests to me it has more to do with
CXX/Python2/cxx_extensions.cxx given that it's cited by the error msg. I can't say that the fix to #1054 broke 'something else somewhere else', at least not in prior version 1.1.1 which completes the tests and a build of the html docs cleanly

Thanks @idella. I think this is definitely one for you @mdboom.

The reason we need a patched version of PyCXX is because the mainline PyCXX simply doesn't compile on Python 2.7 due to the removal of PyCObject (in favor of PyCapsule). Please ping PyCXX for this -- I have submitted a patch, but it doesn't seem to have been incorporated. I can ping Barry Scott again about this.

I've never understood why the --ob_refcnt lines are in there -- indeed, it intuitively makes sense to not have them, and I'd like to have them removed. I think the reinstating them in #1054 was due to not being able to reproduce the problem and not fully understanding it.

On further investigation, there are two things required to see this bug, which is why it rarely pops up onto the radar of the developers: The binaries must be built in debug mode (by undef'ing NDEBUG), and then the opening of the font file must fail, either because it's an unsupported format or some kind of IO error. Indeed, with NDEBUG defined, the --ob_refcnt lines are not ideal, but harmless, so that's one workaround for Gentoo (it should produce faster code anyway). As for the invalid font file, is Gentoo using something other than the Bakoma fonts that ship with matplotlib? You can determine that using:

In [1]: from matplotlib import font_manager
In [4]: font_manager.findfont("cmr10")

Does the file exist -- is it the same as the one that ships with matplotlib? If not, can you send me a link to it so I can investigate further?

Mike

BTW: We need to fix this bug on 1.2.x and in master -- but I should note also to address the issue of using a system pycxx, see #1454. Note that I was wrong about the PyCapsule support -- it's a non-issue. The remaining issue is the lack of buffer support on Python 3.x. I'll submit a patch upstream for that, and once that's in a PyCXX and we merge the setup retooling in #1454, we should be able to not need the local copy of CXX (though we may keep it around in the source tree for a while to support to keep the source install as easy as possible).

hey Mike;

good to see you're onto this. This is all about the release 1.2.0
archtester matplotlib # eix matplotlib
[D] dev-python/matplotlib
Available versions: 1.0.1-r1 1.1.0 (~)1.1.1 {{cairo doc examples excel fltk gtk latex qt4 test tk traits wxwidgets}}
Installed versions: 1.2.0^m(04:34:02 25/11/12)(cairo gtk qt4 wxwidgets -doc -examples -excel -fltk -latex -test -tk)

A user has at this point gone to quite a bit of effort to prepare an ebuild to bump matplotlib to 1.2.0. So far we have the hindrance of this bug and a tardy pyparsing upstream seemingly sitting on their hands re releasing a next version with a 1 line patch to fix an issue in python3. I've entered the patch and bumped it anyway, the preferred option of course being to have the next release release requiring no patching.

with python2.7 eselected,

matplotlib # bpython

from matplotlib import font_manager
font_manager.findfont("cmr10")
'/usr/lib64/python2.7/site-packages/matplotlib/mpl-data/fonts/ttf/cmr10.ttf'
quit()

Looks fine to me. "BTW: We need to fix this bug on 1.2.x and in master "
Yes. This appears to be the final hurdle holding back bumping matplotlib for gentoo
Is that all you need from this end?

Ok -- on our end, I'll remove the --ob_refcnt lines. Gentoo can either patch to do the same or not build in DEBUG mode (which seems an odd thing to do in the first place). I will probably not be making a 1.2.1 release for this fix alone, but obviously if we have a 1.2.1 release this fix will be in there.

It remains a puzzle as to why loading that font is failing. If you patch out the --ob_refcnt lines, you should get an exception about why the font loading is failing that may provide more clues.

Ideally, this is the sort of thing that would get sorted out during the release candidate cycle. Do you have contact information for the maintainer of the matplotlib Gentoo package? I can add that person to my list of people that I notify about release candidates etc., though better yet would be for that person to monitor the github download page for new releases.

hey Mike, good work. Thanks so much

"Do you have contact information for the maintainer of the matplotlib Gentoo package"
You have been talking to him already, me and my ilk in the python herd. You have the luxury of choice here;

• gentoo-python@gentoo.org
• idella4@gentoo.org

Don't forget the 4 so I can distinguish from idellas 5 & 6. They're quite temperamental you know.

" though better yet would be for that person to monitor the github download page for new releases."
We even have someone else in our irc neighbourhood with not enough to do who is good at such ventures. It's 'all good'. The patch I shall clone and enter and subsequently bump matplotlib-1.2.0 to its rightful place, in working order, in portage.

A pleasure doing business with you. So many bugs I file draw no reply. Well done.

I spoke a fraction too soon. The only test it bails out on now is the building of docs under python3.
I get

x86_64-pc-linux-gnu-g++ -pthread -march=athlon64 -pipe -fomit-frame-pointer -O2 -fno-strict-aliasing -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_ARRAY_API -DPYCXX_ISO_CPP_LIB=1 -DPYCXX_PYTHON_2TO3=1 -I/usr/local/include -I/usr/include -I/usr/include/libpng15 -I/usr/include -I. -I/usr/lib64/python3.2/site-packages/numpy/core/include -I. -I/usr/include/python3.2 -c CXX/cxx_extensions.cxx -o build-3.2/temp.linux-x86_64-3.2/CXX/cxx_extensions.o
x86_64-pc-linux-gnu-gcc -pthread -march=athlon64 -pipe -fomit-frame-pointer -O2 -fno-strict-aliasing -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_ARRAY_API -DPYCXX_ISO_CPP_LIB=1 -DPYCXX_PYTHON_2TO3=1 -I/usr/local/include -I/usr/include -I/usr/include/libpng15 -I/usr/include -I. -I/usr/lib64/python3.2/site-packages/numpy/core/include -I. -I/usr/include/python3.2 -c CXX/cxxextensions.c -o build-3.2/temp.linux-x86_64-3.2/CXX/cxxextensions.o
x86_64-pc-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -march=athlon64 -pipe -fomit-frame-pointer -O2 -fno-strict-aliasing build-3.2/temp.linux-x86_64-3.2/src/_png.o build-3.2/temp.linux-x86_64-3.2/src/mplutils.o build-3.2/temp.linux-x86_64-3.2/CXX/IndirectPythonInterface.o build-3.2/temp.linux-x86_64-3.2/CXX/cxxsupport.o build-3.2/temp.linux-x86_64-3.2/CXX/cxx_extensions.o build-3.2/temp.linux-x86_64-3.2/CXX/cxxextensions.o -L/usr/local/lib -L/usr/lib -L/usr/local/lib64 -L/usr/lib64 -L/usr/lib64 -lpng15 -lz -lstdc++ -lm -lpython3.2 -o build-3.2/lib.linux-x86_64-3.2/matplotlib/_png.cpython-32.so
File "./make.py", line 92
except Error, err:
^
SyntaxError: invalid syntax

• ERROR: dev-python/matplotlib-1.2.0 failed (compile phase):

  matplotlib # find /mnt/gen2/TmpDir/portage/dev-python/matplotlib-1.2.0/work/matplotlib-1.2.0 -name make.py
  /mnt/gen2/TmpDir/portage/dev-python/matplotlib-1.2.0/work/matplotlib-1.2.0/doc/make.py
  /mnt/gen2/TmpDir/portage/dev-python/matplotlib-1.2.0/work/matplotlib-1.2.0/doc/pyplots/make.py

ok, so it appears that make.py is written to python2 format and setup.py doesn't include it for conversion for python3. Is this by intent, ie. python2 is sufficient for building docs? Otherwise a separate issue?

Sorry this response fell through the cracks. We don't currently support building the docs on Python 3.x, since there are still issues with Sphinx in that context. See #1383.

I think for that reason, we should probably merge this if there are no objections.

I'd like to see this rebased so Travis can do the tests for Python 3 before it gets merged.

Good idea. Rebased.