This is a good idea in principle and the patch should be applied to make building with an external PyCXX possible in the future.

However, we have a few patches in our tree that still have not been merged upstream.

a) The Solaris one you reported. It had not been reported upstream but I have now done so.

b) https://sourceforge.net/tracker/?group_id=3180&atid=103180

c) A number of Python3-specific changes (PyCapsule, PyBuffer) that I've been communicating with Barry Scott privately about.

Barry does a great job with matplotlib but he, by his own admission, is having trouble keeping up with the bugfixes and patches matplotlib generates. It would be great to resolve that upstream, but in the meantime, I recommend our local copy for use with matplotlib.

As an aside, we should probably move agg24 into this external directory, too (and again we have some patches submitted upstream that have been ignored). But that doesn't need to be addressed by this pull request.

Marking as "wishlist", since we can't do this until our patches are merged into upstream PyCXX.

it could be merged as the patch still uses the bundled pycxx version and besides modifying the source it does not offer a way to change it.
It should be safe to merge when one adds a comment that a patched version is required.

Mike, your last comment seems inconsistent with the one before it; is the problem that if the patch were applied, packagers would be likely to build mpl with the unpatched upstream PyCXX?

Yes -- building with an unpatched version of PyCXX results in memory leaks. In the case of the matplotlib-py3 branch, it won't even build at all. So packagers etc. should be encouraged to use the local copy. However, the larger issue is to how to get the patches in PyCXX accepted upstream or whether there is a better way to deal with that...

On 08/22/2011 08:39 AM, mdboom wrote:

Yes -- building with an unpatched version of PyCXX results in memory leaks. In the case of the matplotlib-py3 branch, it won't even build at all. So packagers etc. should be encouraged to use the local copy. However, the larger issue is to how to get the patches in PyCXX accepted upstream or whether there is a better way to deal with that...

How hard would it be as a longer-term strategy to migrate away from
PyCXX and standardize on cython? Or are we fundamentally wedded to PyCXX?

I did a little experimentation last year and concluded it would be a great deal of work, and Cython is a terrible fit for interfacing with Agg (and other parts, like the path stuff, which is written in C++). I think the easiest route would be to build a C++ to C shim and use Cython to interface to that. But that's kind of mess and a lot more verbose than what we have now. I'm more compelled to offer some assistance upstream to PyCXX and try to get some sort of more formal relationship so we can shepherd it there.

The latest release of PyCXX (6.2.4) still does not have the necessary changes to be compatible with Python 2.7. (Related to using PyCapsule over PyCObject). These local changes have been submitted upstream, but have not been incorporated (no response given other than perhaps lack of interest). So matplotlib will not run correctly with a stock PyCXX.

@mdboom : Was your assesment of cython interfacing to C++ done before v0.13? (http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html)

Wrapping C++ with Cython sounds more desirable to me than piling support into what appears to be a fairly slow moving (dead?) package.

Where do you think we can go with this PR? I'm inclined to close it - I do not see that we can make use of a non-bundled version of PyCXX in any short term timescale and IMHO it really is not healthy having PRs sitting around in a dormant state. Thoughts?

Though I didn't choose it, I think PyCXX is actually a tool for interfacing with C++, and to the extent that it's abandoned, perhaps we should take it over. It automatically handles reference counting, but still allows the full power of C++.

In order to interface with Agg, you have to be able to write (not just use) templated classes. I don't think even head of Cython allows that, and its a hard problem. I think Cython is just completely the wrong approach for interfacing with that kinds of C++ template programming, but it could be hidden at a layer below.

Therefore, I think our best way out of a PyCXX dependency is to rewrite the extension code in two layers -- a C++ layer and a regular C layer that could then be interfaced with Cython (or the raw Python/C API). There are also parts of the code that predate matplotlib's dependency on Numpy where buffers are manually managed, that should probably just be replaced with Numpy arrays. (FT2Image, for example).

As for this specific PR, I think it can be safely closed. We can't abandon our local copy until our patches for Python 3 support are merged upstream. Even once that's done, I'm working on an entirely different way to handle external dependencies in my setup_rework branch, and the way to optionally use the system PyCXX library will probably change as a result of that.

Thanks for the clarification @mdboom