Looks pretty good. There is still a difference between the png and the pdf; I suspect neither of them is quite right in the boundary region between the unmasked and masked areas. The png version looks worse; it has lines in that regions. The pdf just has a blurred gray. The following images are the mpl native png output, and then a png exported via preview from the pdf. I have looked at the actual pdf, though, and this export is a reasonably faithful representation. The difference between the following images is a difference between the original files, not an artifact of the pdf-to-png export.

Above: the native mpl png. Below: the png exported from the mpl pdf.

I agree that the grey looks wrong, but it could be a result of alpha blending with the white background. The alpha channel is effectively Gouraud interpolated between adjacent vertices, but I wonder if it should just be set to zero for the whole triangle where at least one vertex is masked out. I think that would correspond better to my intuition of how to show missing data, but I suppose in some use cases you might want to blend continuously with the background.

The Agg results look like the renderer is doing linear interpolation for alpha inside the triangles but drawing the edges with opaque colors.

Of course the grey color is because all of the RGBA channels are interpolated between the data color and zero. If you zoom in (on either the png or the pdf) the grey areas closest to the data are tinted with the data color. Here's a variant where the masked data is given another color:

from matplotlib import cm, pyplot as plt
import numpy as np
import matplotlib
matplotlib.use('agg')

X, Y = np.meshgrid(
    np.linspace(0, 10, 31),
    np.linspace(0, 10, 8)
)
mesh = plt.pcolormesh(
    X,
    Y + 3*np.sin(X),
    np.ma.masked_where(np.abs(Y-5)<2, X),
    shading='gouraud',
    cmap='jet',
)
mesh.get_cmap()._rgba_bad = (1.0, 0.0, 1.0, 0.3)

plt.savefig('example.png')
plt.savefig('example.pdf')

png and pdf images:

The question is, who are using pcolormesh and for what purpose, and what do they want to happen with missing data? With shading='flat' the missing quads are simply painted with the _rgba_bad color, and I think that would be the natural thing to do with Gouraud shading too, instead of interpolating the RGBA tuple between data values and _rgba_bad.

Nice example. Your second version (the pdf) makes much more sense; the lines in the first plot (the png) can't be what anyone would want.
I don't know who is using the gouraud shading, and for what--and in particular, with masks.

A Github search isn't very enlightening. There are lots of copies of Matplotlib's _axes.py and quadmesh_demo.py around. Here are two interesting results I spotted:

https://github.com/gravmath/PAD/blob/a58277f2367dae574c16f207c37d31027e3fe7ab/notebooks/Ion%20Flow%20Munging%20and%20Analysis-Jun%208.ipynb

This one plots masked data with flat shading, then there's a cell with a gouraud plot but unfortunately it's not executed so we can't see what the result is.

https://github.com/akpetty/SeaIcePrediction/blob/c248d0b3594fbe920dc9888f966c9d4f6dc5afcc/Scripts/gridding/GriddingDemo.ipynb

This one uses Gouraud shading for data where the missing parts are filled with zeros, then flat shading for a masked array. I suspect they wouldn't like the interpolated grey for the hole at the North Pole.

To make typical masking work as one might expect with G shading, maybe it would be necessary to special-case alpha==0? If alpha == 0 on any vertex of a triangle, set alpha to zero for the entire triangle?
@anntzer would you provide a simple example illustrating how your mplcairo handles, this, please?

mplcairo gets you
(png)

(pdf)
test_pcolormesh.pdf
(png)

(pdf)
example.pdf

Thank you. Well, it looks like mplcairo is not the solution. It isn't working at all for gouraud shading in the first pdf example (at least as rendered by preview). In the second example, it has the same sort of difference between png and pdf as backend_pdf.

The first example works for me?
This is what I get:

(haven't really followed the discussion, but at least it's not giving you the big black area of #14327 (comment))

This is what I get when I export to png from preview:

And this is what I get on the screen from preview:

Conclusion: preview on Mojave is buggy. It looks like cairo is using pdf features that preview can't handle.

I can't do anything about that :/
Can you try with chrome?

I'm on High Sierra and I see

It's difficult to say if Preview is wrong or if there's a problem in the pdf file. None of the pdf reader applications I have tried give any error messages for subtly malformed files, so developing a pdf writer is a lot of guesswork.

In any case, this PR adds alpha channel support to the draw_gouraud_triangles method in the pdf backend, which was missing before. I think the best place to fix the rest of pcolormesh is not in the backends but in the code that replaces missing data with zeros.