I can't repro the problem locally: the background gets converted to white, not to black, in my case.
In any case, it would seem that a better fix would be for the testing machinery to not ignore the alpha channel if it is present, no? Does something like the following work for you?

diff --git i/lib/matplotlib/testing/compare.py w/lib/matplotlib/testing/compare.py
index 665e055f12..1119cf10f8 100644
--- i/lib/matplotlib/testing/compare.py
+++ w/lib/matplotlib/testing/compare.py
@@ -371,6 +371,11 @@ def calculate_rms(expected_image, actual_image):
 # 16-bit depth, as Pillow converts these to RGB incorrectly.
 
 
+def _load_image(path):
+    img = Image.open(path)
+    return np.asarray(img if img.mode == "RGBA" else img.convert("RGB"))
+
+
 def compare_images(expected, actual, tol, in_decorator=False):
     """
     Compare two "image" files checking differences within a tolerance.
@@ -435,9 +440,9 @@ def compare_images(expected, actual, tol, in_decorator=False):
         actual = convert(actual, cache=True)
         expected = convert(expected, cache=True)
 
-    # open the image files and remove the alpha channel (if it exists)
-    expected_image = np.asarray(Image.open(expected).convert("RGB"))
-    actual_image = np.asarray(Image.open(actual).convert("RGB"))
+    # open the image files
+    expected_image = _load_image(expected)
+    actual_image = _load_image(actual)
 
     actual_image, expected_image = crop_to_same(
         actual, actual_image, expected, expected_image)
@@ -486,9 +491,8 @@ def save_diff_image(expected, actual, output):
     output : str
         File path to save difference image to.
     """
-    # Drop alpha channels, similarly to compare_images.
-    expected_image = np.asarray(Image.open(expected).convert("RGB"))
-    actual_image = np.asarray(Image.open(actual).convert("RGB"))
+    expected_image = _load_image(expected)
+    actual_image = _load_image(actual)
     actual_image, expected_image = crop_to_same(
         actual, actual_image, expected, expected_image)
     expected_image = np.array(expected_image).astype(float)

That's odd. It's noteworthy that in #22852, the latest commit (22faffc) contains two wrong images (one each for Computer Modern and DejaVu Sans), and the tests still passed on CI!

I'll try with these suggested changes, but perhaps it is more important to reproduce the problem first?

Just checked. These changes work; the tests fail if the text is changed. Probably unrelated, but this made me think: since we know we are loading PNG images, if it is guaranteed that they will have a transparency layer, we can avoid checking for it and just compare the images directly. That way, if savefig stops adding the transparency, we'd know immediately. (Unless there already is a test for that?)

After d23db61, all checks (other than PR cleanliness) have passed, which can be considered a reproduction of the issue?

Was there a good reason the transparency was not being considered?

@anntzer I'm not sure if I should be the one to make your suggested changes to this PR, since it was your idea and effort!:sweat_smile:

It's OK, I donate the patch above to you if it works, feel free to give it a try. (Sorry, I may not have the time to investigate the repro for now, although I agree it would be nice to figure out why it only occurs sometimes.)
My guess is that back when the test machinery was introduced, transparency support was more brittle and therefore skipped? Haven't checked the git history, though.

The CI failures are legitimate. Looks like you need to add an out so that a fully opaque (alpha==255) RGBA image can compare equal to a RGB image?

Understood; I'll take a look at the failures and check.

I updated the function like this.

def _load_image(path):
    img = Image.open(path)
    if img.mode == "RGBA" and img.getextrema()[3][0] < 255:
        return np.asarray(img)
    return np.asarray(img.convert("RGB"))

One test (test_bbox_inches_tight_suptile_legend[svg]) fails with the error: Image sizes do not match expected size: (569, 885, 4) actual size (569, 885, 3). This is a genuine error—not due to these changes, but one which was already present. It was not detected earlier because the alpha channel was ignored. Here's what their alpha channels look like.

The actual image is completely opaque; the expected image isn't—its rightmost column has transparency (note the value reported at the mouse position).

This transparency problem is a lot more complicated than I thought! On the current main branch, this is what I get when I run test_bbox_inches_tight_suptile_legend[svg].

• bbox_inches_tight_suptile_legend-expected.svg has a transparent strip at its right edge.
• bbox_inches_tight_suptile_legend.svg also has a transparent strip at its right edge, but it is thinner. Update: this might be an artefact of my image viewer, so this point and the the two questions below are moot.
• bbox_inches_tight_suptile_legend_svg.png does not have the transparent strip.

Questions:

• Why does the actual SVG have transparency? As far as I can see, the test doesn't do anything to make that happen.
• Why does the actual SVG-converted-to-PNG have no transparency?

(Small update: only Linux tests failed, so might it be something to do with Inkscape? I encountered the transparency problem on Linux Mint 20.2 and Manjaro Linux.)

@tacaswell A visual inspection of the history of this image shows that that latest commit f15c26b might have introduced the transparent strip. Could you throw some light here?:sweat_smile:

After manually converting the actual and expected SVG images (using Inkscape) to PNG, I believe that the original image should not have a transparent strip.

My very vague memory (refreshed from looking at #9817) is that after changing from ints -> floats in the svg without any other changes. If it affected anything with transparency that was unintentional.

My guess is that the source of the strip is related to rounding / snapping on the clip box.

It is oddly re-assuring how much our SVGs have changed while still producing the same visual result!

Thank you @tfpf !