Fixed several accuracy bugs with image resampling by ayshih · Pull Request #30184 · matplotlib/matplotlib · GitHub

PR summary

This PR fixes several accuracy bugs with image resampling:

The filtering weight at one extreme is not being calculated correctly. In the case of linear interpolation, this makes the one weight that should be zero be instead nonzero. This then results in a pixels of constant values manifesting a small blip after linear interpolation if the center of an input pixel exactly coincides with the center of a output pixel (within the agg backend, its pixels may not be exactly aligned with the true pixels).
The filtering weights are not aligned correctly in their array as expected by subsequent code. For a weight array of N values, the "pivot" needs to be at N / 2 - 1, not at N / 2. This results in a bias where the linearly interpolated values are calculated slightly to the right of where they should be calculated.
Specific to nonaffine transforms, the values in the lookup table used to store the calculation are being truncated when they should be rounded. This results in a bias (smaller than the above) to the right.
[subsequently added, with updated images therein] Specific to affine transforms, the end point for interpolation is not specified correctly, which can slightly degrade the accuracy of the sampling locations.

Below are illustrative plots and the generating script for linear resampling of a 2-pixel array. There are fundamental limitations of the accuracy due to the subpixel approach of the agg backend, but the motivation here is for the average behavior to be accurate (i.e., eliminate bias).

Before this PR

After this PR

Script

import matplotlib.pyplot as plt
import numpy as np
from matplotlib.transforms import Affine2D, Transform
from matplotlib.image import resample, BILINEAR

in_data = np.array([[0.1, 0.9]])
in_shape = in_data.shape
in_edges = np.arange(in_shape[1] + 1)

out_shape = (1, 20)
out_edges = np.arange(out_shape[1] + 1)

ideal_data = np.array([[0.1, 0.1, 0.1, 0.1, 0.1, 0.14, 0.22, 0.3, 0.38, 0.46,
                        0.54, 0.62, 0.7, 0.78, 0.86, 0.9, 0.9, 0.9, 0.9, 0.9]])

# Create a simple affine transform for scaling the input array
affine = Affine2D().scale(sx=out_shape[1] / in_shape[1], sy=1)

# Create a nonaffine version of the same transform by compositing with a nonaffine identity transform
class NonAffineIdentityTransform(Transform):
    input_dims = 2
    output_dims = 2

    def inverted(self):
        return self
nonaffine = NonAffineIdentityTransform() + affine

affine_data = np.empty(out_shape)
nonaffine_data = np.empty(out_shape)
resample(in_data, affine_data, affine, interpolation=BILINEAR)
resample(in_data, nonaffine_data, nonaffine, interpolation=BILINEAR)

fig, axs = plt.subplots(3, 1, figsize=(4.8, 6.4), layout="constrained")

axs[0].stairs(in_data[0, :], in_edges)
axs[0].grid(ls='dotted')
axs[0].set_xticks(in_edges)
axs[0].set_title('Original data')

axs[1].stairs(affine_data[0, :], out_edges, label='affine')
axs[1].stairs(nonaffine_data[0, :], out_edges, ls='dashed', label='nonaffine')
axs[1].grid(ls='dotted')
axs[1].set_xticks(out_edges)
axs[1].legend()
axs[1].set_title('Interpolated data')

axs[2].stairs(affine_data[0, :] - ideal_data[0, :], out_edges, label='affine')
axs[2].stairs(nonaffine_data[0, :] - ideal_data[0, :], out_edges, ls='dashed', label='nonaffine')
axs[2].grid(ls='dotted')
axs[2].set_xticks(out_edges)
axs[2].set_ylim(-0.006, 0.006)
axs[2].legend()
axs[2].set_title('Interpolated data minus ideal result')

plt.show()

PR checklist

[N/A] "closes #0000" is in the body of the PR description to link the related issue
new and changed code is tested
[N/A] Plotting related features are demonstrated in an example
[N/A] New Features and API Changes are noted with a directive and release note
[N/A] Documentation complies with general and docstring guidelines

This line is one of the bugs. 0 and end are different distances from pivot, so the corresponding weights typically should not be the same.

Please put the new version under a macro guard, as was done in #28122, so that we keep a track of what's the original Agg and what we changed on top of it.

Makes sense, done

It turns out that I can't add a preprocessor definition in _image_resample.h a la #28122. There is buggy code in agg_image_filters.cpp that needs to be skipped, but since it is a CPP file rather than a H file, it gets compiled separately with no awareness of _image_resample.h. I have put the preprocessor definition (MPL_FIX_IMAGE_FILTER_LUT_BUGS) here at the top of agg_image_filters.h, which is lower level than would be nice, but at least still makes it clear what is custom code.

You could also put the define in the corresponding meson.build if you prefer.

Ah, that's a nicer approach. Done!

The filtering weight at one extreme is not being calculated correctly. In the case of linear interpolation, this makes the one weight that should be zero be instead nonzero. This then results in a pixels of constant values manifesting a small blip after linear interpolation if the center of an input pixel exactly coincides with the center of a output pixel (within the agg backend, its pixels may not be exactly aligned with the true pixels).

You can see such a blip in the example above, but it's easy to overlook. Here's a starker example: an array of two pixels with the same value (0.1) is resampled to an array of ten pixels. Before this PR, two of the pixels in the output array have values slightly greater than 0.1.

>>> import numpy as np
>>> from matplotlib.transforms import Affine2D
>>> from matplotlib.image import resample, BILINEAR
>>>
>>> in_data = np.array([[0.1, 0.1]])
>>> in_shape = in_data.shape
>>>
>>> out_shape = (1, 10)
>>>
>>> # Create a simple affine transform for scaling the input array
>>> affine = Affine2D().scale(sx=out_shape[1] / in_shape[1], sy=1)
>>>
>>> affine_data = np.empty(out_shape)
>>> resample(in_data, affine_data, affine, interpolation=BILINEAR)
>>>
>>> print(affine_data)
[[0.1        0.10001221 0.1        0.1        0.1        0.1
  0.10001221 0.1        0.1        0.1       ]]

After this PR, the output is the expected:

>>> print(affine_data)
[[0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]]

This killed the log-scale images?

Also this changed a bunch of image tests, but I can't see any difference by eye. Is it worth just relaxing the tolerance on those tests a bit and at the same time adding new tests for these fixes?

This killed the log-scale images?

Indeed, I need to track down why.

Also this changed a bunch of image tests, but I can't see any difference by eye. Is it worth just relaxing the tolerance on those tests a bit and at the same time adding new tests for these fixes?

Yes, the other 17 image tests are mostly subtle differences, but merely relaxing the tolerance doesn't make sense to me. The updated baseline images do represent what the output should be.

Fair enough - however, it would be good if there were some tests devised where it showed a visual effect of these inaccuracies.

The following code block is removed because it not only has the same bugs as the code in agg_image_filters.h, but also it shouldn't even exist because it modifies the weight array after it has been normalized, thus potentially destroying the normalization.

I fixed the issue with log_scale_image, so this PR is now ready for review. The final count is 65 baseline images that need to be updated. Maybe a third of those images have (subtle) changes in the rendering of content, e.g.:

Before this PR:

After this PR:

Difference:

On the other hand, most of the updated images have solely very slight changes in the appearance of text. As @jklymak suggested, it might make more sense in those cases to simply increase the tolerance instead for the comparison instead of updating the baseline image, so let me know if I should do that instead.

Are you sure you have the right set of images? lib/matplotlib/tests/baseline_images/test_axes/formatter_ticker_004.png for example doesn't have any images to be resampled.

OK, I see that RendererAgg::draw_text_image uses image_filter_spline36 when rotating text, so it only affects tests with rotated text. So for any tests that are purely rotated text, we should just increase the tolerance and make a note of them, then revert the tolerance changes in #30161 when regenerating the images. Or re-target this PR to the text-overhaul branch without test image changes (but then they'd all be regenerated much later.)

…rrect

I've updated this PR now that #30824 has been merged

Whew, thanks!

The tolerance on these was increased in various PRs, with notes to reduce/remove them, so they were removed in matplotlib#30184, but should have been reverted to a smaller value. - `test_floating_axes.py::test_curvelinear4[png]` fails on macOS - `test_grid_helper_curvelinear.py::test_axis_direction[png]` fails on ARM and macOS - `test_grid_helper_curvelinear.py::test_polar_box[png]` fails on Windows, ARM, and macOS These fail on macOS; there's a tolerance on them already, they fail by just a small amount, so increase it. - `test_patheffects.py::test_collection[png]` - `test_axes3d.py::test_scale3d_artists_log[png]`

The tolerance on these was increased in various PRs, with notes to reduce/remove them, so they were removed in matplotlib#30184, but should have been reverted to a smaller value. - `test_floating_axes.py::test_curvelinear4[png]` fails on macOS - `test_grid_helper_curvelinear.py::test_axis_direction[png]` fails on ARM and macOS - `test_grid_helper_curvelinear.py::test_polar_box[png]` fails on Windows, ARM, and macOS These fail on macOS; there's a tolerance on them already, they fail by just a small amount, so increase it. - `test_bbox_tight.py::test_bbox_inches_tight_suptile_legend[png]` - `test_patheffects.py::test_collection[png]` - `test_axes3d.py::test_scale3d_artists_log[png]`

The tolerances on these were increased in various PRs, with notes to reduce/remove them, so they were removed in matplotlib#30184, but should have been reverted to a smaller value: - `test_floating_axes.py::test_curvelinear4[png]` fails on macOS - `test_grid_helper_curvelinear.py::test_axis_direction[png]` fails on ARM and macOS - `test_grid_helper_curvelinear.py::test_polar_box[png]` fails on Windows, ARM, and macOS These fail on macOS; there's tolerances on them already, they fail by just a small amount, so increase them: - `test_bbox_tight.py::test_bbox_inches_tight_suptile_legend[png]` - `test_patheffects.py::test_collection[png]` - `test_axes3d.py::test_scale3d_artists_log[png]` This fails only on AppVeyor (Windows), but not Azure. There is a tolerance there already, so just add it to the list: - `test_mathtext.py::test_mathtext_rendering`

github-actions Bot added the topic: images label Jun 18, 2025

ayshih commented Jun 18, 2025

View reviewed changes

ayshih force-pushed the more_resample_bugs branch from d2fa1e5 to 7eb18b2 Compare June 18, 2025 13:54

github-actions Bot added the topic: mpl_toolkit label Jun 18, 2025

ayshih force-pushed the more_resample_bugs branch from 7eb18b2 to d9f8bb5 Compare June 18, 2025 14:35

github-actions Bot removed the topic: mpl_toolkit label Jun 18, 2025

ayshih force-pushed the more_resample_bugs branch from 7eb1cea to 5cbb7bb Compare June 18, 2025 15:31

ayshih force-pushed the more_resample_bugs branch 3 times, most recently from e198af6 to 717a221 Compare June 18, 2025 20:39

ayshih commented Jun 18, 2025

View reviewed changes

github-actions Bot added topic: mpl_toolkit topic: mplot3d and removed topic: mpl_toolkit labels Jun 19, 2025

ayshih force-pushed the more_resample_bugs branch 7 times, most recently from 43b912c to 90bd95f Compare June 19, 2025 19:20

ayshih marked this pull request as ready for review June 19, 2025 20:47

ayshih force-pushed the more_resample_bugs branch from 90bd95f to 9fdca21 Compare July 10, 2025 13:51

github-actions Bot added the status: needs rebase label Jan 8, 2026

ayshih added 10 commits January 8, 2026 10:46

Fix bugs in the calculation of the weight array

71bb56f

Fix bug with using the distortion table for nonaffine transformations

ca634d5

Move macro-guard definition to meson.build

e2f1a5f

Fix bug where the interpolation endpoint was one step further than co…

632e4e3

…rrect

Adjust test to use the downsampled image as reference

072f1a4

Loosen tolerances for text-only image differences

7b59187

Refactor bug fixes in the calculation of the weight array

131648a

Support differing subpixel alignment for nonaffine distortion table

acbedc6

Avoid affine code path when scaling will make subpixels visible

86100aa

Update baseline images

c19b295

ayshih force-pushed the more_resample_bugs branch from 48e28b8 to c19b295 Compare January 8, 2026 15:48

github-actions Bot removed topic: color/color & colormaps status: needs rebase labels Jan 8, 2026

QuLogic added this to the v3.11.0 milestone Jan 15, 2026

QuLogic merged commit cf2b13c into matplotlib:main Jan 15, 2026
38 of 41 checks passed

This was referenced Jan 15, 2026

Misalignment imshow vs. grid lines #1441

Closed

imshow pixel boundaries wrong when zoomed in #14143

Closed

ayshih mentioned this pull request Jan 21, 2026

TST: image comparison checks fail against matplotlib-dev astropy/astropy#19201

Closed

pllim mentioned this pull request Jan 21, 2026

TST: mpldev CircleCI job has 35 failures (Image files did not match) astropy/astropy#19170

Closed

ayshih mentioned this pull request Jan 23, 2026

Fixed inaccurate image placement and even more resampling bugs #31021

Merged

3 tasks done

ayshih deleted the more_resample_bugs branch March 2, 2026 14:02

QuLogic mentioned this pull request Mar 27, 2026

TST: Restore some tolerances for some arch/platform-specific failures #31343

Merged

1 task done

QuLogic mentioned this pull request Apr 30, 2026

Test failures with Matplotlib 3.11.0rc1 boutproject/animatplot-ng#31

Closed

+                              if(i <= pivot) m_weight_array[pivot - i] = value;
                           }
-                          unsigned end = (diameter() << image_subpixel_shift) - 1;
-                          m_weight_array[0] = m_weight_array[end];

                           }
                       }
+              #ifndef MPL_FIX_IMAGE_FILTER_LUT_BUGS

Fixed several accuracy bugs with image resampling by ayshih · Pull Request #30184 · matplotlib/matplotlib · GitHub

Repository navigation

Fixed several accuracy bugs with image resampling - #30184

Fixed several accuracy bugs with image resampling#30184QuLogic merged 10 commits intomatplotlib:mainmatplotlib/matplotlib:mainfrom ayshih:more_resample_bugsayshih/matplotlib:more_resample_bugsCopy head branch name to clipboard

Conversation

ayshih commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR summary

Before this PR

After this PR

Script

PR checklist

Uh oh!

ayshih Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anntzer Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

QuLogic Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jklymak commented Jun 18, 2025

Uh oh!

ayshih commented Jun 18, 2025

Uh oh!

jklymak commented Jun 18, 2025

Uh oh!

ayshih Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih commented Jun 19, 2025

Uh oh!

QuLogic commented Jun 26, 2025

Uh oh!

QuLogic commented Jun 26, 2025

Uh oh!

ayshih commented Jan 8, 2026

Uh oh!

Uh oh!

ayshih commented Jan 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Fixed several accuracy bugs with image resampling#30184
QuLogic merged 10 commits into
matplotlib:mainfrom
ayshih:more_resample_bugs

ayshih commented Jun 18, 2025 •
edited
Loading

ayshih Jun 18, 2025 •
edited
Loading

ayshih Jun 18, 2025 •
edited
Loading

ayshih commented Jun 18, 2025 •
edited
Loading