Repository navigation
Image interpolation wrong for pixel values exceeding vmax #8631
Description
Activity
- added this to the milestone on May 16, 2017
No, I do not expect #8300 to help with this.
The issue is that Agg does some aggressive clipping (not sure exactly when in the interpolation steps) so I am not sure that the 2.0 case is actually interpolating correctly (ex a value that should have been scaled to 2 in the normalized units is clipped to 1 before it is interpolated).
More generally, poisoning every screen pixel that include out-of-range source pixels seems like the 'right' thing to do.
Hi,
thanks for all responses.
Indeed, changes of commit 1a7c4ef induced the new behaviour, and it seems this was intentionally. I am missing the background why. But from my point of view this very strict handling of over/under values is not a good way: My current use case is displaying real-world grayscale images, to adjust contrast I set vmin/vmax. Some pixel values are outside this range, with the new behaviour they mess up the result, with the old they just saturate - a very natural behaviour.With some testing and looking at the sources, I could observe that Agg clips values after interpolation, in this sense interpolation is done correctly, only the information about under/over values is lost. To recover this, I am currently playing with an approach to compare the interpolation result of the raw image and one with values scaled by 0.5. If the interpolation results don't match up, there must have been some clipping. Do you think this makes sense?
Or would it be better to modify Agg, such that for float data clipping can be suppressed?Gregor
Doing the interpolation twice is almost certainly too expensive. Comparing where pixels == 1 or 0 in the output plane and are in the mask where they have been poisoned might be a better approach? I am definitely open to changing this (I was thinking much more about nan / inf when I did this before) and it would break some of the degeneracy in over/under regions overlapping.
Another reasonable idea may be to just scale down the input to [.1, .9] so we can do the clipping our selves.
This will still not correctly capture pixels where the interpolation pushes the result value out of range despite everything around it being with-in range (which I think some of the higher-order interpolations can do.
We have a vendored version of the last non-gpl version of Agg which has slightly divereged from upstream. We are not the only project with a vendored-and-changed Agg and it does cause problems (like seg-fault problems) so I am wary of making aggressive changes to Agg it's self, but using different functions from Agg is on the table.
- Am 23.05.2017 um 15:05 schrieb Thomas A Caswell ***@***.***>: Doing the interpolation twice is almost certainly too expensive.At the moment, interpolation is actually performed four times, for the raw image data, and over/under/missing mask, so I don’t see this is really an obstacle.Comparing where pixels == 1 or 0 in the output plane and are in the mask where they have been poisoned might be a better approach? I am definitely open to changing this (I was thinking much more about nan / inf when I did this before) and it would break some of the degeneracy in over/under regions overlapping.I played with this, this tends to give too many false attributions of over/under pixels after interpolation, especially for larger interpolation kernels and sparse over/under pixels, since in this case the poisoned pixels cover much larger area. Interpolation for missing/invalid/masked seems much more involved, as interpolation is likely give unreasonable results, e.g. by replacing missing by 0. I think there should be better ways to handle this, but thats a different topic.Another reasonable idea may be to just scale down the input to [.1, .9] so we can do the clipping our selves.Thats my favourite approach.This will still not correctly capture pixels where the interpolation pushes the result value out of range despite everything around it being with-in range (which I think some of the higher-order interpolations can do.You could introduce some safety margin to determine over/over values to reduce these kind of false positives.We have a vendored version of the last non-gpl version of Agg which has slightly divereged from upstream. We are not the only project with a vendored-and-changed Agg and it does cause problems (like seg-fault problems) so I am wary of making aggressive changes to Agg it's self, but using different functions from Agg is on the table.Ok, if you are open for changes, I would like to play with the approaches above and report back on my findings. It would be helpful if you have some test patterns at hand that demonstrate the problematic cases.…— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#8631 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7Gs3UCKH91JxdeY6Nme0nmn_3RmmXBks5r8tmhgaJpZM4NcWvN>.
First results, I tried the approach to scale the image values to 0.25-0.75 before interpolation, so clipping for the unscaled image can be detected, for a proper assignment of under/over mask. This needs very few changes
diff --git a/lib/matplotlib/image.py b/lib/matplotlib/image.py index b80f3dd..d2c6cde 100644 --- a/lib/matplotlib/image.py +++ b/lib/matplotlib/image.py @@ -375,8 +375,10 @@ class _ImageBase(martist.Artist, cm.ScalarMappable): # this is to work around spurious warnings coming # out of masked arrays. with np.errstate(invalid='ignore'): - rgba[..., 1] = np.where(A < 0, np.nan, 1) # under data - rgba[..., 2] = np.where(A > 1, np.nan, 1) # over data + # rgba[..., 1] = np.where(A < 0, np.nan, 1) # under data + # rgba[..., 2] = np.where(A > 1, np.nan, 1) # over data + rgba[..., 1] = A * 0.5 + 0.25 #scale raw image to 0.25...0.75 + # Have to invert mask, Agg knows what alpha means # so if you put this in as 0 for 'good' points, they # all get zeroed out @@ -457,9 +459,11 @@ class _ImageBase(martist.Artist, cm.ScalarMappable): invalid_mask = ~output.mask * ~np.isnan(output.data) # relabel under data. If any of the input data for # the pixel has input out of the norm bounds, - output[np.isnan(hid_output[..., 1]) * invalid_mask] = -1 + # output[np.isnan(hid_output[..., 1]) * invalid_mask] = -1 # relabel over data - output[np.isnan(hid_output[..., 2]) * invalid_mask] = 2 + hid_nonclipped = (hid_output[..., 1]-0.5)*2 + 0.5 # interpolation without clipping + output[hid_nonclipped < -0.05] = -1 # with safety margin + output[hid_nonclipped > 1.05] = 2 output = self.to_rgba(output, bytes=True, norm=False)
Demonstration of the outcome for different strength of the outliers (1%, 10%, 100%, 1000% of max) with over/under masking based on values after interpolation (new approach):
for comparison result with mpl 2.0.0, where over/under mask relies on interpolating the raw over/under mask:
I like it! The oscillations around the strong peak from the higher-order kernels is wild, but I think correct.
Why rescale the values in plane 1, wouldn't it be simpler to just compare them against [.25, .75]?
I also do not understand the 5% margin.
Is the over/under pair in the top-right symetric? It looks like the red one is a bit weaker than the cyan one.
Reacted by Gregor Thalhammer-Thurner- modified the milestones: , on Jun 17, 2017
- marked this as a duplicate of Interpolating with imshow makes some squares appear on plot #8964 on Jul 30, 2017
I have a dev branch where we interpolate on the raw data before normalization which seems to do better:
import numpy as np from matplotlib import pyplot as plt from matplotlib import cm import copy W=np.random.randn(200,200) cmap = copy.copy(cm.get_cmap('seismic')) cmap.set_over('k') cmap.set_under('m') plt.imshow(W, interpolation='bicubic', cmap=cmap, vmin=-1, vmax=1) plt.colorbar(extend='both')
import matplotlib.pyplot as plt import numpy as np cmap = copy.copy(cm.get_cmap()) cmap.set_over('k') cmap.set_under('r') t = np.zeros((50, 50)) t[10, 10] = 1 t[10, 20] = 2 plt.imshow(t, vmin=-0.01, vmax=1, interpolation='lanczos', cmap=cmap)
Which seems better in both cases (I set the over/under values to see the wild things that these higher-order interpolations do)
Also does better on the pathological stress-test, but still some issues at the edges. Also not clear why the bottom left shows more ringing that the top left...
Hi,
Thanks for tackling this issue! Missed your comment from May 25, sorry. Here, for completeness, the
gist for the interpolation test I used to create the test images, nothing special.The aim of the 5% safety margin I used in my patch was to somewhat mitigate excessive marking regions as over/under due to ringing, or due to rounding errors. After playing more with this I think it is not needed, does not give a significant improvement.
Gregor
- modified the milestone: on Aug 6, 2017
Is it asking too much to have the fix come out in a 2.0.3 bug-fix release? I think this was the original idea, but then apparently 2.0.3 is not happening anymore.
This bug seems kind of critical and, while we could downgrade to 2.0.0, I think it's worth it to have it fixed soon, as opposed to waiting longer for 2.1 to be ready.
Reacted by Nicolas BudynThere will be no 2.0.3, but there should be a 2.1rc1 by the end of the month. This is one of the few remaining blocking issues.
@tacaswell I already did. I haven't tested it myself to be 100% sure but it appears to be inline with my issue.
- added this to the milestone on Aug 31, 2017
Bug report
Bug summary
With imshow(), image interpolation gives wrong results near pixels with values exceeding vmax. A large rectangular region (given by the interpolation kernel size) is filled with color corresponding to maximum value.
Code for reproduction
see gist
Actual outcome
Expected outcome
Ok with matplotlib 2.0.0
Matplotlib version
matplotlib installed via conda-forge