imshow in 2.1: color resolution depends on outliers? · Issue #10567 · matplotlib/matplotlib · GitHub

Since 2.1 the effective color resolution of imshow(x, vmin=-1, vmax=1) seems to depend on large outliers in x.

Is this intended behavior?

Thanks for clarification!

import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt

x = np.linspace(-1, 1, 500 )
x = np.ones([20, 1]) * x[np.newaxis, :]

x1 = x.copy()
x1[0, 0] = 1e16

x2 = x.copy()
x2[0, 0] = 1e17

_, axes = plt.subplots(nrows=3)
axes[0].imshow(x, vmin=-1, vmax=1)
axes[1].imshow(x1, vmin=-1, vmax=1)
axes[2].imshow(x2, vmin=-1, vmax=1)

axes[0].set_title(mpl.__version__)
plt.show()

Actual outcome

Expected outcome

Matplotlib version

Operating system: Linux
Matplotlib version: 2.1.0 (expected behavior in 2.0.2)
Matplotlib backend: Qt5Agg
Python version: 3.6.4
Other libraries: numpy 1.13.3
via conda

Hmm, I'm not sure about whether this is intended or not, it bisects to 12c27f3 so maybe @tacaswell will know more?

Comment in that new code: ``` # This may introduce numeric instabilities in very broadly # scaled data ``` Looks like we might have found that situation. The data is getting scaled for the purposes of the AGG renderer (somewhat independent of the norm that was specified). Unfortunately, due to some fun effects with floating point numbers, that normalization gets very digitized when the range is very large. I would say that this is unintentional, but I am not sure how we can fix it, though.

…

On Thu, Feb 22, 2018 at 12:03 PM, David Stansby ***@***.***> wrote: Hmm, I'm not sure about whether this is intended or not, it bisects to 12c27f3 <12c27f3> so maybe @tacaswell <https://github.com/tacaswell> will know more? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10567 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARy-OVeKeKqxTPE6_6m-ZIB0IJ_JJ3Uks5tXZ3ogaJpZM4SPgUA> .

We messed around w/ all of this for 2.1.2. #10133 W/o looking carefully, does that fix this?

Pretty sure it does - it was a round off error that caused this, but its now been fixed for large floats.

It didn't seem fixed when I ran the above test on the master branch.

agreed. The image is stored as float64. You lose resolution at 2^53=9e15. I’m not sure how this worked in 2.1, but that’s the problem here.

the rescaling is relatively recent, iirc, as part of the work to better handle over/under/invalid.

…

On Thu, Feb 22, 2018 at 4:15 PM, Jody Klymak ***@***.***> wrote: agreed. The image is stored as float64. You lose resolution at 2^53=9e15. I’m not sure how this worked in 2.1, but that’s the problem here. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10567 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARy-Chvzb8jOCgsCtG6TG-vonUm2Oreks5tXdjUgaJpZM4SPgUA> .

I tried to set the dtype to float 128 fro big numbers, but I was told there is no such thing...

Image interpolation is the bug that just keeps giving!

The chain of bugs and fixes here goes:

while doing 2.0 we noticed that we were interpolating after doing norm + rgb mapping -> leads to colors not in the color map in the final image so we moved the interpolation to be done on the normed, but not color mapped data.
this introduced imshow interpolation uses masked values #8012, so we better masked around 'invalid' pixels when color mapping [MRG+1] Fix imshow masked interpolation #8024
as a consequence of this over/under/invlaid pixels effectively poisoned everything in their in their kernel window (Image interpolation wrong for pixel values exceeding vmax #8631) so we moved interpolation to the raw data (Fix image interpolation #8966)
we now have numerics issues for data with large dynamic range (this bug + imshow doesn't properly display some images #10072 which was fixed by FIX: Image scaling for large dynamic range ints #10133)

It’s the same bug, it’s just that #10133 only fixed up to 10^16 or so.

Is there some reason to not apply the norm (but not RGB mapping) first? If I do that, it seems to work for this case. It fails a bunch of the image tests meant to catch the other bugs, but I'm not clear why my version is worse than the test version...

Test

Moving the norm step up (pre interp)

The ringing / saturation in the lower left of the test is correct. If you normalize first you are effectively clipping the impact of the outliers.

Well, we could do something like pre-clip the data to some suitably large number around the vmin/vmax, but not 17 orders of magnitude. That should preserve both behaviours

Or folks could mask their invalid data instead of passing in huge (I assume) invalid values.

I would not assume that huge data points are automatically invalid. In our application we have values between -1 and 1 in most of the image, but it can include regions with values approaching Inf.

I would find it strange to mask some of the data before plotting them. I'm also not aware that you have to do something like this in Matlab or gnuplot.

See #10613 for a fix....

closed by #10613

imshow in 2.1: color resolution depends on outliers? · Issue #10567 · matplotlib/matplotlib · GitHub

Repository navigation

imshow in 2.1: color resolution depends on outliers? #10567

Test

Moving the norm step up (pre interp)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

imshow in 2.1: color resolution depends on outliers? · Issue #10567 · matplotlib/matplotlib · GitHub

Repository navigation

imshow in 2.1: color resolution depends on outliers? #10567

Description

Activity

dstansby commented on Feb 22, 2018

WeatherGod commented on Feb 22, 2018

jklymak commented on Feb 22, 2018

jklymak commented on Feb 22, 2018

dstansby commented on Feb 22, 2018

jklymak commented on Feb 22, 2018

WeatherGod commented on Feb 22, 2018

jklymak commented on Feb 22, 2018

tacaswell commented on Feb 23, 2018

jklymak commented on Feb 23, 2018

jklymak commented on Feb 23, 2018

Test

Moving the norm step up (pre interp)

tacaswell commented on Feb 23, 2018

jklymak commented on Feb 23, 2018

hagenw commented on Feb 23, 2018

jklymak commented on Feb 26, 2018

tacaswell commented on Mar 3, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions