Document input requirements on `Colormap.call` · Issue #7129 · matplotlib/matplotlib · GitHub

Hi,

Experimenting my usual list of test on Matplotlib-2.0.0b4, I get this error

Matplotlib-2.0.0b4, Python-3.5.2, Windows10, Ipython-5.1.0, Ipywidget-5.2.2, Numpy-1.11.1 and pandas-0.19.0rc1 of of cgohlke,
WinPython, matplotlib-3.0.0b4 from cgohlke
piece of code and image below
was ok in WinPython with Matplotlib-1.5.2 (only Matplotlib may have changed since this)

Error:

%matplotlib inline
# Pandas interactive
import pandas as pd
import numpy as np
import seaborn as sns

# create a df with random datas
np.random.seed(24)
df = pd.DataFrame({'A': np.linspace(1, 10, 10)})
df = pd.concat([df, pd.DataFrame(np.random.randn(10, 4), columns=list('BCDE'))],
               axis=1)
df.iloc[0, 2] = np.nan

# interactive
from IPython.html import widgets
@widgets.interact
def f(h_neg=(0, 359, 1), h_pos=(0, 359), s=(0., 99.9), l=(0., 99.9)):
    return (df
             .style
             .background_gradient(
                cmap=sns.palettes.diverging_palette(
                     h_neg=h_neg, h_pos=h_pos, s=s, l=l, as_cmap=True)
             ).highlight_null()
           )

C:\WinPython\basedir35\buildQt5\winpython-64bit-3.5.x.3\python-3.5.2.amd64\lib\site-packages\matplotlib\colors.py:486: RuntimeWarning: invalid value encountered in less
  cbook._putmask(xa, xa < 0.0, -1)

Do you have any idea what values are going in when the warning is raised? If I am reading in right, it looks like the warning is coming out of the xa < 0 .

the seed paramater generates always the same subset:

And there IS no error if I tag the nan affectation:

#df.iloc[0, 2] = np.nan

maybe it's a practice that is no more allowed ? maybe it's a rather a pandas bug ?

I am pretty sure that this is coming out of numpy. What version of numpy/pandas do you have installed?

Can you reproduce this without pandas, jupyter, or seaborn?

Colormap.__call__() is assuming it's input contains no unmasked invalid values. Normally this doesn't cause a problem because we use masked_invalid in enough places that invalid values don't find their way into the call argument. Evidently we need to call masked_invalid in even more places, such as this.

I'm using Numpy-1.11.1 and pandas-0.19.0rc1 of of cgohlke.

This test was the what's new in pandas 0.18 presentation of Jeff Reback ... can't find the youtube. Just to say there was no issue with that with previous Pandas/Matplotlib

ok:

the error is there even if I back-pedal down to Matplotlib-1.5.2,
the error disappears if I back-pedal from Pandas-0.19rc1 down to Pandas-0.18.1

(numpy being untouched)

==> wouldn't it be rather a Pandas issue ?

This is some very strange interactions between the libraries. I am 90% sure that the warning is actually coming out of numpy, but very confused why changing the pandas version affects it.

So I could trim the reproducing example down to df.style.background_gradient().
What this background_gradient method more or less does under the hood is the following (and this may have changed):

#s = df['C']
s = pd.Series(np.random.randn(10)); s[0] = np.nan
rng = s.max() - s.min()
cmap = 'PuBu'
low, high = 0, 0
from matplotlib import colors
norm = colors.Normalize(s.min() - (rng * low),
                                s.max() + (rng * high))
normed = norm(s.values)
c = [colors.rgb2hex(x) for x in plt.cm.get_cmap(cmap)(normed)]

And it is the last line that triggers the warning. So in any case anything to do with pandas itself anymore (but maybe with a wrong way we call things in matplotlib)

So in the above example, normed is a masked array but where the data includes a NaN. It is this were the colormap gives a warning:

In [3]: plt.cm.get_cmap('viridis')(np.array([np.nan, 1, 0.5]))
/home/joris/miniconda3/envs/dev/lib/python3.5/site-packages/matplotlib/colors.py:581: RuntimeWarning: invalid value encountered in less
  cbook._putmask(xa, xa < 0.0, -1)
Out[3]: 
array([[ 0.267004,  0.004874,  0.329415,  1.      ],
       [ 0.993248,  0.906157,  0.143936,  1.      ],
       [ 0.127568,  0.566949,  0.550556,  1.      ]])

In [5]: plt.cm.get_cmap('viridis')(np.array([np.nan]))
Out[5]: array([[ 0.267004,  0.004874,  0.329415,  1.      ]])

A strange thing is that it does not give the warning when it is a array with a single nan value.

Is cmap supposed to be able to deal with NaNs? Or should we in pandas filter them out first?

Digged a little bit further, in the call method of Colormaps, basically the following happens:

In [16]: from matplotlib import cbook

In [17]: X = np.array([np.nan, 1, 0.5])

In [18]: xa = np.array([X])

In [19]: cbook._putmask(xa, xa < 0, -1)
/home/joris/miniconda3/envs/dev/bin/ipython:1: RuntimeWarning: invalid value encountered in less
  #!/home/joris/miniconda3/envs/dev/bin/python

In [20]: xa < 1
/home/joris/miniconda3/envs/dev/bin/ipython:1: RuntimeWarning: invalid value encountered in less
  #!/home/joris/miniconda3/envs/dev/bin/python
Out[20]: array([[False, False,  True]], dtype=bool)

So it is indeed just coming from numpy by this comparison with NaNs

I am unclear what changed that there is now a warning where there was not before.

Since it is coming straight from numpy (the last line in my example, pure numpy), I am also not sure.

Ah, pandas 0.19.0rc1 no longer suppresses certain numpy warnings as it did before: pandas-dev/pandas#13145. That is probably the reason.

It should be masked on the pandas side, we should clarify the docs on __call__ if needed.

@jorisvandenbossche Thanks for doing the leg work to track this down, it is greatly appreciated.

No problem! OK, then I keep the issue open on the pandas side and repurposed it to deal with NaN values before passing them to cmap

cbook._putmask no longer exists, and Colormap.__call__ masks NaNs:

matplotlib/lib/matplotlib/colors.py

Line 606 in 4d1cd5e

mask_bad = X.mask if np.ma.is_masked(X) else np.isnan(X) # Mask nan's.

The single line reproducer plt.cm.get_cmap('viridis')(np.array([np.nan, 1, 0.5])) doesn't warn either, so is there really anything left to do here?

This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

Document input requirements on `Colormap.call` · Issue #7129 · matplotlib/matplotlib · GitHub

Repository navigation

Document input requirements on Colormap.call #7129

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Document input requirements on `Colormap.__call__` · Issue #7129 · matplotlib/matplotlib · GitHub

Repository navigation

Document input requirements on Colormap.__call__ #7129

Description

Activity

tacaswell commented on Sep 18, 2016

stonebig commented on Sep 18, 2016

stonebig commented on Sep 18, 2016

tacaswell commented on Sep 18, 2016

efiring commented on Sep 19, 2016

stonebig commented on Sep 19, 2016

stonebig commented on Sep 20, 2016

tacaswell commented on Sep 20, 2016

jorisvandenbossche commented on Sep 20, 2016

jorisvandenbossche commented on Sep 20, 2016

jorisvandenbossche commented on Sep 20, 2016

tacaswell commented on Sep 20, 2016

jorisvandenbossche commented on Sep 20, 2016

jorisvandenbossche commented on Sep 20, 2016

1 remaining item

tacaswell commented on Sep 20, 2016

tacaswell commented on Sep 20, 2016

jorisvandenbossche commented on Sep 20, 2016

QuLogic commented on Jul 8, 2021

github-actions commented on Mar 29, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Document input requirements on `Colormap.call` · Issue #7129 · matplotlib/matplotlib · GitHub

Document input requirements on Colormap.call #7129