`Normalize` should not need to define an `inverse()` method · Issue #7320 · matplotlib/matplotlib · GitHub

The colors.Normalize class currently defines an inverse() method (http://matplotlib.org/devdocs/api/colors_api.html#matplotlib.colors.Normalize -- by the way it is not documented), which maps the normalized value (i.e., index into the colormap) back to image data value.

In practice this method is only used in one place: the colorbar is drawn by taking uniformly spaced points in normalized space, transform back into data space (at the end of Colorbar._process_values()), then later drawn by calling pcolormesh on such data values (applying the normalization back into normalized space). I guess the reason for this is that one may want to draw the colorbar using user-specified data values instead.

However, this could just as easily be done by always working internally in normalized values: by default, just plot uniformly spaced normalized values; if the user specifies the values to use, just normalize them first.

The advantage of making such a change was highlighted to me by #7294, a PR which proposes supporting arbitrary normalization functions. While there are a number of other points I make in that PR, one issue is that this PR requires the user to specify both the normalization function and the inverse normalization function (in order to provide inverse). But, as mentioned above, with a fix to the way colorbars are drawn, there wouldn't be a requirement for the user to specify the inverse normalization (which may be nontrivial to compute).

I think there are two separate cases:

The user specifies where he wants the ticks. In that case you are completely right, there is no need at all for the inverse.
- The user gives values in the data space.
- The direct function is calculated to obtain the normalised values.
- The normalised values are used for the tick positions.
- The original values in the data space are used for the tick labels.
The user does not specify an inverse function. In this case we have to do something automatically. There are multiple criteria but something that every criteria will have in common is that ticks and labels have to be "roughtly" uniformly distributed through the colorbar, to avoid having ticks too close together so the labels do not ovelap. An example of this is the typical logscale ticks, where the ticks are not strictly speaking equispaced, but still the distribution of ticks is roughly uniform. To achieve this is normally necessary define some uniform intervals, in the colormap space, and the translate that into the data space, to "get an idea" of how the intervals stretched differently. Of course, some guesses of this could also be done using the direct function, but it would implied doing it iteratively:
- Starting with a tick at vmin, and calculate f(vmin).
- We try the next tick at vmin+dv, where this is equal to some "desirable" number.
- Calculate f(vmin+dt), and check how far is from f(vmin).
- If it too close, if not, we go back to step 2 increasing dv.
- If it too far, if not, we go back to step 2 increasing dv.
- If it is good, we keep the tick and try to find another tick.

Considering that the inverse function would only be useful for this, and there is an algorithm to solve it without the inverse function (It essentially calculates numerically the inverse function), I agree that it may not justify having to provide an inverse function. However, strictly speaking, the inverse function it is necessary.

Maybe a middle term would be for the inverse method to numerically calculate the inverse function in cases where it is not provided.

EDIT: For a more developed version of this functionality that I have been developing in the past few days: https://github.com/alvarosg/pynverse

I spent some time today making a function to obtain the inverse of a function.

It works like this:
vmin,vmax=0,1
f=lamda x: x**2
finv=getinv(f,vmin,vmax)

where finv is a callable which can take an array or a scalar, and return the numerical inverse for all points.

As you can see it works quite well:

generated with

N=50
square=lambda x: x**2
cos=lambda x: -np.cos(x)
log=lambda x: np.log10(x)
funcs=[(square,0.,1.),(cos,0.,np.pi),(log,0.001,1)]
fig,axes=plt.subplots(len(funcs),2,figsize=plt.figaspect(0.5*len(funcs)))
for i,(func,vmin,vmax) in enumerate(funcs):
    x1=np.linspace(vmin,vmax,N)
    y1=func(x1)
    axes[i][0].plot(x1,y1,'.-')

    funcinv=getinv(func,vmin,vmax)
    y2=np.linspace(y1.min(),y1.max(),N)
    x2=funcinv(y2)
    axes[i][1].plot(x2,y2,'.-')
axes[0][0].set_title('Calculating direct')
axes[0][1].set_title('Calculating inverse')
plt.show()

And this is the function itself.

import matplotlib.pyplot as plt
import scipy.optimize
import numpy as np

def getinv(f,vmin=0.,vmax=1.):
    vmin=float(vmin)
    vmax=float(vmax)
    midval=(vmax+vmin)/2
    def inv(xin):
        xin=np.asarray(xin)
        shapein=xin.shape
        xin=xin.flatten()

        results=xin.copy()
        for j in range(xin.size):
            optimizer=lambda x, j=j,f=f: (((f(x)-xin[j]))**2).sum()
            result=scipy.optimize.minimize_scalar(optimizer,bounds=(vmin,vmax), method='bounded')
            results[j]=result.x

        return results.reshape(shapein)
    return inv

Of course it will not work for weird functions, but it works really well for monotonic functions, which is always the case for normalizations. So maybe we could implement something like this for the inverse method. Of course, it is a bit slower than direct analytic calculation (even this plot is generated instantaneously), but considering that the inverse function is not used that much..., I guess someone caring about speed could always explicitly indicate the inverse function.

The main problem is that matplotlib does not depend on scipy, and I am not sure if there is any other matplotlib dependence with a fast minimizer....

attn @efiring

I don't think there is any way to avoid the inverse while preserving all of the present behavior. Specifically, the midpoints of the boundaries in data space (the "values") are normed and then used to look up the colors.
Ways of handling this:

Require an inverse for every norm. It looks like ENH: Added FuncNorm and PiecewiseNorm classes in colors #7294 is handling this nicely. I don't think it is an onerous requirement.
1a) Possibly, as a fallback, use a crude but adequate inverse generator using numpy only. We should not add a scipy dependency. I would look for a reasonably efficient way of generating a table to be used for linear interpolation.
Require that if one uses a norm without an inverse, one specifies the boundaries in the call to colormap.
Write a simpler version of colormap and don't worry about matching every feature and present behavior.

I don't see any strong reason to deviate from the first option for now.

Unless I am mistaken, the process you are describing ("Specifically, the midpoints of the boundaries in data space (the "values") are normed and then used to look up the colors.") does not require the inverse of the transform, only the forward transform. Again, the process I propose for drawing the colorbar (for a continuous colorbar -- discrete colorbars are of course in a completely separate category) is:

Draw a uniform colorbar using the cmap, limited to the interval transform(vmin) and transform(vmax).
Select points to label. Currently, this is done using a Locator instance -- MaxNLocator or AutoLocator in all cases except for log normalization which is special cased to use LogLocator. Alternatively, use the points that were explicitly passed in. In either case, apply the transform to them to know where to draw the ticks.

I don't see where you'd need the inverse.

I think you are making an artificial distinction between a continuous colorbar and a discrete one. All of our colorbars are discrete, but the default cmap LUT size of 256 is nearly continuous, so we think of them as being continuous. There is nothing to stop a person from making a colormap with a LUT size of 30, or 3, however, and as the size decreases, the difference in behavior increases between what we have now and what you are proposing. Your proposal might be a better design, but it is option 3. Maybe now is a good time to switch to it. I think that doing it consistently it would entail changing the color selection in discrete colorbars so that the interpolation into the LUT would be done with the midpoint between normed boundaries instead of with the norm of the midpoint between boundaries. I don't recall whether I originally chose the latter to preserve pre-existing behavior, because of some other perceived advantage, or because it was simpler at the time.

I agree that I confused discrete transforms and discrete colormaps in the previous message.

However, I think there's still a fundamental misunderstanding here. My argument is that, as long as the user does not specify boundaries explicitly (which is the most common case), you can draw the colorbar just from knowing the colormap, and completely independently of everything else: you just list all the colors in the colormap (for the case of a LinearSegmentedColormap, take the colors corresponding to, say, 256 points uniformly spaced in [0, 1]), and draw a small rectangle of the same size for each of these colors on top of each other (of course this can be done using pcolormesh as is the case currently). You then label the axis, using whatever ticker is appropriate.

If the user passes in both boundaries and values, then again there is no problem: use the normalization to map both of them to [0, 1], and draw correctly sized rectangles with the colors corresponding to the user-specified values.

If one of boundaries or values is given, but not the other: in fact the behavior in this case is not documented:

    The following will probably be useful only in the context of
    indexed colors (that is, when the mappable has norm=NoNorm()),
    or other unusual circumstances.

    ============   ===================================================
    Property       Description
    ============   ===================================================
    *boundaries*   None or a sequence
    *values*       None or a sequence which must be of length 1 less
                   than the sequence of *boundaries*. For each region
                   delimited by adjacent entries in *boundaries*, the
                   color mapped to the corresponding value in values
                   will be used.
    ============   ===================================================

However, experimenting a bit indicates that the behavior is the one that you describe. For example,

plt.imshow([[1, 11, 99, 1000]], cmap=ListedColormap(["red", "green", "blue"]), norm=LogNorm())
plt.colorbar(boundaries=[1, 18, 100, 1000])

plots a 3-color (red, green, blue) colorbar whereas

plt.imshow([[1, 11, 99, 1000]], cmap=ListedColormap(["red", "green", "blue"]), norm=LogNorm())
plt.colorbar(boundaries=[1, 20, 100, 1000])

makes the bottom two entries both green, suggesting that the color of the bottommost entry is obtained by mapping (1+18)/2 (<10) and (1+20)/2 (>10) respectively (the actual boundaries of the colormap being at values 10 and 100).

As a side note, I'm not sure how to make values work if boundaries isn't given... both passing non-normalized values

plt.imshow([[1, 11, 99, 1000]], cmap=ListedColormap(["red", "green", "blue"]), norm=LogNorm())
plt.colorbar(values=[5, 15, 105])

and passing normalized values

plt.imshow([[1, 11, 99, 1000]], cmap=ListedColormap(["red", "green", "blue"]), norm=LogNorm())
plt.colorbar(values=[.1, .2, .3])

currently raise an exception. But perhaps that should be another issue, and one should just disallow passing values without passing boundaries? (because it's not clear how big the rectangles should be drawn, in any case)

Coming back to the case where only boundaries is given: if you want to use the color corresponding to the normalized midpoint value, you still don't need the inverse: you literally compute the midpoint, apply the normalization, and use the corresponding color.

I would, in fact, think that using the midpoint of the normalized values instead (the new behavior that you describe, and that is not a necessary consequence of the switch I propose, as I tried to argue above), is consistent with the new way image colormapping is done (first normalize, then resample to the correct resolution, then map to colors). But regardless of the rationale, this is just as easy to implement: you normalize, compute the midpoint, and use the corresponding color. But this would entail back-compatibility issues, and I don't really care about uses of boundaries and values separately from one another, so I'm only (barely) +0 on such a change (basically, I think it makes the behavior of a completely underspecified case a bit more intuitive).

TL;DR: AFAICT, it should be possible to generate colorbars exactly identical to the current ones while dropping any use of a transform's inverse (up to the floating point error that the current method suffers from, as it successively applies the inverse and then the transform itself to get back to the original value).

However, I think there's still a fundamental misunderstanding here. My argument is that, as long as the user does not specify boundaries explicitly (which is the most common case), you can draw the colorbar just from knowing the colormap, and completely independently of everything else: you just list all the colors in the colormap (for the case of a LinearSegmentedColormap, take the colors corresponding to, say, 256 points uniformly spaced in [0, 1]), and draw a small rectangle of the same size for each of these colors on top of each other (of course this can be done using pcolormesh as is the case currently). You then label the axis, using whatever ticker is appropriate.

I think I now understand the problem.. Essentially what you propose transfers the responsibility of implementing the inverse function, to having the TickerLocator, like LinearLocator or LogLocator. In those cases, the designers of each of those locator have good knowledge of the function and the inverse for a linear transformation and a logarithmic transformation.

Doing it like in your proposal, would involve making a new ticker locator class for every new normalization. Now, if we want to support custom normalization, like FuncNorm, or any other class inheriting from Normalize, this would mean the user having to define a custom locator for each case. So in the end, you need either the inverse function, which may be used to give a good guess of tick locations, or to explicitly say how to automatically set ticks defining a Locator.

Your comment is correct and does clarify my position. In fact, a posteriori, it seems obvious (in the mathematical, not "it's dumb" sense) that it is a Locator class that should be used to, well, locate ticks?

Pushing the argument further, what we really want is not only a custom Locator subclass per normalization, but in fact a custom Scale subclass (http://matplotlib.org/devel/add_new_projection.html) this would, in particular, include the inverse normalization (the inverse Transform) and the custom formatter (which is necessary at least for log normalization and its variants (symlog, logit, etc.)).

Your work on #7294 on automatic tick location could then be integrated as some factory function that would construct such a scale and use it.

Ok, but then we agree that the inverse function for the transforms must be defined at least somewhere, I guess it makes sense for that to de defined along the normalization class, to make it compatible with custom classes.

Right now:
-Normalize does not need the inverse function becuase there is AutoMinorLocator(), which knows how to set ticks for linear normalizations.
-LogNorm does not need the inverse function becuase there is LogLocator(), which knows how to set ticks for log normalizations.
-SymLogNorm does not need the inverse function becuase there is SymmetricalLogLocator(), which knows how to set ticks for SymLog normalizations.

I agree, that for none of does the inverse method is really needed, because each of the individual locators already know everything they need to know about the inverse function.

Now following the same logic, we add FuncNorm. Following the same logic, then we would do:
-FuncNorm, and then we make a FuncNormLocator. However in this case we cannot hardcode the information about the inverse function in the locator class, because the function itself is not defined. Hence, we need to access the inverse fucntion.
-PiecewiseNorm, and then we make a PiecewiseNormLocator. This one would be very similar to the previous one, but with some extra logic to make sure that there are ticks in the piecewise range changes. Again, it will need to access the information about the inverse of the custom function.

So I agree that the inverse method is not really required for the existing normalization classes, but any more customizable classes will need this to at least be accessed by the corresponding locators, so we might as well encourage everyone to provide the inverse function, as in my opinion it is much better to have all mathematical aspects of the normalization within the normalization class, than spread between a Normalization, and a corresponding locator class.

By the way, in case any of you need a numerical inverse function, I just uploaded a package to PyPI with a function that does it:

https://pypi.python.org/pypi/pynverse

It is still built on SciPy though, so not useful for this hehehe, but it is quite versatile!

This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

I think this is largely fixed by the association of Norms w/ scales. Not all norms have scales, but most of the continuous ones now do. But maybe @anntzer could comment.

I guess(?) it may still be nice, conceptually, if we didn't draw the pcolormesh in data space (taking uniform points in axes space, inverse-transforming them to get the corresponding data values, and using that to seed the pcolormesh that will apply the norm/scale and get back uniform points in axes space), but instead directly draw it uniformly in axes space? (Note that this is only for continuous norms).

If you did that you lose the ability to change the yscale to something other than the norm's scale eg currently you can have a linear norm, set the scale to log and have the colorbar stretch accordingly.

Fair enough. Is putting the colorbar axes on a scale different from the norm's scale really supported? TBH I have not really been following the recent colorbar developments, so I'll trust your word on that; feel free to close this if this use case is indeed supported.

Yes, that is now the case, and was probably the primary motivation behind making colorbar axes as similar as possible to normal axes (and hence plotting the colors in data space). There is almost no special ticking now - they are just normal axes, except for some geometry to get the extends right:

import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots(1, 2, layout='constrained')
pc = ax[0].pcolormesh(np.abs(np.random.randn(30, 30)*1000))
cb = fig.colorbar(pc)

pc = ax[1].pcolormesh(np.abs(np.random.randn(30, 30)*1000))
cb = fig.colorbar(pc)
cb.ax.set_yscale('log')

Or quite often twoslope norm users just want the linear scale:

import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
import numpy as np

fig, ax = plt.subplots(1, 2, layout='constrained')
pc = ax[0].pcolormesh(np.random.randn(30, 30), norm=mcolors.TwoSlopeNorm(vcenter=0, vmin=-1, vmax=3), cmap='RdBu_r')
cb = fig.colorbar(pc)

pc = ax[1].pcolormesh(np.random.randn(30, 30), norm=mcolors.TwoSlopeNorm(vcenter=0, vmin=-1, vmax=3), cmap='RdBu_r')
cb = fig.colorbar(pc)
cb.ax.set_yscale('linear')

Ah, the twoslope case is a good example of why this is useful. Thanks for the clear explanation, and sorry for not keeping track :)

`Normalize` should not need to define an `inverse()` method · Issue #7320 · matplotlib/matplotlib · GitHub

Repository navigation

Normalize should not need to define an inverse() method #7320

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

`Normalize` should not need to define an `inverse()` method · Issue #7320 · matplotlib/matplotlib · GitHub

Repository navigation

Normalize should not need to define an inverse() method #7320

Description

Activity

alvarosg commented on Oct 21, 2016

alvarosg commented on Oct 23, 2016

tacaswell commented on Oct 27, 2016

efiring commented on Oct 28, 2016

anntzer commented on Oct 28, 2016

efiring commented on Oct 29, 2016

anntzer commented on Oct 29, 2016

alvarosg commented on Oct 29, 2016

anntzer commented on Oct 29, 2016

alvarosg commented on Oct 30, 2016

alvarosg commented on Nov 1, 2016

github-actions commented on Apr 1, 2023

jklymak commented on Apr 1, 2023

anntzer commented on Apr 2, 2023

jklymak commented on Apr 2, 2023

anntzer commented on Apr 2, 2023

jklymak commented on Apr 3, 2023

anntzer commented on Apr 3, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions