Discussion: Use of masked arrays internally · Issue #31008 · matplotlib/matplotlib · GitHub

Chatting with @timhoffm in #31004 (comment), I'd like to more widely discuss our use of masked arrays in internal code. My position is that they add a lot of maintenance headache and fragility, and we would be best off excising them from the codebase altogether.

Currently we have 185 counts of np.ma in the core library. Most of them are pass-through checks, or are immediately .filled() with nans.

Problems

Internal APIs do not usually document whether they expect or can handle masked arrays. It's easy to miss needing to handle the masked array case, contributors may not be familiar with them, and numpy functions still work so we can fail silently
Lots of code clutter and specific paths needed to handle masked arrays
Poor test coverage of masked array inputs/outputs
Performance degradation from passing around extra data and not being able to rely on more efficient array operations.
Note also that a = np.ma.masked_array(a, mask=mask).filled() is slower than a = np.where(mask, np.nan, a) is slower than a[mask] = np.nan.

Proposal

Replace all instances of masked arrays with float arrays filled with NaNs, or in instances where the underlying data needs to be preserved (eg when clipping) then use an explicitly defined boolean mask variable
Sanitize all user inputs to pure numpy arrays and do not return masked arrays
Document a dev policy to minimize internal use of masked arrays, and to never return them

Migration issues

cbook.safe_masked_invalid() is used a few places and causes some other functions to return a masked array. I haven't traced this all the way through to see where those come out
All colors.Normalize subclasses preserve input masks in their output (this is the only docstring I see with a masked array output documented for the public API)

Deprecation warnings might be difficult to raise without spamming regular uses of these functions, since we can't tell when users use the mask on returned arrays. We may need to accept some silent breakage and just make the release notes clear.

Overall I'd be ok with this as I think masked arrays are a concept that never caught on

I think the objection is going to be that we have the concept of "invalid" color in colormaps, and it is not the same as missing or masked data. I think this is a valid distinction but I'm not sure if anyone uses it in real life data analyses or publications. I'd love to see published examples!

I come back from 2 days of "masked headache". I had issues in:

using np.percentile asI had to fill the masked values with nan and use np.nanpercentile.
usng sklearn functions as sklearn doesn't support masks
and something else that now I don't remember.

Next time I will think 2 times before introducing masked arrays.

As a user I'd expect matplotlib to continue supporting masked arrays as input from the typical high level plotting functions, as they can arise from many different sources, especially when plotting meteorological observations.

However, I don't really care whether they are maintained within matplotlib, and I've never considered using the arrays returned from matplotlib for anything other than passing to other plotting functions, as it is typically at the end of my processing.

So if the only thing matplotlib does with a masked array is to infill it with NaNs, that seems fine to me.

As a user, I'm fully on board with this. The ecosystem friction with np.ma is real, and simplifying the internals seems like a win for stability.
plt.plot(masked_array) needs to keep working because a lot of upstream libraries (like netCDF4) output masks by default. I really don't care what happens internally or what format gets returned, as long as I don't have to manually call .filled(np.nan) every time I want to make a plot.

The only spot where "sanitize to NaNs" worries me is with masked integers (e.g. categorical data or counts). Converting those to floats just to handle missing values can get messy with precision or memory, so the explicit boolean mask option mentioned in the proposal seems strictly necessary there.

Regarding the colormap question—I honestly can't recall a time I needed to visually distinguish between "invalid" (math error) and "masked" (missing data). They are both just "no data" to the viewer.

Colormaps: While we have _get_rgba_and_mask(), the rgba values of masked entries is the "bad" color and the mask is not used in the colormap. I therefore believe we currently do not have any visual distinction between NaN and masked values

matplotlib/lib/matplotlib/colors.py

Line 779 in ec3e68d

rgba, mask = self._get_rgba_and_mask(X, alpha=alpha, bytes=bytes)

The only spot where "sanitize to NaNs" worries me is with masked integers (e.g. categorical data or counts

In those cases, it's likely an option to discard the respective data points.

More generally, I suspect, we currently do not have any special visualization for masked points. In the end, they are always either discarded or handled like NaN (which in some cases is again the same).

Re: Colormaps — That makes sense. If the visual output is identical, collapsing the "bad" vs "masked" distinction is safe.

Re: Integers — I would caution against the idea of "discarding" data points for integers.

For scatter, discarding works fine. But for plot (lines), removing a point changes the topology: it connects the neighbors instead of leaving a gap. To draw the gap, the renderer explicitly needs a placeholder in the data stream.

Since we can't use NaN as that placeholder for integers (without casting to float and risking precision loss), we likely need a dedicated mechanism for these paths—perhaps passing an explicit valid boolean array alongside the data, rather than relying on sentinel values or filtering.

What does "NaN" stand for? It stands for "Not a Number", and is used in computing to represent a mathematical calculation that cannot be shown as a number. e.g., 0/0.

Clearly "NaN" should not be used to represent missing data in a dataset, as then it would be impossible to tell apart data that has been deliberately set as being "missing", or data that is the result of a mathematical operation that is unrepresentable as a number.

In my field, climate science, missing data in source data (before getting to python) is usually set to a number well outside the range of data being presented and plotted, e.g., -9999, -1e+30. It is extremely problematic when I come across datasets with "NaN"s in them as I don't know if that was deliberate or an error!

Using "NaN" as some sort of replacement for masked arrays is not the way forward.

@mo-garethsjones this proposal will not affect your data set, only (potentially) your plots. Do you have examples where Matplotlib currently gives you a different result for masked points vs Nan points?

@rcomer For floats, NaN and Masked usually render identically (invisible).

However, for integers, there is a hard difference because integers cannot hold NaN. To use NaN, we would have to cast the data to float (changing the type/precision) or "discard" the masked points.

If we "discard" points in a line plot, the rendering changes completely: Masked draws a gap, Discarded connects the neighbors. So for integer line plots, the distinction is visually critical.

Colormaps: While we have _get_rgba_and_mask(), the rgba values of masked entries is the "bad" color and the mask is not used in the colormap. I therefore believe we currently do not have any visual distinction between NaN and masked values

I'm surprised that is the case, but you are correct. I'd have expected below that missing data (middle cell) would remain "transparent" while the NaN cell would become red.

Given this, I think the argument for keeping the distinction between NaN and masked is even weaker...

X = np.arange(15.0).reshape((3, 5))
fig, ax = plt.subplots(3,1, sharex=True, layout='constrained', figsize=(3, 3))
ax[0].pcolormesh(X, cmap='viridis', shading='nearest')

Xm = np.ma.masked_where(X==7, X)
Xm[2, 3] = np.nan
ax[1].pcolormesh(Xm, cmap='viridis', shading='nearest')

cmap = plt.get_cmap('viridis').with_extremes(bad='red')
ax[2].pcolormesh(Xm, cmap=cmap, shading='nearest')

In my field, climate science, missing data in source data (before getting to python) is usually set to a number well outside the range of data being presented and plotted, e.g., -9999, -1e+30. It is extremely problematic when I come across datasets with "NaN"s in them as I don't know if that was deliberate or an error!

@mo-garethsjones This is indeed the usual argument for distinguishing masked and NaN data. But do you actually find that extremely problematic? I'm in your field, and everyone I know uses xarray to handle multi-dimensional data sets. xarray does not have the concept of masked data versus NaN despite being developed well after masked arrays were introduced to numpy. In fact if you have a netcdf file with both NaN and fill_value in it, the distinction will not survive a round trip with xarray (the NaN will get converted to fill_value). I take this as a pretty strong signal that while formally there is a distinction to be made between NaN and missing data, practically the difference is not worth the extra code headaches.

However, as @rcomer says, I don't think we are proposing to stop accepting masked arrays as an input, just that they will be converted to NaN-filled non-masked arrays for internal use very early in the lifescycle.

With respect to integers - at some point all our data gets converted to floats - there is no reason to keep the arrays internally as integers.

@jklymak The pcolormesh demonstration is convincing—visually, the handling of "bad" vs "masked" is effectively consistent there.

at some point all our data gets converted to floats - there is no reason to keep the arrays internally as integers.

While true for final screen coordinates, enforcing an early cast to float64 (to support NaN) poses a data integrity risk for large integers. float64 loses integer precision above $2^{53}$ ($\approx 9 \times 10^{15}$).

Common inputs like nanosecond timestamps (if treated as raw ints) or large database IDs often exceed this range. If we convert these to float internally to handle missing values, we effectively corrupt the data before the Locators or Formatters can see it. Preserving the integer type (via a mask) ensures that the axis labels remain accurate even if the render pipeline eventually floats the coordinates.

@sanrishi locators and formatters and our conversion pipeline for axes-mappable data types are actually a good question. I'm not sure how we currently handle those, and agree there is no such thing as NaN in integer arrays. Note there is NaT for datetime64 arrays.

Options for integers:

carry an extra mask
use a special sentnel value - Since we control the data, we can basically always find an unused value and use that.
If a masked array: If values are not too large, convert to float; error out otherwise (somewhat limits support for ints, so needs a deprecation)
drop support for masked int arrays completely (with deprecation).

Remarks:

1, 2 will only be feasible after encapsulating the data (see our data prototype). Otherwise we'd need a lot of special casing on internal processing code.
Apart from Line2D where "missing" breaks the connection between points, are there other Artists that make that distinction? If not, there maybe could be a Line2D internal solution (e.g. introducing an internal notion of segments; i.e. list of arrays, so that "missing" data separate segments).
pandas has introduced a nullable integer https://pandas.pydata.org/docs/user_guide/integer_na.html. However, from what I've seen, this is a lot of hassle and we should try to avoid that if possible. (Though accepting it like we accept a masked array may be ok).

@timhoffm Regarding the options:

Option 2 (Sentinel): This seems risky for performance. To guarantee a sentinel is truly unused, we'd potentially need an $O(N)$ scan of the data on every draw, which defeats the purpose of optimizing the pipeline.

Option 3 (Convert/Error): This effectively creates a "feature trap" where plotting works for small integers (IDs/timestamps) but crashes or corrupts as data scales up. Preserving int64 precision seems critical for scientific/engineering use cases.

Regarding Artists: fill_between and stackplot share the Line2D topology constraint. A missing value implies a generic "gap" in the filled area. If we drop the point (instead of masking it), the fill would bridge the gap, which falsifies the visualization.

So i think option 1 is best, even if it requires the data prototype to handle cleanly.

I'm not tracking the point about lines introducing gaps where other artists don't. We explicitly support this behavior, and there's no difference between nans and masked array inputs. To get a bridge over the gap, the user needs to strip out nans before plotting. See: https://matplotlib.org/devdocs/gallery/lines_bars_and_markers/masked_demo.html

The axis converters convert to float pretty early. Datetime64 particularly has a resolution problem with nanoseconds that has been noted a few times.

Similarly, we do not preserve int64 resolution in any meaningful way:

base = np.int64(2**60)
arr = np.array([base, base + 1, base + 2, base + 3], dtype=np.int64)
fig, ax = plt.subplots()
ax.plot(arr)
ax.set_ylim([base - 1, base + 4])

The last command is (almost) a no-op and certainly cannot zoom in to see the int64 information - that information is long lost above 2^45 or so. If someone needs to preserve int64 resolution around some large number, they need to apply an offset before passing to Matplotlib.

So unless we change our pipeline, the current state is that we convert to float pretty early, and that has a loss of resolution for int64-based input that is just part of dealing with computers.

I'm not tracking the point about lines introducing gaps

The point is about int arrays with masks. We could convert to float, which would allow missing values as NaN. But that potentially looses precision. Alternatively, we could stick to int and strip out the masked values. But that would connect previously separated segments.

In other words

To get a bridge over the gap, the user needs to strip out nans before plotting

Exactly: If we would do the stripping internally to get rid of the masked values we would create that bridging through an implementation detail (internal handling of masked ints), which is undesired.

@timhoffm @jklymak Agreed. Since stripping masked integers causes bridging (breaking topology) and they can't hold NaNs, Option 1 (Sidecar Mask) is the only path that preserves the visual structure. The Data Prototype (Remark 1) seems like the perfect place to encapsulate this logic cleanly.

@sanrishi my point was that we lose int64 precision early in the pipeline, so filling masks for NaN is fine to happen at the same time.

A possible benefit of data encapsulation is not losing the precision, and hopefully the data encapsulation work is taking masking into account.

@jklymak The constraint is integer arrays: you can't insert np.nan without casting to float, and stripping the masked values creates bridging:

# Integer data with masked value at index 2
x = np.array([1, 2, 3, 4], dtype=np.int32)
y_masked = np.ma.array([10, 20, -999, 40], mask=[0, 0, 1, 0])

plt.plot(x, y_masked, 'o-')  # Current: gap at x=2 ✓

# If we strip the masked value:
plt.plot([1, 2, 4], [10, 20, 40], 's--')  # Bridges from x=2 to x=4 ✗

I agree the Data Prototype is the right architectural home for preserving topology metadata.

The constraint is integer arrays: you can't insert np.nan without casting to float, and stripping the masked values creates bridging:

Yes - I'm suggesting that we insert NaN when it gets converted to float, which means we do not need to carry around masked arrays internally.

Yes - I'm suggesting that we insert NaN when it gets converted to float, which means we do not need to carry around masked arrays internally.

We already to that

matplotlib/lib/matplotlib/cbook.py

Lines 1365 to 1373 in ec3e68d

    
           def _to_unmasked_float_array(x): 
        
               """ 
        
               Convert a sequence to a float array; if input was a masked array, masked 
        
               values are converted to nans. 
        
               """ 
        
               if hasattr(x, 'mask'): 
        
                   return np.ma.asanyarray(x, float).filled(np.nan) 
        
               else: 
        
                   return np.asanyarray(x, float)

Line2D is a bit complicated in that it already keeps the original data (_xorig, _yorig) and maintains a cached version of the normalized floats in _xy:

matplotlib/lib/matplotlib/lines.py

Lines 674 to 687 in ec3e68d

    
           def recache(self, always=False): 
        
               if always or self._invalidx: 
        
                   xconv = self.convert_xunits(self._xorig) 
        
                   x = _to_unmasked_float_array(xconv).ravel() 
        
               else: 
        
                   x = self._x 
        
               if always or self._invalidy: 
        
                   yconv = self.convert_yunits(self._yorig) 
        
                   y = _to_unmasked_float_array(yconv).ravel() 
        
               else: 
        
                   y = self._y 
        
               self._xy = np.column_stack(np.broadcast_arrays(x, y)).astype(float) 
        
               self._x, self._y = self._xy.T  # views

So actually, Line2D is fine.

IIRC @efiring was a proponent of masked arrays, perhaps he can comment here? (sorry if this isn't the case, it's just a fuzzy memory...)
I personally don't see much use for them.

@anntzer, your memory is correct. Dating back at least to Nov., 2005 (commit 6587aec, based on a cursory git search) I have been involved in the implementation of masked array support in matplotlib. This was pre-numpy; we had to support Numeric and numarray via "numerix", our compatibility shim. It was also before there was any usable NaN support. Searching for "nan" (case-insensitive) in the repo at that revision turns up no instances related to "Not a Number". Instead there are examples of how to use masked arrays to plot a line with gaps, for example.

Even if there had been Matlab-style NaN support in the early Python numeric libraries, I would have worked for masked array support. There are advantages of masked arrays over resorting to NaN for missing or invalid data.

Masking works with any dtype, including integers of any size, and, for example, Python strings (as objects). I work with instrumentation that encodes output in integers of sizes from 1 to 4 bytes.
Masking is a natural approach when applying editing algorithms. This does not require the use of masked arrays, but they are a convenience, potentially saving quite a bit of manual manipulation of data arrays and masks. I used to use Matlab; I prefer what I can do with masked arrays.
Having masked arrays means one doesn't need a whole new set of "nan" functions.
Personal aesthetics: to me, using "nan" is a hack; direct support for missing or invalid data via masking (and potentially via sentinels) is more elegant.

I was disappointed to see the cancellation of Mark Wiebe's work towards integrating mask support into numpy.

I view the absence of mask functionality in xarray as a shortcoming, not a feature (though in general I am a fan of xarray).

Regarding masked arrays in matplotlib:

Internally, if operations can now be done more efficiently (in terms of performance and overall code quality), without reducing any functionality or changing plot outputs, by using some combination of explicit masks and NaNs, then I would support that change.
Facing outward, continuing to support masked array inputs is essential for the foreseeable future.

Eric

@greglucas do you know what became of the ndarray_ducktypes MaskedArray? IIRC at one point there was talk of it replacing numpy's internal masked arrays.

@greglucas do you know what became of the ndarray_ducktypes MaskedArray? IIRC at one point there was talk of it replacing numpy's internal masked arrays.

Nope, it looks like it hasn't been touched in several years now. There was a discussion a few years ago to try and figure out how to better handle np.ndarray subclassing vs duck-typing which I think a lot of the discussion has now moved into the array_api discussions.

I have a few PRs up in numpy trying to add the array-ufunc and array-function semantics to MaskedArray which would make them more interoperable and not need all these special special masked function calls, but I don't think there is a large appetite for those (In particular I think it isn't the direction Astropy wants to go). The array_ufunc made it into numpy once or twice and had to be reverted because there was a few downstream packages it broke with corner cases if I remember right.

numpy/numpy#22914
numpy/numpy#22913

I'm not sure I have a huge preference one way or the other on the issue at hand here. Another thing that we should think about is not just masked arrays but gpu-arrays and other fancy things and how we want to handle that. I think that we might be best off looking at how big of a lift it would be to do the array_namespace stuff xp = x.__array_namespace__() and do operations with whatever native array library was passed in. This might also help us get rid of some of the jax / pytorch special checks we have for things too. The reason I bring this up is because it seems like someone could make a MaskedArray with a specific namespace that follows the array API spec and then we'd be able to handle it all.

I'm all for accepting any array that supports the array API spec.

We can discuss here where we want to go. IMHO any implementation will need to be part of or follow-up to the data prototype work.

I think there are actually two different use cases being conflated under the term "masked array", and they may call for different designs.

1. Genuinely missing data: "there is no known value here"
This is the common semantics of NA / SQL NULL: the value is not part of the information available to us.

For this use case, I think a nullable dtype is a better abstraction than a mask over an existing value. I'm exploring representing NA as a reserved bit pattern of the wrapped dtype, so that the result still behaves like an ordinary ndarray at the storage/array level.

The important semantic property here is that once a value is missing, the underlying value should not be recoverable. "Hiding" a value destroys it; there isn't supposed to be a hidden value waiting underneath the NA.

2. Complete data, selectively hidden: "there is a value here, but this particular view does not expose/use it"

Some examples: access control/redaction; algorithms that temporarily exclude observations;...

For this use case, I think an explicit mask attached to an ndarray is a more natural abstraction than a nullable dtype. I have been experimenting with this thought: essentially refactoring ndarray to have an optional mask field (defaulting to no mask), with the intention that this could eventually replace much of what np.ma provides without requiring a separate ndarray subclass.

This is semantically different from missing data. The underlying value still exists and should remain recoverable; the mask describes the current view of the data.

I think np.ma is trying to accommodate both semantics under the same abstraction, which may be why its conventions feel somewhat ambiguous. That distinction also makes me hesitant about treating "MaskedArray -> float array + NaN" as the general solution.
My main point is that it may be useful to distinguish these two semantics when discussing what should replace np.ma internally:

             "masked array"
                    │
          ┌─────────┴─────────┐
          │                   │
      missing data       selective masking
          │                   │
     nullable dtype      data + mask
       (NA/NULL)        (view metadata)

Matplotlib may then be able to choose the appropriate representation (or only accept one type of semantic) at its API boundary and use a more explicit internal data abstraction, rather than requiring one representation to serve both purposes.

For all our purposes there is no difference between “missing” and “hidden”. Both are just not shown. Matplotlib is only a consumer of data. We typically copy, so that external references cannot change the data under us. And internally, we don’t need to unhide any data. Therefore we can choose nullable data as the internal representation.

Discussion: Use of masked arrays internally · Issue #31008 · matplotlib/matplotlib · GitHub

Repository navigation

Discussion: Use of masked arrays internally #31008

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def _to_unmasked_float_array(x):
	"""
	Convert a sequence to a float array; if input was a masked array, masked
	values are converted to nans.
	"""
	if hasattr(x, 'mask'):
	return np.ma.asanyarray(x, float).filled(np.nan)
	else:
	return np.asanyarray(x, float)

	def recache(self, always=False):
	if always or self._invalidx:
	xconv = self.convert_xunits(self._xorig)
	x = _to_unmasked_float_array(xconv).ravel()
	else:
	x = self._x
	if always or self._invalidy:
	yconv = self.convert_yunits(self._yorig)
	y = _to_unmasked_float_array(yconv).ravel()
	else:
	y = self._y

	self._xy = np.column_stack(np.broadcast_arrays(x, y)).astype(float)
	self._x, self._y = self._xy.T # views

Discussion: Use of masked arrays internally · Issue #31008 · matplotlib/matplotlib · GitHub

Repository navigation

Discussion: Use of masked arrays internally #31008

Description

Activity

jklymak commented on Jan 21, 2026

MO-marcomilan commented on Jan 22, 2026

jfrost-mo commented on Jan 22, 2026

sanrishi commented on Jan 22, 2026

timhoffm commented on Jan 22, 2026

sanrishi commented on Jan 22, 2026

mo-garethsjones commented on Jan 22, 2026

rcomer commented on Jan 22, 2026

sanrishi commented on Jan 22, 2026

jklymak commented on Jan 22, 2026

sanrishi commented on Jan 22, 2026

jklymak commented on Jan 22, 2026

timhoffm commented on Jan 22, 2026

sanrishi commented on Jan 22, 2026

scottshambaugh commented on Jan 22, 2026

jklymak commented on Jan 22, 2026

timhoffm commented on Jan 22, 2026

sanrishi commented on Jan 22, 2026

jklymak commented on Jan 22, 2026

sanrishi commented on Jan 22, 2026

jklymak commented on Jan 22, 2026

timhoffm commented on Jan 22, 2026

anntzer commented on Jan 22, 2026

efiring commented on Jan 23, 2026

rcomer commented on Jan 23, 2026

greglucas commented on Jan 24, 2026

timhoffm commented on Jan 25, 2026

tuan2k33 commented on Sep 18, 2026

timhoffm commented on Sep 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions