fig.savefig gives OverflowError for high DPI's · Issue #19325 · matplotlib/matplotlib · GitHub

Bug report

Even with short timeseries, fig.savefig() throws an OverflowError for high dpi's.
The example below uses 'Agg' with a significantly increased chunksize, yet it still throws an error for dpis > 250.

import numpy as np
import matplotlib
import matplotlib.pyplot as plt

matplotlib.use('Agg')
matplotlib.rcParams['agg.path.chunksize'] = 20000000

x = np.linspace(0, 100, 100000)
y = np.random.randint(1, 100, 100000)

fig = plt.figure()
plt.plot(x, y)

try:
    fig.savefig('Figure.png', format='png')
    print('Successfully plotted without dpi')
except Exception as e:
    print(e)
try:
    fig.savefig('Figure.png', format='png', dpi=500)
    print('Successfully plotted with high dpi')
except Exception as e:
    print(e)

The output here is:

Successfully plotted without dpi
Exceeded cell block limit (set 'agg.path.chunksize' rcparam)

Process finished with exit code 0

whereas I expected both to be successfully plotted, given that the chunksize OverflowError should only show when the number of plots exceeds the chunksize.

Matplotlib version

Operating system: Windows 10
Matplotlib version: 3.3.3
Matplotlib backend: agg
Python version: 3..9.1
Other libraries: NumPy 1.19.4
Everything is installed with pip

I can reproduce that, but it works fine with a much more modest chunk size of 2000. By chunking with N larger than your data set, you are basically not chunking.

I'll be honest, I don't know why you get an overflow. The limit is somewhere near 29350 on my machine before I have to chunk. I am somewhat suspicious we have a short-int sized buffer in there somewhere, which seems unnecessary, but maybe that is an agg limitation. No doubt @anntzer will tell us that cairo has no such limitation :-)

I think(?) getting rid of that limit is just a matter of

diff --git i/src/_backend_agg.cpp w/src/_backend_agg.cpp
index 6ae11167e..565cc9a62 100644
--- i/src/_backend_agg.cpp
+++ w/src/_backend_agg.cpp
@@ -45,7 +45,7 @@ RendererAgg::RendererAgg(unsigned int width, unsigned int height, double dpi)
       rendererBase(),
       rendererAA(),
       rendererBin(),
-      theRasterizer(8192),
+      theRasterizer(-1),
       lastclippath(NULL),
       _fill_color(agg::rgba(1, 1, 1, 0))
 {

(the argument is an unsigned upper bound, so -1 means "until ints overflow" :)) but of course that means that you'll run out of memory, instead of cleanly raising an exception (much) earlier, if you try to draw something too big. (Perhaps that's better?)

Or raise the limit given that it is likely tailored to relatively old machines? Like we don't want someone to thrash, but 100,000 data points seems small?

Agreed, but I'm not going to pick a limit for you here :)

Unfortunately I have no idea what that number means. Agg isn't very well documented so far as I can tell...

I assume it's the (max)size of some internal rasterization buffer, but I don't know either.

Running out of memory is definitely worse.

This is documented in an odd place: https://matplotlib.org/tutorials/introductory/usage.html#splitting-lines-into-smaller-chunks .

When we draw a Line2D eventually that falls through to render.draw_path(...). Due to details of Agg, there is a maximum number of vertices that we can feed to it (and I believe there is something in the draw process who's run time scales badly with the number of vertices we are feeding it).

An additional wrinkle is that the number of of verities that we feed the renderer is actually data dependent as internally the (x, y) data you provide is simplified to remove redundant vertices (also documented in a surprising place : https://matplotlib.org/tutorials/introductory/usage.html#line-segment-simplification). Thus if you change the data to

x = np.linspace(0, 100, N:=100000)
y = np.ones(N)

it saves both fine because the even though you are giving Matplotlib 100k points, internally we are probably only passing 2 points into the rasterizer (way down at the bottom). This also means that for "smooth" data, we get some automatic smart down sampling (e.g. np.sin(x) also works fine with this data because it is locally it is close enough to linear that the curve gets simplified to something that fits in to the renderer). In that sense the test data here is the worst-case scenario as it in general never goes in the same direction.

As of 2.1 (via #8776) we picked up "back tracking" simplification as well (so if you have data that is just going up and down in a column of the rendered image we only only keep the vertices at the top and bottom. In the normal case that is enough to us under the threshold, but not for the 500dpi case.

Tweaking the simplification threshold to be higher e.g.

matplotlib.rcParams['path.simplify_threshold'] = 1

also makes the noise example render (and it works down to 0.5 ish).

In #4404 @anntzer suggested making the chunk size automatic (e.g. it fails, we reduce it and try again until it works), but I was skeptical then and more skeptical now given that the simplification logic is another knob to reach for.

At a minimum I am going to put in a PR making that error message give better direction as to how to fix the problem!

I don't think allowing memory to fill is a good idea. OTOH if we can allow larger chunks because on-board memory has increased an order of magnitude, I dont see why we should burden users with twiddling anything. 100,000 points doesn't sound like a very big dataset in 2021.

The issue isn't the number of points in the dataset it is how much irreducible texture it has.

I chased this through and it is an upper bound

matplotlib/extern/agg24-svn/include/agg_rasterizer_cells_aa.h

Lines 181 to 196 in 752c376

    
           template<class Cell> 
        
           AGG_INLINE void rasterizer_cells_aa<Cell>::add_curr_cell() 
        
           { 
        
               if(m_curr_cell.area | m_curr_cell.cover) 
        
               { 
        
                   if((m_num_cells & cell_block_mask) == 0) 
        
                   { 
        
                       if(m_num_blocks >= m_cell_block_limit) { 
        
                           throw std::overflow_error("Exceeded cell block limit"); 
        
                       } 
        
                       allocate_block(); 
        
                   } 
        
                   *m_curr_cell_ptr++ = m_curr_cell; 
        
                   ++m_num_cells; 
        
               } 
        
           }

We are already upping the limit from the default value by a factor of 8 which was last doubled in #4408 . That we have this knob at all seems to be a local patch we are carrying (via 1b937a4 in #3777 ).

I have a patch to make the error messages more actionable and am becoming convinced than a 8-16x bump on the max is in order.

One more comment while testing this, if we make the size a bit bigger we can get a 100k pathological data set to draw at 400dpi, but by setting the chunk_size to 10k we get a ~5k speed up in draw time.

Process Python finished
Python 3.10.0a4+ (heads/main:d16f6176ab, Jan  9 2021, 17:51:02) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.0.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: import io

In [3]: import numpy as np

In [4]: from matplotlib.figure import Figure

In [5]: buff = io.BytesIO()

In [6]: fig = Figure()

In [7]: ax = fig.subplots()

In [8]: points = np.ones(100_000)

In [9]: points[::2] *= -1

In [10]: ax.plot(points)
Out[10]: [<matplotlib.lines.Line2D at 0x7f9304728a00>]

In [11]: fig.savefig(buff, format='png', dpi=400)

In [12]: fig.savefig(buff, format='png', dpi=400)

In [13]: %timeit fig.savefig(buff, format='png', dpi=400)
7.27 s ± 17.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [14]: from matplotlib import (
     ...     collections, path, pyplot as plt, transforms as mtransforms, rcParams)

In [15]: rcParams['agg.path.chunksize'] = 10_00

In [16]: %timeit fig.savefig(buff, format='png', dpi=400)
1.28 s ± 11.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [17]:

So I think the main trade off on upping the chunk limit is less memory usage, and more run time. With a good error message we can hopefully direct the users who run into this to the chunk size with a sense of what "good" values are and save them run time. Where as if we push the limit higher and higher we are going to get bug reports of "Matplotlib is slow" which is harder to identify and point people to solutions.

Another option could be to set the default chunk size to be like 10k or 50k. That would, with an upped limit, be enough to have saved the OP out of the box and should be a no-op for a vast majority of users (who I assume are mostly plotting things shorter than 50k).

Is there a way to check how much memory each case is using? The 8192 number seemed ridiculously small, but I don't know how that corresponded to a memory.

To quote myself from 2014 #3777 (comment)

I am happy enough with this and I suspect getting any happier would involve spending a week doing nothing but reading the Agg code (and get worse before it got better) so 👍 from me.

I'm not fully sure what "cells" in the context are. But observational there is some super-linear scaling in the rasterization time with the number of cells (I suspect quadratic as I think each cell requires you to scan all of the other cells based on vague memories of conversations with Mike a few years ago). I am starting to think that the original limit was more about run time rather than memory usage (which is probably where we should have started given that the chunksize in documented under a "performance" heading).

ENH: make error messages from exceeding agg cell block count clearer

fig.savefig gives OverflowError for high DPI's · Issue #19325 · matplotlib/matplotlib · GitHub

Repository navigation

fig.savefig gives OverflowError for high DPI's #19325

Bug report

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	template<class Cell>
	AGG_INLINE void rasterizer_cells_aa<Cell>::add_curr_cell()
	{
	if(m_curr_cell.area \| m_curr_cell.cover)
	{
	if((m_num_cells & cell_block_mask) == 0)
	{
	if(m_num_blocks >= m_cell_block_limit) {
	throw std::overflow_error("Exceeded cell block limit");
	}
	allocate_block();
	}
	*m_curr_cell_ptr++ = m_curr_cell;
	++m_num_cells;
	}
	}

fig.savefig gives OverflowError for high DPI's · Issue #19325 · matplotlib/matplotlib · GitHub

Repository navigation

fig.savefig gives OverflowError for high DPI's #19325

Description

Bug report

Activity

jklymak commented on Jan 20, 2021

anntzer commented on Jan 20, 2021

jklymak commented on Jan 20, 2021

anntzer commented on Jan 20, 2021

jklymak commented on Jan 20, 2021

anntzer commented on Jan 21, 2021

tacaswell commented on Jan 21, 2021

jklymak commented on Jan 21, 2021

tacaswell commented on Jan 22, 2021

tacaswell commented on Jan 22, 2021

jklymak commented on Jan 22, 2021

tacaswell commented on Jan 23, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions