Repository navigation
Axes instantiation is very slow #6664
Description
> python -V
Python 2.7.11
> pip freeze | grep matplotlib
matplotlib==1.5.1
> python -m platform -s "print platform()"
Windows-7-6.1.7601-SP2I have discovered that my application spends a lot of time in Axes creation. Results of class instantiation profiling is below:
> python -m timeit -s "from matplotlib.figure import Figure; from matplotlib.axes import Axes; fig = Figure()" "Axes(fig, (0, 0, 1 ,1))"
10 loops, best of 3: 33.2 msec per loop
Inside Axes.__init__ most of the execution time takes self.cla() call and about the half of it is
this loop below
for name, spine in six.iteritems(self.spines):
spine.cla()After commenting this two lines I have got this:
> python -m timeit -s "from matplotlib.figure import Figure; from matplotlib.axes import Axes; fig = Figure()" "Axes(fig, (0, 0, 1 ,1))"
100 loops, best of 3: 17.9 msec per loopSo we have 33 ms per axes creation, and the half of this time it is used for spines creation that actually may be not used (in my case I have bunch of sharex axes and only one of them have spines).
I see a possible solution where creation of spines, scales (you can see creation of logarithmic scale, which I do not use too) and other stuff is delayed to the point when it actually needed/initialized.
And the fun part. For 20 created Axes we have 60115 calls to Rectangle.stale (3005 call per Axes instantiation)
Activity
tacaswell commented on Jun 28, 2016
Any help improving the performance would be greatly appreciated!
It is a bit odd that the stale that gets clobbered is the one on rectangle. Can you tell which rectangle artists that is on?
In the orginal implementation of stale the the propagation was short circuited, but that lead to the figure auto-refresh getting stuck.
Kojoley commented on Jun 28, 2016
Here is the test and profiler diagram
efiring commented on Jun 28, 2016
How did you generate that profiler diagram?
Kojoley commented on Jun 28, 2016
It is generated in PyCharm IDE which uses yFiles and yappi
Kojoley commented on Jun 28, 2016
The bottleneck is reset_ticks() call in the Axis.cla() call
After commenting it out I have got only 2 ms instead of 33 ms per Axes instantiation
100 loops, best of 3: 2.41 msec per loopCall graph becomes much simpler too
anntzer commented on Jun 28, 2016
I think stale is a red herring, mostly there because of the overhead of using an instrumenting profiler (?). On the other hand reset_ticks is indeed a bottleneck, see http://vmprof.com/#/207c1fb01e6cedab167a82487d425da1 for the vmprof (statistical profiler) analysis of
from matplotlib.figure import Figure; from matplotlib.axes import Axes for _ in range(1000): fig = Figure() Axes(fig, (0, 0, 1 ,1))(matplotlib 2.0b1)
WeatherGod commented on Jun 30, 2016
Wow... that is a very informative graph. There seems to be several
degeneracies going on here. First and foremost, I think it is obvious that
we are calling cla() a bit indiscriminately. This may partly be due to
multiple entry points that may require calls to cla(), but we probably
could cut back on it a bit. Second, each call to cla() does a lot of work
that may not be necessary. We might want to have a sentinel that indicates
whether the axis is still in a cleared state, thereby some calls to cla()
would short-circuit.
On Tue, Jun 28, 2016 at 6:45 PM, Antony Lee notifications@github.com
wrote:
efiring commented on Jun 30, 2016
Every possible grid and tick line is being instantiated; in practice, most of them don't get used. Lazy instantiation would solve that, at the cost of a little more complexity.
Kojoley commented on Jun 30, 2016
On newly created axis we do not even have ticks, but here we have creation of single tick for the internal purposes. This should not be problem, but it is, because for single Axes we have multiple reset_ticks calls (in Axes.__init__, XAxis.cla, YAxis.cla). It is definitely some kind of recursion.
I have tried to figure out if it is possible to simplify reset_tick method by removing that tick creation (and entirely remove _lastNumMajorTicks and _lastNumMinorTicks variables). I have not found things that will prevent such optimization, result of this refactoring will be about 11 times performance boost to Axes instantiation (my application runs 30% faster after that) and cleaner code too.
But even after that optimization multiple reset_ticks will not gone. Yes them will be much less resource heavy than now, but it is still not good.
As this optimization will affect multiple places and include rewriting of get_ticks_position, get_minor_ticks, get_major_ticks, set_tick_params methods, I'm going to setup a test environment to develop a PR.
larsoner commented on Apr 27, 2017
It seems like using such changes to get a performance boost is still worth it, even if it's not the best solution. After all, these changes were proposed about 10 months ago, so that's 10 months where things could have been faster...
@efiring would you be okay with such a workaround in the meantime until lazy instantiation can be implemented?
efiring commented on Apr 28, 2017
@Kojoley, any more thoughts on this? Or did you run into a roadblock?
Override GeoAxes Spine and Axis creation.
QuLogic commented on May 25, 2017
Somehow, Ticks are inexplicably tied with Spines. If I apply this patch to defer Tick creation till they're used:
then the first test that fails looks like this:
which is really odd, as I'd have expected the ticks to be the ones that break.
11 remaining items
QuLogic commented on Feb 5, 2018
So #8626 was applied and then reverted due to some issues with polar stuff. For 1-, 2-, and 10-square subplots, timings are 118.42ms, 374.95ms, and 3.91s, respectively. In the meantime, #10302 was merged, which drops time to 79.00ms, 284.09ms, and 2.90s, respectively. You can see that change in the benchmarks. master also includes #10327, which nets 77.55ms, 279.91ms, and 2.84s.
If we re-apply #8626, we could perhaps save another 2-4% at 77.03ms, 268.03ms, and 2.77s. But unfortunately, it's still breaking polar plots somehow.
TomNicholas commented on Jan 16, 2019
What is the status with this? I'm using matplotlib in a Jupyter notebook and it seems that plt.subplots() is taking longer than processing GBs of data is! Is there another reason why jupyter in particular would be very slow?
timhoffm commented on Jan 16, 2019
We have a speedup of a factor 4 by lazily instantiating ticks ( #9727 / #10302 ). However, the essential problems still persist. One is the tick handling, which handles each tick as a separate object. The second is the layered architecture and initialization / update mechanism. Both are hard to change.
efiring commented on Jan 16, 2019
There is certainly more work to do on this, but it is usually not a noticeable when working interactively unless you are making a plot with a large number of axes. Is this your use case, @TomNicholas? If not, I wonder whether you might be running into some other problem.
rth commented on Jan 22, 2019
Another place where this might have a significant impact is animations / interactive plot updates. Blitting is a workaround, but a user perspective it's sometimes easier to just redraw everything in which case axis instantiation time will be non-negligible.
aasensio commented on Feb 12, 2019
I'm finding the same slow behavior when doing a bunch of imshow in a figure in jupyter and I don't know if the issue belongs here or to jupyter. Something like this simple takes almost 18 s in my computer. The larger the image, the longer it takes:
f, ax = plt.subplots(nrows=3, ncols=3, figsize=(13,8)) for i in range(3): for j in range(3): ax[i,j].imshow(np.random.randn(100,100))Kojoley commented on Feb 12, 2019
The original issue is about axes creation, redrawing is a different problem. There is RGBA<->BGRA conversion which takes a considerable amount of time if you make hundreds redraws per second. (I have a patch to make it faster but it actually three patches for every of GCC, Clang, and MSVC compilers because their optimizers and vectorizers are too far from being good).
I tried your code and it takes 600ms for me.
The same comment as for the case above applies here. In other words the original issue was about plt.subplots call of your example. Axes instantiation time is not dependent on plot size, so you experience some other issue.
timhoffm commented on Feb 13, 2019
@aasensio which matplotlib version? Also please time with
%%timeit -n1 -r1 f, ax = plt.subplots(nrows=3, ncols=3, figsize=(13,8)) for i in range(3): for j in range(3): ax[i,j].imshow(np.random.randn(100,100))and
%%timeit -n1 -r1 f, ax = plt.subplots(nrows=3, ncols=3, figsize=(13,8)) for i in range(3): for j in range(3): ax[i,j].imshow(np.random.randn(100,100))to see the which fraction of the time is covered by the subplot creation.
aasensio commented on Feb 14, 2019
@timhoffm My version of matplotlib is 3.0.2, running with notebook server 5.7.4. The timing results are 684 ms for the subplots creation only and 12.7 s when the imshows are added. I'm starting to suspect that it is somehow related to the server-client I/O in my system, which might be firewalled. I'll investigate on that.
dstansby commented on Dec 30, 2021
I think the original issue is solved now - at least for me the performance bottlnecks mentioned at the top are not present in the currently main branch. I'm going to close this to avoid it becoming a general posting of performance issues, and recommend others open new and more specific issues if they notice any performance problems.
Kojoley commented on Dec 30, 2021
I would not say it is solved, but the situation is indeed improved, from 'very slow' it became just 'slow'. The #6664 (comment) implied that there could be 13-14x improvement, the #10302 gave 3-4x, so there still should be some bottlenecks. Also there were performance regressions since 2.2.5 totaling in 1.6x slowdown:
jklymak commented on Dec 30, 2021
@Kojoley i think the point here is that it's pretty hard to track this in a really old issue. For sure, if we think there are current bottlenecks that could be fixed we should fix them.