Animation of contourf becomes extremely slow · Issue #6985 · matplotlib/matplotlib · GitHub

When creating an animation with FuncAnimation and contourf I have run into an issue where the more frames I have the slower things become. This is illustrated in the code below. I print a progress bar, which displays both progress and the time required for the current step. As the animation creation progresses you will see that each step takes longer and longer. I was able to mitigate this by creating a derived class MyFuncAnimation, where I overloaded the function save. In the new version of save I go through and clear all collections associated with self._drawn_artists, after writer.grab_frame, in the following way:
for collection in self._drawn_artists.collections: gca().collections.remove(collection)

I am not sure if there is a better way to do this, but it was the only way I could figure out how to address the issue.

Matplot Version = 1.5.1
Python Version = 3.5.2
Platform = MAC OS 10.11.6

Installed matplotlib through pip

Example:

#!/usr/bin/env python3

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import mlab, cm
from matplotlib.animation import FuncAnimation
import sys
import time

barWidth = 40
n_steps = 100

class ProgressBar():
    '''
    Provides a simple progress bar class
    '''
    def __init__(self, nsteps, width=barWidth):
        self._start = self._stop = time.time()
        self._nsteps = nsteps
        self._width = width
        self._status = ""

    def update(self, step):
        '''
        This function produces and updates a progress bar.
        It was stolen and modified from
        http://stackoverflow.com/a/15860757/1552338
        '''
        self._start = self._stop
        self._stop = time.time()
        self._status = self._stop - self._start

        progress = float(step)/float(self._nsteps - 1)
        if progress >= 1:
            progress = 1
            self._status = "Complete...\r\n"
        block = int(round(self._width * progress))
        text = "\rProcessing: [{}] {:.1%} {:.3}".format("#" * block
                                                        + "-" * (self._width
                                                                 - block),
                                                        progress,
                                                        self._status)
        sys.stdout.write(text)
        sys.stdout.flush()

# Create x, y data
delta = 0.5

extent = (-3, 4, -4, 3)

x = np.arange(-3.0, 4.001, delta)
y = np.arange(-4.0, 3.001, delta)
X, Y = np.meshgrid(x, y)

# Boost the upper limit to avoid truncation errors.
levels = np.arange(-2.0, 1.601, 0.4)

fig, ax = plt.subplots(1, 1)  # create our figure and axes
progress = ProgressBar(n_steps)  # initialize the progress bar


def update(frame_number):
    '''
    This is where our contour is creating
    '''
    sigma = np.random.uniform(0, 1, 4)
    Z1 = mlab.bivariate_normal(X, Y, sigma[0], sigma[1], 0.0, 0.0)
    Z2 = mlab.bivariate_normal(X, Y, sigma[2], sigma[3], 1, 1)
    Z = (Z1 - Z2) * 10

    norm = cm.colors.Normalize(vmax=abs(Z).max(), vmin=-abs(Z).max())
    cmap = cm.PRGn

    contf = plt.contourf(X, Y, Z, levels,
                         cmap=cm.get_cmap(cmap, len(levels) - 1),
                         norm=norm)

    progress.update(frame_number)

    return contf

# Construct the animation, using the update function as the animation
# director.
anim = FuncAnimation(fig, update, interval=n_steps)
anim.save("AnimContourf.mp4")
plt.close(fig)
sys.stdout.write("\n")

The reason it is taking progressively longer is because the previous
contourf's aren't getting cleared out. So, each "save()" call is rendering
more and more contourf's that aren't actually visible because the newer
ones are on top.

On Fri, Aug 26, 2016 at 12:49 PM, Erich L Foster notifications@github.com
wrote:

When creating an animation with FuncAnimation and contourf I have run into
an issue where the more frames I have the slower things become. This is
illustrated in the code below. I print a progress bar, which displays both
progress and the time required for the current step. As the animation
creation progresses you will see that each step takes longer and longer. I
was able to mitigate this by creating a derived class MyFuncAnimation,
where I overloaded the function save. In the new version of save I go
through and clear all collections associated with self._drawn_artists,
after writer.grab_frame, in the following way:
for collection in self._drawn_artists.collections:
gca().collections.remove(collection)

I am not sure if there is a better way to do this, but it was the only way
I could figure out how to address the issue.

Matplot Version = 1.5.1
Python Version = 3.5.2
Platform = MAC OS 10.11.6

Installed matplotlib through pip

Example:

#!/usr/bin/env python3

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import mlab, cm
from matplotlib.animation import FuncAnimation
import sys
import time

barWidth = 40
n_steps = 100

class ProgressBar():
'''
Provides a simple progress bar class
'''
def init(self, nsteps, width=barWidth):
self._start = self._stop = time.time()
self._nsteps = nsteps
self._width = width
self._status = ""
def update(self, step):
    '''
    This function produces and updates a progress bar.
    It was stolen and modified from
    http://stackoverflow.com/a/15860757/1552338
    '''
    self._start = self._stop
    self._stop = time.time()
    self._status = self._stop - self._start

    progress = float(step)/float(self._nsteps - 1)
    if progress >= 1:
        progress = 1
        self._status = "Complete...\r\n"
    block = int(round(self._width * progress))
    text = "\rProcessing: [{}] {:.1%} {:.3}".format("#" * block
                                                    + "-" * (self._width
                                                             - block),
                                                    progress,
                                                    self._status)
    sys.stdout.write(text)
    sys.stdout.flush()
Create x, y data

delta = 0.5

extent = (-3, 4, -4, 3)

x = np.arange(-3.0, 4.001, delta)
y = np.arange(-4.0, 3.001, delta)
X, Y = np.meshgrid(x, y)

Boost the upper limit to avoid truncation errors.

levels = np.arange(-2.0, 1.601, 0.4)

fig, ax = plt.subplots(1, 1) # create our figure and axes
progress = ProgressBar(n_steps) # initialize the progress bar

def update(frame_number):
'''
This is where our contour is creating
'''
sigma = np.random.uniform(0, 1, 4)
Z1 = mlab.bivariate_normal(X, Y, sigma[0], sigma[1], 0.0, 0.0)
Z2 = mlab.bivariate_normal(X, Y, sigma[2], sigma[3], 1, 1)
Z = (Z1 - Z2) * 10
norm = cm.colors.Normalize(vmax=abs(Z).max(), vmin=-abs(Z).max())
cmap = cm.PRGn

contf = plt.contourf(X, Y, Z, levels,
                     cmap=cm.get_cmap(cmap, len(levels) - 1),
                     norm=norm)

progress.update(frame_number)

return contf
Construct the animation, using the update function as the animation

director.

anim = FuncAnimation(fig, update, interval=n_steps)
anim.save("AnimContourf.mp4")
plt.close(fig)
sys.stdout.write("\n")

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#6985, or mute the thread
https://github.com/notifications/unsubscribe-auth/AARy-OwemAlgHkdMVMVOHQ4rGtw8de-Rks5qjxkMgaJpZM4JuQxC
.

This is what I understood, but what is the proper method to do this?

oh, and I think there is a tiny memory leak that is fixed in master with
respect to contourf(), but it wouldn't be the main reason for the slowdown.

On Fri, Aug 26, 2016 at 12:53 PM, Benjamin Root ben.v.root@gmail.com
wrote:

The reason it is taking progressively longer is because the previous
contourf's aren't getting cleared out. So, each "save()" call is rendering
more and more contourf's that aren't actually visible because the newer
ones are on top.

On Fri, Aug 26, 2016 at 12:49 PM, Erich L Foster <notifications@github.com
wrote:

When creating an animation with FuncAnimation and contourf I have run
into an issue where the more frames I have the slower things become. This
is illustrated in the code below. I print a progress bar, which displays
both progress and the time required for the current step. As the animation
creation progresses you will see that each step takes longer and longer. I
was able to mitigate this by creating a derived class MyFuncAnimation,
where I overloaded the function save. In the new version of save I go
through and clear all collections associated with self._drawn_artists,
after writer.grab_frame, in the following way:
for collection in self._drawn_artists.collections:
gca().collections.remove(collection)

I am not sure if there is a better way to do this, but it was the only
way I could figure out how to address the issue.

Matplot Version = 1.5.1
Python Version = 3.5.2
Platform = MAC OS 10.11.6

Installed matplotlib through pip

Example:

#!/usr/bin/env python3

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import mlab, cm
from matplotlib.animation import FuncAnimation
import sys
import time

barWidth = 40
n_steps = 100

class ProgressBar():
'''
Provides a simple progress bar class
'''
def init(self, nsteps, width=barWidth):
self._start = self._stop = time.time()
self._nsteps = nsteps
self._width = width
self._status = ""
def update(self, step):
    '''
    This function produces and updates a progress bar.
    It was stolen and modified from
    http://stackoverflow.com/a/15860757/1552338
    '''
    self._start = self._stop
    self._stop = time.time()
    self._status = self._stop - self._start

    progress = float(step)/float(self._nsteps - 1)
    if progress >= 1:
        progress = 1
        self._status = "Complete...\r\n"
    block = int(round(self._width * progress))
    text = "\rProcessing: [{}] {:.1%} {:.3}".format("#" * block
                                                    + "-" * (self._width
                                                             - block),
                                                    progress,
                                                    self._status)
    sys.stdout.write(text)
    sys.stdout.flush()
Create x, y data

delta = 0.5

extent = (-3, 4, -4, 3)

x = np.arange(-3.0, 4.001, delta)
y = np.arange(-4.0, 3.001, delta)
X, Y = np.meshgrid(x, y)

Boost the upper limit to avoid truncation errors.

levels = np.arange(-2.0, 1.601, 0.4)

fig, ax = plt.subplots(1, 1) # create our figure and axes
progress = ProgressBar(n_steps) # initialize the progress bar

def update(frame_number):
'''
This is where our contour is creating
'''
sigma = np.random.uniform(0, 1, 4)
Z1 = mlab.bivariate_normal(X, Y, sigma[0], sigma[1], 0.0, 0.0)
Z2 = mlab.bivariate_normal(X, Y, sigma[2], sigma[3], 1, 1)
Z = (Z1 - Z2) * 10
norm = cm.colors.Normalize(vmax=abs(Z).max(), vmin=-abs(Z).max())
cmap = cm.PRGn

contf = plt.contourf(X, Y, Z, levels,
                     cmap=cm.get_cmap(cmap, len(levels) - 1),
                     norm=norm)

progress.update(frame_number)

return contf
Construct the animation, using the update function as the animation

director.

anim = FuncAnimation(fig, update, interval=n_steps)
anim.save("AnimContourf.mp4")
plt.close(fig)
sys.stdout.write("\n")

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#6985, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AARy-OwemAlgHkdMVMVOHQ4rGtw8de-Rks5qjxkMgaJpZM4JuQxC
.

Set blit=True and return the list of artists to clear, but unfortunately, that doesn't work with contourf due to #6139.

The proper way is not to do blitting. The QuadContourSet object doesn't
have the ability to reuse itself in the same way that an image could. You
just need to remove the artist before computing the next contourf(). All
artists should come with a .remove() method, so you'd just need to store
that artist somewhere and do a conditional remove() on it before calling
contourf() and replacing the previous object with the new contourf object.

This usually means having a global variable of some sort to store the
object between frames, initialized to None for the first frame (which is
why you would need to conditionally call .remove()).

Does that help?

On Fri, Aug 26, 2016 at 5:39 PM, Elliott Sales de Andrade <
notifications@github.com> wrote:

Set blit=True and return the list of artists to clear, but unfortunately,
that doesn't work with contourf due to #6139
#6139.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#6985 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARy-C0pk-haLauALF6QcS_IFEg1IdBTks5qj10JgaJpZM4JuQxC
.

Seems like this would be handled better by the Animation class, but I can certainly do it the other way too.

It's simpler to pass the artists to the update function via fargs and update the existing artists, though:

def init():
    '''
    This is where our contour is created
    '''
    sigma = np.random.uniform(0, 1, 4)
    Z1 = mlab.bivariate_normal(X, Y, sigma[0], sigma[1], 0.0, 0.0)
    Z2 = mlab.bivariate_normal(X, Y, sigma[2], sigma[3], 1, 1)
    Z = (Z1 - Z2) * 10

    norm = cm.colors.Normalize(vmax=abs(Z).max(), vmin=-abs(Z).max())
    cmap = cm.PRGn

    contf = plt.contourf(X, Y, Z, levels,
                         cmap=cm.get_cmap(cmap, len(levels) - 1),
                         norm=norm)

    return contf


def update(frame_number, contf):
    '''
    This is where our contour is updated
    '''
    sigma = np.random.uniform(0, 1, 4)
    Z1 = mlab.bivariate_normal(X, Y, sigma[0], sigma[1], 0.0, 0.0)
    Z2 = mlab.bivariate_normal(X, Y, sigma[2], sigma[3], 1, 1)
    Z = (Z1 - Z2) * 10

    norm = cm.colors.Normalize(vmax=abs(Z).max(), vmin=-abs(Z).max())
    cmap = cm.PRGn

    contf.set_array(Z)
    contf.set_cmap(cm.get_cmap(cmap, len(levels) - 1))
    contf.set_norm(norm)

    progress.update(frame_number)

# Construct the animation, using the update function as the animation
# director.
contf = init()
anim = FuncAnimation(fig, update, interval=n_steps, fargs=(contf, ))
anim.save("AnimContourf.mp4")
plt.close(fig)
sys.stdout.write("\n")

though I'm not sure changing the cmap like that is the best idea. Depending on what you're doing, the init bit could be just adding a nearly empty contourf instead of the full stuff.

Sorry, I may have been thrown by the randomness; alas, I think it doesn't update the contour. I guess the longer method is the only option until contourf returns a more normal artist.

Correct. Just updating the Z data isn't sufficient. It still has to
calculate the polygons and finalize them. Unfortunately, we never really
cleaned up that code too well, so the only place that handles everything
correctly is in contourf().

On Fri, Aug 26, 2016 at 6:12 PM, Elliott Sales de Andrade <
notifications@github.com> wrote:

Sorry, I may have been thrown by the randomness; alas, I think it doesn't
update the contour. I guess the longer method is the only option until
contourf returns a more normal artist.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#6985 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARy-INCEOqMSSFzC_50oX7RcUg_SDdQks5qj2TkgaJpZM4JuQxC
.

So how would the solution given by @QuLogic work for a contourf which uses tri=True? In this case the z data is not in matrix form. The example I gave isn't my real example, but my real example is too complicated and I use a netcdf file to fill in data.

It wouldn't work. That's what his apology was about because the update data stuff doesn't update the entire object, and as you point out, the other contour objects don't work the same way.

You would have to follow what I originally suggested, which is to keep track of the objects via a global variable (or some other specialized class that is passed in through fargs that manages the lifecycle of the contourf object). Some psuedo-code:

def update(num, ...):
    if update.theobj is not None:
        update.theobj.remove()
    update.theobj = plt.contourf(data, ....)
    return (update.theobj,)
update.theobj = None

Something to that effect.
By the way, I think you are supposed to return an iterable of objects from the update function, not a single object.

With contour, just clear the axes and remake the contour plus whatever else needs to go there. I don't see any point in trying to track and remove objects associated with the contours.

@WeatherGod When I do
return (contf,)
I get AttributeError: 'tuple' object has no attribute 'autoscale_None'
Maybe I am not doing this correctly.

@efiring: In my code removing collections is about 8% faster than ax.clear(). Also, ax.clear forces me to redraw continents in basemap. In fact, the 8% slower was without even redrawing the continents.

Hmm, that's strange. If you are not blitting at all, you can get away with
not returning anything at all. But that error definitely shouldn't happen,
I don't think.

On Mon, Aug 29, 2016 at 1:47 PM, Erich L Foster notifications@github.com
wrote:

@WeatherGod https://github.com/WeatherGod When I do
return (contf,)
I get AttributeError: 'tuple' object has no attribute 'autoscale_None'
Maybe I am not doing this correctly.

@efiring https://github.com/efiring: In my code removing collections is
about 8% faster than ax.clear().

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#6985 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARy-JMkjzpnTRvEvjatRaY3PlzZjocGks5qkxsmgaJpZM4JuQxC
.

So what is different between self._drawn_artists.collections and ax.collections? If I go through and remove collections from ax this will of course remove my basemap continents and I lose about 10% in time, but using this method on self._drawn_artists.collections doesn't remove the continents. Right now, for efficiency, using my derived class with overloaded save is definitely the fastest.

If you want to remove only the collections added by contourf, can't you just cycle through the list you saved in your contf variable?

I tried (before):

if contf is not None:
    for collection in contf.collections:
        gca().collections.remove(collection)

But,

File "./plotCDAT.py", line 149, in animate
gca().collections.remove(collection)
ValueError: list.remove(x): x not in list

I understand what the error is saying, but I don't understand why collection wouldn't be in gca().collections.

No, the artist objects themselves have a remove() method. I wasn't
referring to the list's remove() method. You want the artist's remove()
method because the artist knows everything it needs to do to clean itself
up.

On Mon, Aug 29, 2016 at 4:15 PM, Erich L Foster notifications@github.com
wrote:

I tried (before):

if contf is not None:
for collection in contf.collections:
gca().collections.remove(collection)

But,

File "./plotCDAT.py", line 149, in animate
gca().collections.remove(collection)
ValueError: list.remove(x): x not in list

I understand what the error is saying, but I don't understand why
collection wouldn't be in gca().collections.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#6985 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARy-GpIOeZd-iHUMqKEu1FJSO8u3a_dks5qkz3LgaJpZM4JuQxC
.

Maybe I am still confused, but using

for collection in contf.collections:
    collection.remove

Causes things to be as slow as if I hadn't removed the collections at all, i.e. 10x slower. So, for some reason removing the collection from gca().collections and then adding back in the coastlines is actually much faster.

You need collection.remove() with the parentheses.

@efiring Then it says

ValueError: list.remove(x): x not in list

So clearly, I must be accessing the wrong object.

Does it really do that if you execute

for collection in contf.collections:
    collection.remove()

That seems very strange?

Does the following work for you?

Python 2.7.12 (default, Jul 19 2016, 14:17:21) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> z = np.random.random((10, 10))
>>> foo = plt.contourf(z)
>>> foo
<matplotlib.contour.QuadContourSet object at 0x7fc97aff4210>
>>> foo.collections
<a list of 7 mcoll.PathCollection objects>
>>> for c in foo.collections:
...     c.remove()
... 
>>> plt.show()

With that, you should get a blank plot with limits from 0 to 9.

As a note to self, perhaps we should augment ContourSet so that it has its own remove() method so that it can clear out not just the PatchCollections, but also any contour labels and such if they exist.

After some debugging I discovered that a collection was being removed more than once. However, doing the contf.collection method results again in an extremely slow code. Anyway, I am just going to use the method of overriding save.

I hope you guys consider blitting for contours.

The time consuming step (as I understand it) with contours is actually computing the contours.

Blitting only helps if you only need to re-draw a single artist and the slowest part of the process is redrawing everything else in the figure.

Blitting is also only used when displaying to the screen, anim.save("AnimContourf.mp4") always fully-rerenders the figure.

Thanks for the discussion above! Is there a way to deal with this if you're using ArtistAnimation and not FuncAnimation? If I remove artists after every iteration of my algorithm, there's nothing to plot at the end, e.g. you get:

    if artist.axes.figure not in figs:
AttributeError: 'NoneType' object has no attribute 'figure'

But otherwise it becomes painfully slow (~hours).

I don't see how ArtistAnimation helps anything; it just puts the creation of your contours for each frame into an initial step to create the list of lists of artists.

There is simply no way to get around the time taken for the contouring calculation itself.

A penalty of 8% for a simpler algorithm is not worth spending a lot of programming time on.

It sounds like the real problem is that you don't want to recreate the Basemap from scratch each time. The solution is to initialize the Basemap object (which includes the time-consuming step of extracting the coastlines), save it, and reuse it as the starting point for each frame. Unfortunately I don't have a simple ready-to-go example to point to, because all of my code along these lines predates the animation module (or at least my use of it), involves working with remote data sources, and uses my Basemap wrapper code.

In the interest of reducing the number of open issues we have to keep scanning and checking, I am going to close this now.

Animation of contourf becomes extremely slow · Issue #6985 · matplotlib/matplotlib · GitHub

Repository navigation

Animation of contourf becomes extremely slow #6985

Create x, y data

Boost the upper limit to avoid truncation errors.

Construct the animation, using the update function as the animation

director.

Create x, y data

Boost the upper limit to avoid truncation errors.

Construct the animation, using the update function as the animation

director.

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Animation of contourf becomes extremely slow · Issue #6985 · matplotlib/matplotlib · GitHub

Repository navigation

Animation of contourf becomes extremely slow #6985

Description

Activity

WeatherGod commented on Aug 26, 2016

Create x, y data

Boost the upper limit to avoid truncation errors.

Construct the animation, using the update function as the animation

director.

erichlf commented on Aug 26, 2016

WeatherGod commented on Aug 26, 2016

Create x, y data

Boost the upper limit to avoid truncation errors.

Construct the animation, using the update function as the animation

director.

QuLogic commented on Aug 26, 2016

WeatherGod commented on Aug 26, 2016

erichlf commented on Aug 26, 2016

QuLogic commented on Aug 26, 2016

QuLogic commented on Aug 26, 2016

WeatherGod commented on Aug 27, 2016

erichlf commented on Aug 29, 2016

WeatherGod commented on Aug 29, 2016

efiring commented on Aug 29, 2016

erichlf commented on Aug 29, 2016

WeatherGod commented on Aug 29, 2016

erichlf commented on Aug 29, 2016

efiring commented on Aug 29, 2016

erichlf commented on Aug 29, 2016

WeatherGod commented on Aug 29, 2016

erichlf commented on Aug 29, 2016

efiring commented on Aug 29, 2016

erichlf commented on Aug 30, 2016

jenshnielsen commented on Aug 30, 2016

WeatherGod commented on Aug 30, 2016

erichlf commented on Aug 30, 2016

tacaswell commented on Aug 30, 2016

jfainberg commented on Nov 25, 2016

efiring commented on Nov 25, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions