Repository navigation
[Bug]: Memory leak in 3.4.3 #21259
Description
Bug summary
As reported in jupyterlab/jupyterlab#11182, running below code with a 30M row data frame (pandas) only increases memory.
Code for reproduction
%matplotlib widget # with this commented out or not, same result
import modin.pandas as pd
import matplotlib.pyplot as plt
# get your df
plt.rcParams["figure.figsize"] = [40, 15]
plt.rcParams["figure.autolayout"] = True
# plt.cla() # with this commented out or not, same result
# plt.clf() # with this commented out or not, same result
# plt.close(fig) # with this commented out or not, same result
# this works but then of course the figure is empty
# import gc
# fig.clear() # also this must be commented out on the second run since on the first one it's not defined
# gc.collect()
fig, axs = plt.subplots(1, 1, sharex=True)
# fig.clear() # draws only one plot, still leaks
axs.plot(df[df.columns[0]], color="black")
ax1 = axs.twinx()
ax1.plot(df[df.columns[1]], color="blue")
# with this 2 lines commented out or not, same result
fig.tight_layout()
plt.show()Actual outcome
Memory only increases.
Expected outcome
No memory leak prefferably.
Operating system
Ubuntu 21.04
Matplotlib Version
3.4.3
Matplotlib Backend
Qt5Agg, same with %matplotlib and %matplotlib widget
Python version
3.7.10
Jupyter version
3.1.11
Other libraries
Conda's env.yaml:
name: test
channels:
- pyviz
- conda-forge
- defaults
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=1_gnu
- abseil-cpp=20210324.2=h9c3ff4c_0
- alembic=1.7.3=pyhd8ed1ab_0
- alsa-lib=1.2.3=h516909a_0
- anyio=3.3.0=py37h89c1867_0
- argcomplete=1.12.3=pyhd8ed1ab_2
- argon2-cffi=20.1.0=py37h5e8e339_2
- arrow-cpp=5.0.0=py37hdf48254_5_cpu
- async_generator=1.10=py_0
- attrs=21.2.0=pyhd8ed1ab_0
- autopage=0.4.0=pyhd8ed1ab_0
- aws-c-cal=0.5.11=h95a6274_0
- aws-c-common=0.6.2=h7f98852_0
- aws-c-event-stream=0.2.7=h3541f99_13
- aws-c-io=0.10.5=hfb6a706_0
- aws-checksums=0.1.11=ha31a3da_7
- aws-sdk-cpp=1.8.186=hb4091e7_3
- babel=2.9.1=pyh44b312d_0
- backcall=0.2.0=pyh9f0ad1d_0
- backports=1.0=py_2
- backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
- backports.zoneinfo=0.2.1=py37h5e8e339_4
- bleach=4.1.0=pyhd8ed1ab_0
- bokeh=2.3.3=py37h89c1867_0
- brotlipy=0.7.0=py37h5e8e339_1001
- bzip2=1.0.8=h7f98852_4
- c-ares=1.17.2=h7f98852_0
- ca-certificates=2021.5.30=ha878542_0
- certifi=2021.5.30=py37h89c1867_0
- cffi=1.14.6=py37hc58025e_0
- chardet=4.0.0=py37h89c1867_1
- charset-normalizer=2.0.0=pyhd8ed1ab_0
- click=8.0.1=py37h89c1867_0
- clickhouse-cityhash=1.0.2.3=py37h3340039_2
- clickhouse-driver=0.2.1=py37h5e8e339_0
- cliff=3.9.0=pyhd8ed1ab_0
- cloudpickle=2.0.0=pyhd8ed1ab_0
- cmaes=0.8.2=pyh44b312d_0
- cmd2=2.2.0=py37h89c1867_0
- colorama=0.4.4=pyh9f0ad1d_0
- colorcet=2.0.6=pyhd8ed1ab_0
- colorlog=6.4.1=py37h89c1867_0
- conda=4.10.3=py37h89c1867_1
- conda-package-handling=1.7.3=py37h5e8e339_0
- cramjam=2.3.1=py37h5e8e339_1
- cryptography=3.4.7=py37h5d9358c_0
- cycler=0.10.0=py_2
- cytoolz=0.11.0=py37h5e8e339_3
- dask=2021.9.0=pyhd8ed1ab_0
- dask-core=2021.9.0=pyhd8ed1ab_0
- datashader=0.13.0=pyh6c4a22f_0
- datashape=0.5.4=py_1
- dbus=1.13.6=h48d8840_2
- debugpy=1.4.1=py37hcd2ae1e_0
- decorator=5.1.0=pyhd8ed1ab_0
- defusedxml=0.7.1=pyhd8ed1ab_0
- distributed=2021.9.0=py37h89c1867_0
- entrypoints=0.3=py37hc8dfbb8_1002
- expat=2.4.1=h9c3ff4c_0
- fastparquet=0.7.1=py37hb1e94ed_0
- filelock=3.0.12=pyh9f0ad1d_0
- fontconfig=2.13.1=hba837de_1005
- freetype=2.10.4=h0708190_1
- fsspec=2021.8.1=pyhd8ed1ab_0
- gettext=0.19.8.1=h0b5b191_1005
- gflags=2.2.2=he1b5a44_1004
- gitdb=4.0.7=pyhd8ed1ab_0
- gitpython=3.1.23=pyhd8ed1ab_1
- glib=2.68.4=h9c3ff4c_0
- glib-tools=2.68.4=h9c3ff4c_0
- glog=0.5.0=h48cff8f_0
- greenlet=1.1.1=py37hcd2ae1e_0
- grpc-cpp=1.40.0=h850795e_0
- gst-plugins-base=1.18.5=hf529b03_0
- gstreamer=1.18.5=h76c114f_0
- heapdict=1.0.1=py_0
- holoviews=1.14.5=py_0
- hvplot=0.7.3=py_0
- icu=68.1=h58526e2_0
- idna=3.1=pyhd3deb0d_0
- importlib-metadata=4.8.1=py37h89c1867_0
- importlib_metadata=4.8.1=hd8ed1ab_0
- importlib_resources=5.2.2=pyhd8ed1ab_0
- ipykernel=6.4.1=py37h6531663_0
- ipympl=0.7.0=pyhd8ed1ab_0
- ipython=7.27.0=py37h6531663_0
- ipython_genutils=0.2.0=py_1
- ipywidgets=7.6.5=pyhd8ed1ab_0
- jbig=2.1=h7f98852_2003
- jedi=0.18.0=py37h89c1867_2
- jinja2=3.0.1=pyhd8ed1ab_0
- joblib=1.0.1=pyhd8ed1ab_0
- jpeg=9d=h36c2ea0_0
- json5=0.9.5=pyh9f0ad1d_0
- jsonschema=3.2.0=py37hc8dfbb8_1
- jupyter-server-mathjax=0.2.3=pyhd8ed1ab_0
- jupyter_client=7.0.2=pyhd8ed1ab_0
- jupyter_contrib_core=0.3.3=py_2
- jupyter_contrib_nbextensions=0.5.1=py37hc8dfbb8_1
- jupyter_core=4.7.1=py37h89c1867_0
- jupyter_highlight_selected_word=0.2.0=py37h89c1867_1002
- jupyter_latex_envs=1.4.6=py37h89c1867_1001
- jupyter_nbextensions_configurator=0.4.1=py37h89c1867_2
- jupyter_server=1.11.0=pyhd8ed1ab_0
- jupyterlab=3.1.11=pyhd8ed1ab_0
- jupyterlab-git=0.32.2=pyhd8ed1ab_0
- jupyterlab_pygments=0.1.2=pyh9f0ad1d_0
- jupyterlab_server=2.8.1=pyhd8ed1ab_0
- jupyterlab_widgets=1.0.2=pyhd8ed1ab_0
- kiwisolver=1.3.2=py37h2527ec5_0
- krb5=1.19.2=hcc1bbae_0
- lcms2=2.12=hddcbb42_0
- ld_impl_linux-64=2.36.1=hea4e1c9_2
- lerc=2.2.1=h9c3ff4c_0
- libarchive=3.5.2=hccf745f_0
- libblas=3.9.0=11_linux64_openblas
- libbrotlicommon=1.0.9=h7f98852_5
- libbrotlidec=1.0.9=h7f98852_5
- libbrotlienc=1.0.9=h7f98852_5
- libcblas=3.9.0=11_linux64_openblas
- libclang=11.1.0=default_ha53f305_1
- libcurl=7.78.0=h2574ce0_0
- libdeflate=1.7=h7f98852_5
- libedit=3.1.20191231=he28a2e2_2
- libev=4.33=h516909a_1
- libevent=2.1.10=hcdb4288_3
- libffi=3.3=h58526e2_2
- libgcc-ng=11.1.0=hc902ee8_8
- libgfortran-ng=11.1.0=h69a702a_8
- libgfortran5=11.1.0=h6c583b3_8
- libglib=2.68.4=h3e27bee_0
- libgomp=11.1.0=hc902ee8_8
- libiconv=1.16=h516909a_0
- liblapack=3.9.0=11_linux64_openblas
- libllvm11=11.1.0=hf817b99_2
- libnghttp2=1.43.0=h812cca2_0
- libogg=1.3.4=h7f98852_1
- libopenblas=0.3.17=pthreads_h8fe5266_1
- libopus=1.3.1=h7f98852_1
- libpng=1.6.37=h21135ba_2
- libpq=13.3=hd57d9b9_0
- libprotobuf=3.16.0=h780b84a_0
- libsodium=1.0.18=h36c2ea0_1
- libsolv=0.7.19=h780b84a_5
- libssh2=1.10.0=ha56f1ee_0
- libstdcxx-ng=11.1.0=h56837e0_8
- libta-lib=0.4.0=h516909a_0
- libthrift=0.14.2=he6d91bd_1
- libtiff=4.3.0=hf544144_1
- libutf8proc=2.6.1=h7f98852_0
- libuuid=2.32.1=h7f98852_1000
- libuv=1.42.0=h7f98852_0
- libvorbis=1.3.7=h9c3ff4c_0
- libwebp-base=1.2.1=h7f98852_0
- libxcb=1.13=h7f98852_1003
- libxkbcommon=1.0.3=he3ba5ed_0
- libxml2=2.9.12=h72842e0_0
- libxslt=1.1.33=h15afd5d_2
- llvmlite=0.37.0=py37h9d7f4d0_0
- locket=0.2.0=py_2
- lxml=4.6.3=py37h77fd288_0
- lz4-c=1.9.3=h9c3ff4c_1
- lzo=2.10=h516909a_1000
- mako=1.1.5=pyhd8ed1ab_0
- mamba=0.15.3=py37h7f483ca_0
- markdown=3.3.4=pyhd8ed1ab_0
- markupsafe=2.0.1=py37h5e8e339_0
- matplotlib=3.4.3=py37h89c1867_0
- matplotlib-base=3.4.3=py37h1058ff1_0
- matplotlib-inline=0.1.3=pyhd8ed1ab_0
- mistune=0.8.4=py37h5e8e339_1004
- modin-core=0.10.2=py37h89c1867_3
- modin-ray=0.10.2=py37h89c1867_3
- msgpack-python=1.0.2=py37h2527ec5_1
- multipledispatch=0.6.0=py_0
- mysql-common=8.0.25=ha770c72_2
- mysql-libs=8.0.25=hfa10184_2
- nb_conda_kernels=2.3.1=py37h89c1867_0
- nbclassic=0.3.1=pyhd8ed1ab_1
- nbclient=0.5.4=pyhd8ed1ab_0
- nbconvert=6.1.0=py37h89c1867_0
- nbdime=3.1.0=pyhd8ed1ab_0
- nbformat=5.1.3=pyhd8ed1ab_0
- ncurses=6.2=h58526e2_4
- nest-asyncio=1.5.1=pyhd8ed1ab_0
- notebook=6.4.3=pyha770c72_0
- nspr=4.30=h9c3ff4c_0
- nss=3.69=hb5efdd6_0
- numba=0.54.0=py37h2d894fd_0
- numpy=1.20.3=py37h038b26d_1
- olefile=0.46=pyh9f0ad1d_1
- openjpeg=2.4.0=hb52868f_1
- openssl=1.1.1l=h7f98852_0
- optuna=2.9.1=pyhd8ed1ab_0
- orc=1.6.10=h58a87f1_0
- packaging=21.0=pyhd8ed1ab_0
- pandas=1.3.2=py37he8f5f7f_0
- pandoc=2.14.2=h7f98852_0
- pandocfilters=1.4.2=py_1
- panel=0.12.1=py_0
- param=1.11.1=pyh6c4a22f_0
- parquet-cpp=1.5.1=1
- parso=0.8.2=pyhd8ed1ab_0
- partd=1.2.0=pyhd8ed1ab_0
- patsy=0.5.2=pyhd8ed1ab_0
- pbr=5.6.0=pyhd8ed1ab_0
- pcre=8.45=h9c3ff4c_0
- pexpect=4.8.0=py37hc8dfbb8_1
- pickle5=0.0.11=py37h5e8e339_0
- pickleshare=0.7.5=py37hc8dfbb8_1002
- pillow=8.3.2=py37h0f21c89_0
- pip=21.2.4=pyhd8ed1ab_0
- prettytable=2.2.0=pyhd8ed1ab_0
- prometheus_client=0.11.0=pyhd8ed1ab_0
- prompt-toolkit=3.0.20=pyha770c72_0
- psutil=5.8.0=py37h5e8e339_1
- pthread-stubs=0.4=h36c2ea0_1001
- ptyprocess=0.7.0=pyhd3deb0d_0
- pyarrow=5.0.0=py37h58331f5_5_cpu
- pycosat=0.6.3=py37h5e8e339_1006
- pycparser=2.20=pyh9f0ad1d_2
- pyct=0.4.6=py_0
- pyct-core=0.4.6=py_0
- pygments=2.10.0=pyhd8ed1ab_0
- pykalman=0.9.5=py_1
- pyopenssl=20.0.1=pyhd8ed1ab_0
- pyparsing=2.4.7=pyh9f0ad1d_0
- pyperclip=1.8.2=pyhd8ed1ab_2
- pyqt=5.12.3=py37h89c1867_7
- pyqt-impl=5.12.3=py37he336c9b_7
- pyqt5-sip=4.19.18=py37hcd2ae1e_7
- pyqtchart=5.12=py37he336c9b_7
- pyqtwebengine=5.12.1=py37he336c9b_7
- pyrsistent=0.17.3=py37h5e8e339_2
- pysocks=1.7.1=py37h89c1867_3
- python=3.7.10=hffdb5ce_100_cpython
- python-dateutil=2.8.2=pyhd8ed1ab_0
- python_abi=3.7=2_cp37m
- pytz=2021.1=pyhd8ed1ab_0
- pyviz_comms=2.1.0=py_0
- pyyaml=5.4.1=py37h5e8e339_1
- pyzmq=22.2.1=py37h336d617_0
- qt=5.12.9=hda022c4_4
- ray-core=1.6.0=py37hf931bba_0
- re2=2021.09.01=h9c3ff4c_0
- readline=8.1=h46c0cb4_0
- redis-py=3.5.3=pyh9f0ad1d_0
- reproc=14.2.3=h7f98852_0
- reproc-cpp=14.2.3=h9c3ff4c_0
- requests=2.26.0=pyhd8ed1ab_0
- requests-unixsocket=0.2.0=py_0
- ruamel_yaml=0.15.80=py37h5e8e339_1004
- s2n=1.0.10=h9b69904_0
- scikit-learn=0.24.2=py37hf0f1638_1
- send2trash=1.8.0=pyhd8ed1ab_0
- setproctitle=1.1.10=py37h5e8e339_1004
- setuptools=58.0.4=py37h89c1867_0
- six=1.16.0=pyh6c4a22f_0
- smmap=3.0.5=pyh44b312d_0
- snappy=1.1.8=he1b5a44_3
- sniffio=1.2.0=py37h89c1867_1
- sortedcontainers=2.4.0=pyhd8ed1ab_0
- sqlalchemy=1.4.25=py37h5e8e339_0
- sqlite=3.36.0=h9cd32fc_1
- statsmodels=0.12.2=py37hb1e94ed_0
- stevedore=3.4.0=py37h89c1867_0
- ta-lib=0.4.19=py37ha21ca33_2
- tabulate=0.8.9=pyhd8ed1ab_0
- tblib=1.7.0=pyhd8ed1ab_0
- tensorboardx=2.4=pyhd8ed1ab_0
- terminado=0.12.1=py37h89c1867_0
- testpath=0.5.0=pyhd8ed1ab_0
- threadpoolctl=2.2.0=pyh8a188c0_0
- thrift=0.13.0=py37hcd2ae1e_2
- tk=8.6.11=h27826a3_1
- toolz=0.11.1=py_0
- tornado=6.1=py37h5e8e339_1
- tqdm=4.62.2=pyhd8ed1ab_0
- traitlets=5.1.0=pyhd8ed1ab_0
- typing_extensions=3.10.0.0=pyha770c72_0
- tzdata=2021a=he74cb21_1
- tzlocal=3.0=py37h89c1867_2
- urllib3=1.26.6=pyhd8ed1ab_0
- wcwidth=0.2.5=pyh9f0ad1d_2
- webencodings=0.5.1=py_1
- websocket-client=0.57.0=py37h89c1867_4
- wheel=0.37.0=pyhd8ed1ab_1
- widgetsnbextension=3.5.1=py37h89c1867_4
- xarray=0.19.0=pyhd8ed1ab_1
- xeus=2.0.0=h7d0c39e_0
- xeus-python=0.13.0=py37h4b46df4_1
- xeus-python-shell=0.1.5=pyhd8ed1ab_0
- xorg-libxau=1.0.9=h7f98852_0
- xorg-libxdmcp=1.1.3=h7f98852_0
- xz=5.2.5=h516909a_1
- yaml=0.2.5=h516909a_0
- zeromq=4.3.4=h9c3ff4c_1
- zict=2.0.0=py_0
- zipp=3.5.0=pyhd8ed1ab_0
- zlib=1.2.11=h516909a_1010
- zstandard=0.15.2=py37h5e8e339_0
- zstd=1.5.0=ha95c52a_0
- pip:
- absl-py==0.13.0
- aiohttp==3.7.4.post0
- aiohttp-cors==0.7.0
- aioredis==1.3.1
- async-timeout==3.0.1
- autograd==1.3
- bayesian-optimization==1.2.0
- blessings==1.7
- cachetools==4.2.2
- cma==2.7.0
- colorful==0.5.4
- cython==0.29.24
- future==0.18.2
- google-api-core==1.31.2
- google-auth==1.35.0
- google-auth-oauthlib==0.4.6
- googleapis-common-protos==1.53.0
- gpustat==0.6.0
- gpy==1.10.0
- gpytorch==1.5.1
- grpcio==1.40.0
- hebo==0.1.0
- hiredis==2.0.0
- multidict==5.1.0
- nevergrad==0.4.3.post8
- nvidia-ml-py3==7.352.0
- oauthlib==3.1.1
- opencensus==0.7.13
- opencensus-context==0.1.2
- paramz==0.9.5
- protobuf==3.17.3
- py-spy==0.3.9
- pyasn1==0.4.8
- pyasn1-modules==0.2.8
- pymoo==0.4.2.2
- requests-oauthlib==1.3.0
- rsa==4.7.2
- scipy==1.5.4
- sklearn==0.0
- tensorboard==2.6.0
- tensorboard-data-server==0.6.1
- tensorboard-plugin-wit==1.8.0
- torch==1.9.1
- werkzeug==2.0.1
- yarl==1.6.3Installation
conda
Conda channel
No response
Activity
QuLogic commented on Oct 1, 2021
Please generate some fake data for a reproducible example.
jmakov commented on Oct 1, 2021
Running this cell in jupyter-lab just increases memory every time the cell is executed:
QuLogic commented on Oct 1, 2021
It does appear to use more memory each time, but if you throw in a gc.collect() in between, then there's no leak. I don't know that we can do anything about the lack of garbage collection.
jmakov commented on Oct 1, 2021
Thanks for confirmation. Perhaps you could then update the docs so that anybody that wants to draw a nice plot doesn't have to play a detective for a Week. I don't suppose including the call to gc.collect() is an option since it's probably also a performance hit (but would perhaps make sense to add a flag to plt.show() e.g. plt.show(collect_garbage=True)) or like we have object oriented approach to matplotlib, we could have a section in the docs (e.g. "best practices"/"anti-patterns" like the guys at ray have) about performance and "big" data?
tacaswell commented on Oct 2, 2021
@jmakov I'm sorry you had difficulties with this.
We do have significant reference cycles (the line knows about the axes and the axes knows about the Line2D), so even though you no longer have a direct reference to anything (and if you closed the GUI window we do not either), but due to circular references the reference counts still are not 0. Eventually Python will collect the lingering objects, however for performance reasons only collects the oldest generation every so often (see https://devguide.python.org/garbage_collector/#optimization-generations for details and why in 2014 we backed off on calling gc.collect() see #3045). Because the problem here is a small number of extremely large arrays, it may take more executions of the cell to trigger the automatic collection than you have memory to wait!
I am pretty 👎🏻 on adding additional kwargs to either plt.show() or plt.close() because this is not a Matplotlib specific problem (it can happen with anything with big objects and circular references), but between this, the issues with Qt + C++ cleanup issues, pyplot's holding onto references, and the opposite problem with widgets and animations we I think we need an explanatory page about how Python's gc / object life cycle intersects with Matplotlib.
jmakov commented on Oct 2, 2021
Thank you for the detailed explanation. Agreed, we should then add something to the docs addressing plotting "big" arrays.
jmakov commented on Oct 6, 2021
@QuLogic didn't have time to test before - using gc.collect() before/after plt.show() makes no difference - the memory still only increases after each call. Can you post an example that works for you?
jklymak commented on Oct 6, 2021
How are you testing this? Please test without Jupyter lab. If you can replicate then it is our problem. If you cannot, then it is something in Jupyter lab or ipympl. Thanks.
jmakov commented on Oct 6, 2021
I'm using Jupyter Lab but prepared a script to test:
Thanks for feedback, will raise the issue with Jupyter guys then.
jklymak commented on Oct 6, 2021
In the above, you are not closing anything, so I would expect the memory to increase.
jmakov commented on Oct 6, 2021
Still increases (or do I need to close it in a different way?):
So gc.collect needs to be called explicitly. So as discussed, looks like a Python issue still it's not intuitive and would be great to have something in the docs. The problem in Jupyter-Lab still remains and am trying to reproduce it and then notify the Jupyter guys.
tacaswell commented on Oct 7, 2021
Is that call to plt.show() blocking or not and are you manually closing the GUI windows?
jmakov commented on Oct 7, 2021
It blocks until I manually close the window.
WeatherGod commented on Oct 7, 2021
jmakov commented on Oct 7, 2021
Same behavior with sharex=False. I've run this in a separate script (as recommended here) outside of Jupyiter.
exoticDFT commented on Nov 11, 2021
I am also experiencing a memory increase in my code when using version 3.4.3. I recently upgraded my matplotlib version in hopes to use some of the newer features with the subplots. The code was working with no memory leaks at all in version 3.3.3. However, with version 3.4.3 it seems the figure manager is no longer handling the closing of the figures.
I am not using Jupiter notebook, but running the python interpreter directly from the terminal. I've also run a profiler tool and everything seems to point back to matplotlib. The profiler (FIL) is claiming the issue is somewhere in the new_figure_manager() method. The claim is ~90% of the memory is retained in the first call to creating a figure.
My code looks something like
The close("all") helped solve the same basic memory leak in 3.3.3, but seems to do nothing in 3.4.3.
timhoffm commented on Nov 12, 2021
@exoticDFT your case looks like it's #20300.
tacaswell commented on Dec 5, 2022
I believe this was fixed by #23712, see comments there for why there was a large build up that was fixed by a manual call to gc.collect(2).