Repository navigation
Qt5Agg is leaking #9141
Description
22:39 $ python unit/memleak.py Qt5Agg 50 out.png 0: pymalloc 17627699, rss 116760576, nobjs 55869, garbage 0, files: 2 1: pymalloc 17634545, rss 117694464, nobjs 55872, garbage 0, files: 2 2: pymalloc 17635196, rss 118116352, nobjs 55875, garbage 0, files: 2 3: pymalloc 17635615, rss 118386688, nobjs 55878, garbage 0, files: 2 4: pymalloc 17636002, rss 118386688, nobjs 55881, garbage 0, files: 2 5: pymalloc 17636389, rss 118386688, nobjs 55884, garbage 0, files: 2 6: pymalloc 17636776, rss 118386688, nobjs 55887, garbage 0, files: 2 7: pymalloc 17637176, rss 118386688, nobjs 55890, garbage 0, files: 2 8: pymalloc 17637579, rss 118386688, nobjs 55893, garbage 0, files: 2 9: pymalloc 17637985, rss 118386688, nobjs 55896, garbage 0, files: 2 10: pymalloc 17638340, rss 118386688, nobjs 55899, garbage 0, files: 2 11: pymalloc 17638791, rss 118386688, nobjs 55902, garbage 0, files: 2 12: pymalloc 17639162, rss 118386688, nobjs 55905, garbage 0, files: 2 13: pymalloc 17639533, rss 118386688, nobjs 55908, garbage 0, files: 2 14: pymalloc 17639920, rss 118386688, nobjs 55911, garbage 0, files: 2 15: pymalloc 17640307, rss 118386688, nobjs 55914, garbage 0, files: 2 16: pymalloc 17640694, rss 118386688, nobjs 55917, garbage 0, files: 2 17: pymalloc 17641081, rss 118386688, nobjs 55920, garbage 0, files: 2 18: pymalloc 17641468, rss 118386688, nobjs 55923, garbage 0, files: 2 19: pymalloc 17641902, rss 118386688, nobjs 55926, garbage 0, files: 2 20: pymalloc 17642323, rss 118386688, nobjs 55929, garbage 0, files: 2 21: pymalloc 17642573, rss 118386688, nobjs 55932, garbage 0, files: 2 22: pymalloc 17643421, rss 118386688, nobjs 55935, garbage 0, files: 2 23: pymalloc 17643987, rss 118386688, nobjs 55938, garbage 0, files: 2 24: pymalloc 17644342, rss 118386688, nobjs 55941, garbage 0, files: 2 25: pymalloc 17644745, rss 118386688, nobjs 55944, garbage 0, files: 2 26: pymalloc 17645132, rss 118386688, nobjs 55947, garbage 0, files: 2 27: pymalloc 17645519, rss 118386688, nobjs 55950, garbage 0, files: 2 28: pymalloc 17645906, rss 118386688, nobjs 55953, garbage 0, files: 2 29: pymalloc 17646340, rss 118386688, nobjs 55956, garbage 0, files: 2 30: pymalloc 17646816, rss 118386688, nobjs 55959, garbage 0, files: 2 31: pymalloc 17647075, rss 118386688, nobjs 55962, garbage 0, files: 2 32: pymalloc 17647470, rss 118386688, nobjs 55965, garbage 0, files: 2 33: pymalloc 17647857, rss 118386688, nobjs 55968, garbage 0, files: 2 34: pymalloc 17648244, rss 118386688, nobjs 55971, garbage 0, files: 2 35: pymalloc 17648631, rss 118386688, nobjs 55974, garbage 0, files: 2 36: pymalloc 17649049, rss 118386688, nobjs 55977, garbage 0, files: 2 37: pymalloc 17649461, rss 118386688, nobjs 55980, garbage 0, files: 2 38: pymalloc 17649784, rss 118386688, nobjs 55983, garbage 0, files: 2 39: pymalloc 17650115, rss 118386688, nobjs 55986, garbage 0, files: 2 40: pymalloc 17650531, rss 118386688, nobjs 55989, garbage 0, files: 2 41: pymalloc 17651001, rss 118386688, nobjs 55992, garbage 0, files: 2 42: pymalloc 17651260, rss 118386688, nobjs 55995, garbage 0, files: 2 43: pymalloc 17651727, rss 118386688, nobjs 55998, garbage 0, files: 2 44: pymalloc 17652114, rss 118386688, nobjs 56001, garbage 0, files: 2 45: pymalloc 17652501, rss 118386688, nobjs 56004, garbage 0, files: 2 46: pymalloc 17652888, rss 118386688, nobjs 56007, garbage 0, files: 2 47: pymalloc 17653195, rss 118386688, nobjs 56010, garbage 0, files: 2 48: pymalloc 17653611, rss 118386688, nobjs 56013, garbage 0, files: 2 49: pymalloc 17654065, rss 118657024, nobjs 56016, garbage 0, files: 2 Average memory consumed per loop: 10813.4400 bytes
Activity
QuLogic commented on Sep 1, 2017
Also, I get this at the end:
Also, pretty sure that sum is redundant and could just be (rss_peaks[-1] - rss_peaks[starti]). doesn't seem to work that way.
tacaswell commented on Sep 1, 2017
See #9142 for a fix to that.
anntzer commented on Sep 1, 2017
This bisects to earlier than 1.5.0 (#6854 needs to be manually applied for testing early commits, and I copied unit/memleak.py out of the source tree for the bisection).
I can't test 1.4.3 because trying to build it crashes gcc :-)
src/_png.cpp: In member function ‘Py::Object _png_module::write_png(const Py::Tuple&)’: src/_png.cpp:103:12: internal compiler error: Segmentation fault Py::Object _png_module::write_png(const Py::Tuple& args) ^~~~~~~~~~~ Please submit a full bug report, with preprocessed source if appropriate. See <https://bugs.archlinux.org/> for instructions. error: command 'gcc' failed with exit status 1Given how long the issue has been around I would suggest that this issue, while important, should not be considered as release critical.
Edit: looks like we're leaking dead weakrefs (found by comparing Counter(map(type, gc.get_objects())) between iterations), which is quite funny in a sense :-)
QuLogic commented on Sep 1, 2017
Every time I bisect, I get a different commit, so this seems a bit random. We probably need to figure out a faster&stabler test to really get to the bottom of this.
anntzer commented on Sep 1, 2017
import gc from matplotlib import pyplot as plt for _ in range(10): plt.figure() gc.collect() print(len(gc.get_objects())) plt.close("all")is sufficient to show the leak, so we can in fact just run the loop two or three times and fail if the len's are different.
Specifically, tkagg always returns the same number, whereas qt5agg sees a jump of 158 objects between the first and second iteration, and 3 objects per iteration after that. The test always fails even as far back as 1.5.0 + qt4agg.
tacaswell commented on Sep 1, 2017
I wonder if this due to a change in qt/pyqt not necessarily on our side?
anntzer commented on Sep 7, 2017
I think it's a bug in pyqt. PySide(2) also leaks, but appears to leak only one object per iteration.
I find that the leaked dead weakrefs started as pointing respectively to a FigureManagerQT, a FigureCanvasQTAgg, and a NavigationToolbar2QT.
https://pypi.python.org/pypi/objgraph is useful for this kind of exploration...
dopplershift commented on Sep 7, 2017
If both are leaking, I'm skeptical that matplotlib is doing the right thing.
tacaswell commented on Sep 20, 2017
We are not doing any of the c-work with pyqt which is the easiest way leak references.
The big jump between 1 and 2 does not bother me, that seems like qt starting it's self up.
2 of those classes are things we subclass from qt objects, but all of then are things that I think we have in circular references with Qt objects.
tacaswell commented on May 22, 2020
Going to optimistically close this as fixed by #17468
anntzer commented on May 22, 2020
is is actually fixed, or are you just "optimistically" hoping it is?
tacaswell commented on May 22, 2020
I was going on #17385 (comment)
Going on @QuLogic 's comment in this thread, I think the problem may be worse than it bisecting badly, the leak seems to be non-deterministic:
(bleeding) ✘ ~/source/p/matplotlib [master {origin/master}|… 3⚑ 14] jupiter@09:04 ➤ python tools/memleak.py qt5Agg 50 out.png 0: pymalloc 1118851, rss 110944256, nobjs 62623, garbage 0, files: 1 1: pymalloc 1128306, rss 110813184, nobjs 62626, garbage 0, files: 1 2: pymalloc 1135252, rss 111386624, nobjs 62629, garbage 0, files: 1 3: pymalloc 1137633, rss 111624192, nobjs 62632, garbage 0, files: 1 4: pymalloc 1143859, rss 111624192, nobjs 62635, garbage 0, files: 1 5: pymalloc 1147160, rss 111624192, nobjs 62638, garbage 0, files: 1 6: pymalloc 1148843, rss 111865856, nobjs 62641, garbage 0, files: 1 7: pymalloc 1152449, rss 111865856, nobjs 62644, garbage 0, files: 1 8: pymalloc 1153153, rss 111865856, nobjs 62647, garbage 0, files: 1 9: pymalloc 1162038, rss 111865856, nobjs 62650, garbage 0, files: 1 10: pymalloc 1162482, rss 111865856, nobjs 62653, garbage 0, files: 1 11: pymalloc 1164973, rss 111865856, nobjs 62656, garbage 0, files: 1 12: pymalloc 1166401, rss 111865856, nobjs 62659, garbage 0, files: 1 13: pymalloc 1167127, rss 111865856, nobjs 62662, garbage 0, files: 1 14: pymalloc 1171441, rss 111865856, nobjs 62665, garbage 0, files: 1 15: pymalloc 1180176, rss 112914432, nobjs 62668, garbage 0, files: 1 16: pymalloc 1182889, rss 111857664, nobjs 62671, garbage 0, files: 1 17: pymalloc 1183715, rss 111857664, nobjs 62674, garbage 0, files: 1 18: pymalloc 1183952, rss 111857664, nobjs 62677, garbage 0, files: 1 19: pymalloc 1185027, rss 111857664, nobjs 62680, garbage 0, files: 1 20: pymalloc 1185216, rss 111857664, nobjs 62683, garbage 0, files: 1 21: pymalloc 1187158, rss 111857664, nobjs 62686, garbage 0, files: 1 22: pymalloc 1187527, rss 111857664, nobjs 62689, garbage 0, files: 1 23: pymalloc 1188422, rss 111857664, nobjs 62692, garbage 0, files: 1 24: pymalloc 1189088, rss 111857664, nobjs 62695, garbage 0, files: 1 25: pymalloc 1189721, rss 111857664, nobjs 62698, garbage 0, files: 1 26: pymalloc 1190026, rss 111857664, nobjs 62701, garbage 0, files: 1 27: pymalloc 1190415, rss 111857664, nobjs 62704, garbage 0, files: 1 28: pymalloc 1191142, rss 111857664, nobjs 62707, garbage 0, files: 1 29: pymalloc 1191647, rss 111857664, nobjs 62710, garbage 0, files: 1 30: pymalloc 1192326, rss 111857664, nobjs 62713, garbage 0, files: 1 31: pymalloc 1197864, rss 111857664, nobjs 62716, garbage 0, files: 1 32: pymalloc 1198653, rss 112906240, nobjs 62719, garbage 0, files: 1 33: pymalloc 1198817, rss 111857664, nobjs 62722, garbage 0, files: 1 34: pymalloc 1199437, rss 111857664, nobjs 62725, garbage 0, files: 1 35: pymalloc 1202020, rss 111857664, nobjs 62728, garbage 0, files: 1 36: pymalloc 1202204, rss 111857664, nobjs 62731, garbage 0, files: 1 37: pymalloc 1203234, rss 111857664, nobjs 62734, garbage 0, files: 1 38: pymalloc 1203641, rss 111857664, nobjs 62737, garbage 0, files: 1 39: pymalloc 1203841, rss 111857664, nobjs 62740, garbage 0, files: 1 40: pymalloc 1204025, rss 111857664, nobjs 62743, garbage 0, files: 1 41: pymalloc 1204200, rss 112906240, nobjs 62746, garbage 0, files: 1 42: pymalloc 1205061, rss 112906240, nobjs 62749, garbage 0, files: 1 43: pymalloc 1205366, rss 111857664, nobjs 62752, garbage 0, files: 1 44: pymalloc 1205963, rss 112906240, nobjs 62755, garbage 0, files: 1 45: pymalloc 1206284, rss 112906240, nobjs 62758, garbage 0, files: 1 46: pymalloc 1207128, rss 112906240, nobjs 62761, garbage 0, files: 1 47: pymalloc 1207312, rss 112906240, nobjs 62764, garbage 0, files: 1 48: pymalloc 1208113, rss 111857664, nobjs 62767, garbage 0, files: 1 49: pymalloc 1208313, rss 112906240, nobjs 62770, garbage 0, files: 1 Average memory consumed per loop: 0.0000 bytes (bleeding) ✔ ~/source/p/matplotlib [master {origin/master}|… 3⚑ 14] jupiter@09:05 ➤ python tools/memleak.py qt5Agg 50 out.png 0: pymalloc 1118751, rss 111480832, nobjs 62623, garbage 0, files: 1 1: pymalloc 1128495, rss 111345664, nobjs 62626, garbage 0, files: 1 2: pymalloc 1135367, rss 111472640, nobjs 62629, garbage 0, files: 1 3: pymalloc 1137880, rss 111910912, nobjs 62632, garbage 0, files: 1 4: pymalloc 1144018, rss 112152576, nobjs 62635, garbage 0, files: 1 5: pymalloc 1147251, rss 112152576, nobjs 62638, garbage 0, files: 1 6: pymalloc 1149070, rss 112152576, nobjs 62641, garbage 0, files: 1 7: pymalloc 1152520, rss 112152576, nobjs 62644, garbage 0, files: 1 8: pymalloc 1153208, rss 112148480, nobjs 62647, garbage 0, files: 1 9: pymalloc 1162150, rss 112148480, nobjs 62650, garbage 0, files: 1 10: pymalloc 1162530, rss 112148480, nobjs 62653, garbage 0, files: 1 11: pymalloc 1165166, rss 112148480, nobjs 62656, garbage 0, files: 1 12: pymalloc 1166417, rss 112148480, nobjs 62659, garbage 0, files: 1 13: pymalloc 1167187, rss 112148480, nobjs 62662, garbage 0, files: 1 14: pymalloc 1171352, rss 112148480, nobjs 62665, garbage 0, files: 1 15: pymalloc 1180071, rss 112148480, nobjs 62668, garbage 0, files: 1 16: pymalloc 1182760, rss 112148480, nobjs 62671, garbage 0, files: 1 17: pymalloc 1183586, rss 112148480, nobjs 62674, garbage 0, files: 1 18: pymalloc 1183730, rss 112148480, nobjs 62677, garbage 0, files: 1 19: pymalloc 1185026, rss 112148480, nobjs 62680, garbage 0, files: 1 20: pymalloc 1185308, rss 112148480, nobjs 62683, garbage 0, files: 1 21: pymalloc 1186965, rss 112148480, nobjs 62686, garbage 0, files: 1 22: pymalloc 1187318, rss 112148480, nobjs 62689, garbage 0, files: 1 23: pymalloc 1188277, rss 112148480, nobjs 62692, garbage 0, files: 1 24: pymalloc 1188914, rss 112148480, nobjs 62695, garbage 0, files: 1 25: pymalloc 1189697, rss 112148480, nobjs 62698, garbage 0, files: 1 26: pymalloc 1190057, rss 112148480, nobjs 62701, garbage 0, files: 1 27: pymalloc 1190403, rss 112148480, nobjs 62704, garbage 0, files: 1 28: pymalloc 1190912, rss 113197056, nobjs 62707, garbage 0, files: 1 29: pymalloc 1191654, rss 113197056, nobjs 62710, garbage 0, files: 1 30: pymalloc 1192342, rss 112390144, nobjs 62713, garbage 0, files: 1 31: pymalloc 1197762, rss 112148480, nobjs 62716, garbage 0, files: 1 32: pymalloc 1198780, rss 112390144, nobjs 62719, garbage 0, files: 1 33: pymalloc 1199012, rss 112390144, nobjs 62722, garbage 0, files: 1 34: pymalloc 1199484, rss 112390144, nobjs 62725, garbage 0, files: 1 35: pymalloc 1202083, rss 112390144, nobjs 62728, garbage 0, files: 1 36: pymalloc 1202367, rss 112390144, nobjs 62731, garbage 0, files: 1 37: pymalloc 1203533, rss 112390144, nobjs 62734, garbage 0, files: 1 38: pymalloc 1203856, rss 112390144, nobjs 62737, garbage 0, files: 1 39: pymalloc 1204040, rss 112390144, nobjs 62740, garbage 0, files: 1 40: pymalloc 1204256, rss 112390144, nobjs 62743, garbage 0, files: 1 41: pymalloc 1204420, rss 112390144, nobjs 62746, garbage 0, files: 1 42: pymalloc 1205317, rss 112390144, nobjs 62749, garbage 0, files: 1 43: pymalloc 1205533, rss 112390144, nobjs 62752, garbage 0, files: 1 44: pymalloc 1206151, rss 112390144, nobjs 62755, garbage 0, files: 1 45: pymalloc 1206431, rss 112390144, nobjs 62758, garbage 0, files: 1 46: pymalloc 1207307, rss 112390144, nobjs 62761, garbage 0, files: 1 47: pymalloc 1207507, rss 112390144, nobjs 62764, garbage 0, files: 1 48: pymalloc 1208299, rss 112390144, nobjs 62767, garbage 0, files: 1 49: pymalloc 1208636, rss 112390144, nobjs 62770, garbage 0, files: 1 Average memory consumed per loop: 41779.2000 bytes (bleeding) ✔ ~/source/p/matplotlib [master {origin/master}|… 3⚑ 14] jupiter@09:05 ➤ python tools/memleak.py qt5Agg 50 out.png 0: pymalloc 1119490, rss 111407104, nobjs 62623, garbage 0, files: 1 1: pymalloc 1129503, rss 111271936, nobjs 62626, garbage 0, files: 1 2: pymalloc 1135930, rss 111837184, nobjs 62629, garbage 0, files: 1 3: pymalloc 1138425, rss 112078848, nobjs 62632, garbage 0, files: 1 4: pymalloc 1144442, rss 112078848, nobjs 62635, garbage 0, files: 1 5: pymalloc 1147666, rss 112078848, nobjs 62638, garbage 0, files: 1 6: pymalloc 1149494, rss 112078848, nobjs 62641, garbage 0, files: 1 7: pymalloc 1152908, rss 112078848, nobjs 62644, garbage 0, files: 1 8: pymalloc 1153712, rss 112078848, nobjs 62647, garbage 0, files: 1 9: pymalloc 1162593, rss 112078848, nobjs 62650, garbage 0, files: 1 10: pymalloc 1163098, rss 112078848, nobjs 62653, garbage 0, files: 1 11: pymalloc 1165327, rss 112078848, nobjs 62656, garbage 0, files: 1 12: pymalloc 1167049, rss 112078848, nobjs 62659, garbage 0, files: 1 13: pymalloc 1167819, rss 112078848, nobjs 62662, garbage 0, files: 1 14: pymalloc 1171920, rss 113127424, nobjs 62665, garbage 0, files: 1 15: pymalloc 1180550, rss 112078848, nobjs 62668, garbage 0, files: 1 16: pymalloc 1183536, rss 112078848, nobjs 62671, garbage 0, files: 1 17: pymalloc 1184346, rss 113127424, nobjs 62674, garbage 0, files: 1 18: pymalloc 1184686, rss 112078848, nobjs 62677, garbage 0, files: 1 19: pymalloc 1185834, rss 112078848, nobjs 62680, garbage 0, files: 1 20: pymalloc 1186252, rss 112078848, nobjs 62683, garbage 0, files: 1 21: pymalloc 1187916, rss 112078848, nobjs 62686, garbage 0, files: 1 22: pymalloc 1188374, rss 112078848, nobjs 62689, garbage 0, files: 1 23: pymalloc 1189292, rss 112078848, nobjs 62692, garbage 0, files: 1 24: pymalloc 1190079, rss 112078848, nobjs 62695, garbage 0, files: 1 25: pymalloc 1190705, rss 112078848, nobjs 62698, garbage 0, files: 1 26: pymalloc 1190889, rss 112078848, nobjs 62701, garbage 0, files: 1 27: pymalloc 1191251, rss 113127424, nobjs 62704, garbage 0, files: 1 28: pymalloc 1191853, rss 112078848, nobjs 62707, garbage 0, files: 1 29: pymalloc 1192349, rss 113127424, nobjs 62710, garbage 0, files: 1 30: pymalloc 1193174, rss 112078848, nobjs 62713, garbage 0, files: 1 31: pymalloc 1198719, rss 112078848, nobjs 62716, garbage 0, files: 1 32: pymalloc 1199508, rss 112078848, nobjs 62719, garbage 0, files: 1 33: pymalloc 1199656, rss 112078848, nobjs 62722, garbage 0, files: 1 34: pymalloc 1200224, rss 112078848, nobjs 62725, garbage 0, files: 1 35: pymalloc 1202907, rss 112078848, nobjs 62728, garbage 0, files: 1 36: pymalloc 1203091, rss 112078848, nobjs 62731, garbage 0, files: 1 37: pymalloc 1204077, rss 112078848, nobjs 62734, garbage 0, files: 1 38: pymalloc 1204400, rss 112316416, nobjs 62737, garbage 0, files: 1 39: pymalloc 1204600, rss 112316416, nobjs 62740, garbage 0, files: 1 40: pymalloc 1204816, rss 112316416, nobjs 62743, garbage 0, files: 1 41: pymalloc 1205032, rss 112316416, nobjs 62746, garbage 0, files: 1 42: pymalloc 1205845, rss 112316416, nobjs 62749, garbage 0, files: 1 43: pymalloc 1206061, rss 112316416, nobjs 62752, garbage 0, files: 1 44: pymalloc 1206763, rss 112316416, nobjs 62755, garbage 0, files: 1 45: pymalloc 1206979, rss 112316416, nobjs 62758, garbage 0, files: 1 46: pymalloc 1207871, rss 112316416, nobjs 62761, garbage 0, files: 1 47: pymalloc 1208087, rss 112316416, nobjs 62764, garbage 0, files: 1 48: pymalloc 1208904, rss 112316416, nobjs 62767, garbage 0, files: 1 49: pymalloc 1209120, rss 112316416, nobjs 62770, garbage 0, files: 1 Average memory consumed per loop: 0.0000 bytes (bleeding) ✔ ~/source/p/matplotlib [master {origin/master}|… 3⚑ 14] jupiter@09:06 ➤all 3 run with in a few minutes of each other in the env at v3.2.1-2585-ga1c51a0a1.
If you let it run longer you get:
qt5.pdf
I will open a PR with slight tweaks to the memleak script. We are never spinning the event loop so I think a bunch of the "leaked" memory is Qt objects that we have discarded but need the event loop to spin to finish their end-of-life process.