Repository navigation
Deleted dotnet object does not get garbage collected #1872
Description
Environment
- Pythonnet version: 3.0.0rc3
- Python version: 3.10.5
- Operating System: Windows 10 20H2
- .NET Runtime: 6.0.6
Details
- Describe what you were trying to get done.
Create dotnet-object and delete it. If I do that a lot I get memory issues. It seems the never get cleared by the Python- or dotnet-gc. This is not a problem in pythonnet 2.5.2
- What commands did you run to trigger this issue?
import gc
import clr
import psutil
import matplotlib.pyplot as plt
import System
# Create dict store count of objects left
types = [
str,
System.String,
]
types = {obj: 0 for obj in types}
N = 1_000
str_size = 1_000_000
memory_py = [0 for _ in range(N + 2)]
memory_py[0] = psutil.virtual_memory().used
for ii in range(N):
s_py = str('a' * str_size)
del s_py
memory_py[ii + 1] = psutil.virtual_memory().used - memory_py[0]
gc.collect()
# See what types of objects are still present
for obj in gc.get_objects():
if type(obj) in list(types.keys()):
types[type(obj)] += 1
print(types)
memory_py[ii + 2] = psutil.virtual_memory().used - memory_py[0]
memory_py[0] = 0{<class 'str'>: 0, <class 'System.String'>: 0}
memory_net = [0 for _ in range(N + 2)]
memory_net[0] = psutil.virtual_memory().used
s_py = str('a' * str_size)
for ii in range(N):
s_net = System.String(s_py)
del s_net
memory_net[ii + 1] = psutil.virtual_memory().used - memory_net[0]
gc.collect()
# See what types of objects are still present
for obj in gc.get_objects():
if type(obj) in list(types.keys()):
types[type(obj)] += 1
print(types)
memory_net[ii + 2] = psutil.virtual_memory().used - memory_net[0]
memory_net[0] = 0{<class 'str'>: 0, <class 'System.String'>: 1000}
You can also visualize the memory consumption:
fig, ax = plt.subplots()
ax.plot(memory_py, label='Python object')
ax.plot(memory_net, label='DotNet object')
ax.legend()
plt.show(fig)
You can execute the same code with pythonnet 2.5.2 (Python 3.9.13) and get the following figure
Activity
lostmsu commented on Jul 13, 2022
You are not calling .NET GC
m-rossi commented on Jul 14, 2022
@lostmsu I get the same results if I add System.GC.Collect()
Full code for reference:
{<class 'str'>: 0, <class 'System.String'>: 0}{<class 'str'>: 0, <class 'System.String'>: 1000}filmor commented on Jul 14, 2022
Hmm, this tells us that Python is not releasing the wrapper objects for some reason. Can you check the refcount and referrers for these?
m-rossi commented on Jul 14, 2022
I used objgraph to display the backrefs. The output of gc.get_referrers(obj) is really huge or have you another advice to display the referrers?
lostmsu commented on Jul 14, 2022
@filmor an update from my debugging the issue:
From this code I am getting a refcount rc of 3:
Replacing Uri with Python's str gives rc 1.
The issue does not seem to happen when objects are created from .NET. E.g. this does not produce extra references:
Looks like we leak references when constructing new .NET objects from Python.
fixed leak in NewReference.Move
fixed leak in NewReference.Move
fixed leak in NewReference.Move
filmor commented on Jul 14, 2022
@lostmsu Great work, thanks! And thank you @m-rossi for testing and reporting! Could you verify that the newest release branch indeed fixes the issue?
m-rossi commented on Jul 15, 2022
Looks good, the gc sees no System.String-objects left and the memory consumption is better too:
Thanks @lostmsu !
fixed leak in NewReference.Move