FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

.pack and .idx file descriptors left open · Issue #304 · gitpython-developers/GitPython · GitHub

  1. gitpython-developers
  2. GitPython

Repository navigation

.pack and .idx file descriptors left open #304

Closed
Closed
.pack and .idx file descriptors left open#304
Labels
acknowledgedtag.leaks
Milestone
v1.0.2 - Fixes

Description

jantman
opened on Jun 26, 2015

Sorry, but I believe I found another issue. It appears (maybe I'm missing something here, because this seems very strange to me) that file descriptors are left open after the Repo object is unused or destroyed.

(foo)jantman@jarvis:pts/8:~/tmp/foo$ cat test.py
#!/usr/bin/env python

import git
import commands
import os
import time

pid = os.getpid()

def show_lsof():
    global pid
    cmd = "lsof -p {p} | grep -v /usr/lib".format(p=pid)
    print("=============================")
    print("running {c}:".format(c=cmd))
    print(commands.getoutput(cmd))
    print("=============================")

for rname in ['testrepo', 'testrepo2', 'testrepo3']:
    print("=> Creating repo object for {rname}:".format(rname=rname))
    r = git.Repo(rname)
    show_lsof()

    print("=> iterating commits")
    for c in r.iter_commits():
        pass
    show_lsof()

    time.sleep(2)
    print("=> del r")
    del r
    time.sleep(2)

    show_lsof()
(foo)jantman@jarvis:pts/8:~/tmp/foo$ ./test.py  
=> Creating repo object for testrepo:
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  88651 pipe
=============================
=> iterating commits
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  91431 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> del r
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  91434 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> Creating repo object for testrepo2:
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  91435 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> iterating commits
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  87601 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    6r   REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    7r   REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> del r
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  90605 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    6r   REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    7r   REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> Creating repo object for testrepo3:
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  87612 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    6r   REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    7r   REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> iterating commits
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  87616 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    6r   REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    7r   REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    8r   REG   0,30    42220  79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    9r   REG   0,30     3032  79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> del r
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  88654 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    6r   REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    7r   REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    8r   REG   0,30    42220  79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    9r   REG   0,30     3032  79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================

The same thing happens if I explicitly r = None. This is a real problem for me, as I'm trying to write an analysis script that looks at commits in about 450 repositories. On Linux with a default open file handle limit of 1024, the script ends up crashing as every repository I examine results in open file handles for the pack and index files, that appear to never be cleaned up.

I feel like I have to be doing something wrong here, as I don't understand why these aren't cleaned up even after I'm done using the Repo object, and I set it to None or del it.

Activity

  1. Byron commented on Jun 26, 2015

    Byron
    on Jun 26, 2015
    Member

    I think this one might be very close to issue #287, which also provides a workaround: You can use a different back-end which will not leak (at least not in the python process):

    import git
r = git.Repo(odbt=git.GitCmdObjectDB)

    Also I don't think I can fix this permanently, except for maybe changing the default object database type to the one mentioned. The reason for me being so pessimistic about it is that I came to believe destructors in python cannot be trusted, especially in python 3. Maybe there is some sort of cycle that prevents cleanup in python 2.

    Please let me know what you think.

  2. Byron
    added this to the v1.0.2 - Fixes milestone on Jun 26, 2015
  3. Byron
    added
    waiting for feedback
    on Jun 26, 2015

  4. jantman commented on Jun 29, 2015

    jantman
    on Jun 29, 2015
    Author

    Ok. It's working OK now with the GitCmdObjectDB backend. I couldn't easily find the code that actually opens these files, but it seems to me that perhaps using a context manager would help? Or, since this is an edge case, maybe rather than relying on destructors, this could be handled using an explicit .close_db() method on the Repo object?

  5. Byron commented on Jun 29, 2015

    Byron
    on Jun 29, 2015
    Member

    Good to hear.
    I think the open handles problem originates in smmap, which memory maps everything and provides windowed access to the mapped regions. It is made to de-allocate unused regions when a certain amount of max memory is used. If I remember correctly, it will also deallocate in case it can't open files anymore due to OS limits. However, that functionality was never tested and probably doesn't work.

    Everything is possible, yet I am in bare-minimum maintenance mode which means that workarounds like the one you use will be all I can do in this case.
    Thanks for your understanding.

  6. Byron
    added
    acknowledged
    and removed
    waiting for feedback
    on Jun 29, 2015

  7. FrostyX commented on Aug 18, 2015

    FrostyX
    on Aug 18, 2015

    I've spent two days on this and I have no idea how to fix it. 💩 :guilty: ...

    I dig down through the code of GitPython, gitdb and then smmap and in my opinion the true issue lays down in the smmap project. Fixing it above would be just workaround I think.

    Please look at the following code, which is based on PackFile::_set_cache_(...):

    from smmap import SlidingWindowMapManager
packpath = "/.../.git/objects/pack/pack-022615213253184faeb810dbff2eb8585786d874.pack"
mman = SlidingWindowMapManager()
x = mman.make_cursor(packpath).use_region()

    This is place, where those descriptors appears (because of use_region()). We expect that it properly ends, because there is some cleaning in __del__ method. However, any of these will not do the cleaning up:

    x.unuse_region()
del x

    I've found two ways how to do it, but one which is kinda violent and second one I couldn't manage to pass tests, so there will probably be something stinky about it too.

    Firstly, deleting the mman variable definitely helps. Problem here is that it is defined on module-level so deleting it will be, lets say, not particularly shiny.

    Secondly, there is method unuse_region() on WindowCursor in which is line self._region = None. I think that exactly this is the broken place. This line in fact throws out the current reference to MapRegion object so we don't have an opportunity to access it again, but it still exists and holds the descriptors. Try to put self._region.release() above the line (to the condition, cause _region can be None) and my example code will work. Unfortunately it breaks some other stuff which I can't understand. Even so I think that deleting WindowCursor should free the descriptors.

    I hope this post will spare some time to the person who will try to fix it.

  8. Byron commented on Aug 18, 2015

    Byron
    on Aug 18, 2015
    Member

    Thank you for posting your findings !
    Also I am very sorry to hear you spent so much time on this ... really ... nobody should have to do that as the code was built with the false premises of having working (i.e. deterministic) destructors :( !

    Also I hope you are aware of the fix using the GitCmdObjectDB as odbt when instantiating Repo objects - this solved the problem for jantman, and is the only workaround I know.

  9. jantman commented on Aug 19, 2015

    jantman
    on Aug 19, 2015
    Author

    For me at least, I think I'd be happy with an update to the docs somewhere to mention the workaround. I understand being in maintenance mode, and I'm probably never going to have to deal with repos that big again (I hope)...

  10. Byron
    added a commit that references this issue on Aug 22, 2015

    fix(repo): use GitCmdObjectDB by default

    6eb3af2

  11. Byron commented on Aug 22, 2015

    Byron
    on Aug 22, 2015
    Member

    In order to fix it once and for all, the default object database type is now one using the system's git command. Additionally, the usage of the pure-python GitDB type is now discouraged.

    Even though that won't fix this issue, it's the best I can do. If you see anything else that could be done here, please let me know, and feel free to reopen the issue.

  12. Byron
    closed this as completedon Aug 22, 2015

  13. Byron commented on Aug 22, 2015

    Byron
    on Aug 22, 2015
    Member

    You can watch the development stream on youtube.

    GitPython #21 [issue 304 - fix resource leaks once and for all]

  14. ankostis
    added
    tag.leaks
    on Oct 30, 2016
  15. ankostis
    added a commit that references this issue on Oct 30, 2016

    refact(odb): remove unneeded PY24 compat code after gitpython-develop…

    ad088b9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    acknowledgedtag.leaks

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      Back | FazBrowse Home | New Git URL