Activity

1. voxik

   changed the title [-]GitPython corrupts git repository[/-] [+]GitPython corrupts git index[/+] on Mar 2, 2015
2. Byron

   added

   acknowledged

   critical

   on Mar 2, 2015
3. Byron

   added this to the v0.3.7 - Fixes milestone on Mar 2, 2015
4. Byron commented on Mar 2, 2015

   

   Byron

   on Mar 2, 2015

   Member

   More actions

   I have seen this too, but in conjunction with submodules. GitPython writes the index itself, and apparently it doesn't write back the expected format.

   A workaround, as described there, is rewriting the index using something like git reset HEAD. The problem certainly occours when using repo.index.add(...), as this one will rewrite the index. As a quick fix, one could forcefully use the git command for this in the downstream source, such as in repo.git.add(...).

   Can you point me to code (e.g. bash) that helps me reproducing this, and the original source code of fedpkg ? If I can reproduce it, I can certainly fix it.

   Thank you
5. Byron

   added

   waiting for feedback

   on Mar 2, 2015
6. voxik commented on Mar 2, 2015

   

   voxik

   on Mar 2, 2015

   Author

   More actions

   Thank you very much.

   This should be the steps:

   $ fedpkg co GitPython --anonymous   # --anonymous in case you are not Fedora contributor ;)
   $ cd GitPython
   $ vim GitPython.spec   # and do some editing.
   $ fedpkg srpm
   $ fedpkg import GitPython-0.3.2-0.7.RC1.fc23.src.rpm
   

   fedpkg [1] is just thin wrapper abouve pyrpkg, so this [2] should be the place where the index is modified after fedpkg import

   [1] https://git.fedorahosted.org/cgit/fedpkg.git/tree/
   [2] https://git.fedorahosted.org/cgit/rpkg.git/tree/src/pyrpkg/__init__.py#n1425
7. Byron

   removed

   waiting for feedback

   on Mar 2, 2015
8. Byron commented on Mar 2, 2015

   

   Byron

   on Mar 2, 2015

   Member

   More actions

   Thank you - it's about time that bug gets squashed. I will dedicate my time tomorrow, in favour of finishing other work today.
   Thanks to the workarounds we have, there are options in case this is not fast enough.
   Besides: My apologies - after all, that one could have been fixed earlier.
9. voxik commented on Mar 2, 2015

   

   voxik

   on Mar 2, 2015

   Author

   More actions

   Thanks a lot. pyrpkg maintainer is now aware about the repo.git.add workaround, so he might apply it as well.
10. Byron

    self-assigned this

    on Apr 7, 2015
11. Byron commented on Apr 8, 2015

    

    Byron

    on Apr 8, 2015

    Member

    More actions

    It turned out that the index is not actually corrupted, which is good news. What happens is that git writes TREE extension data into the index, which causes it to write out the given tree as is next time a git commit is executed. When using git add, this extension data is maintained automatically. However, GitPython doesn't do that ... . Usually this is no problem at all, as you are supposed to use IndexFile.commit(...) along with IndexFile.add(...).

    Thanks to a shortcoming in the GitPython API, the index was automatically written out whenever files have been added, without providing control over whether or not extension data will be written along with it.

    My fix consists of an additional flag in IndexFile.add(...), which causes extension data not to be written by default, so commits can be safely done via git commit or IndexFile.commit(...).

    However, this might introduce new subtle bugs in case someone is relying on extension data to be written. As this can be controlled through the said flag though, a fix is easily done in that case.

    How to apply the fix to your codebase

    • Use GitPython v0.3.7 (due for release this week)
    • use git add instead of IndexFile.add(...)
    • use IndexFile.commit(...) instead of git commit

    Unfortunately, there is no other way to make it work with older versions of GitPython, as there is yet another bug in IndexFile.write(...) which would cause extension data to written even if that was explicitly disabled. It is fixed in v0.3.7 .

    YouTube

    The development stream can be watched on youtube

    • Session 1: Figure out what the problem is
    • Session 2: Finalize it
12. Byron

    added a commit that references this issue on Apr 8, 2015

    test(index): test for #265

    1fd07a0
13. Byron

    closed this as completedin 8f043d8on Apr 8, 2015
14. voxik commented on Apr 8, 2015

    

    voxik

    on Apr 8, 2015

    Author

    More actions

    Wow, thanks for fixing this. Hope it will be fixed in Fedora soon (right @pbabinca ? 😉 )

    And thanks especially for the video with analysis. Shame I don't have time to watch it (I watched just first 5 minutes). It would be definitely enlightening. BTW the "Session 2" video is private ;)
15. Byron commented on Apr 8, 2015

    

    Byron

    on Apr 8, 2015

    Member

    More actions

    Hehe, it's only the most-hardcore people who have the time and are willing to watch me do my thing for two hours ... no one will do that, yet it's good to archive the process of fixing issues just in case I will be wondering why the heck I fixed it the way I did :).

    The second video just wasn't ready yet - it works now, and youtube probably said it's still being processed when you tried it.
16. bewest

    added a commit that references this issue on Oct 29, 2015

    reduce git corruption issues

    b03492c