This is a reimplementation of Karpathy's scalar-valued autograd engine for micrograd as a Python C extension. There's about a 9x speedup overall during the training of a MLP on the two moons dataset. See the micrograd repository and Karpathy's backpropagation explanation video for more information. Because micrograd's engine is nicely self-contained and isolated from the Python neural network code, micrograd's engine is easily replaced with a compiled shared object file, as shown in the demo below:

This is micrograd's original engine in pure Python, about a 100 LOC Value engine class object:

This is the Python C extension version, the Value engine reimplemented in C in about 500 LOC:

1. Clone https://github.com/karpathy/micrograd
2. Build engine.c:

pip install -r requirements.txt
python setup.py build_ext --inplace
python -m pytest test.py

3. Replace https://github.com/karpathy/micrograd/blob/master/micrograd/engine.py with the built shared object file (as in the above demo)

I found a good introduction to the Python C API and the reference counting semantics to be Sam Gross's nogil talk. Sam Gross also makes a good point about using reversible debuggers (rr) which I found helpful, as well as Valgrind (while this extension works on any OS, Valgrind is Linux-only).

MIT