This project aims at finding the better distributed training architecture between Parameter Server (Standard distributed TensorFlow) and Ring Allreduce (Horovod).