[ICLR 2018] Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

vector quantization for stochastic gradient descent.

[CCS 2021] "DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation" by Boxin Wang*, Fan Wu*, Yunhui Long*, Luka Rimanic, Ce Zhang, Bo Li

Simple Implementation of the CVPR 2024 Paper "JointSQ: Joint Sparsification-Quantization for Distributed Learning"

We present a set of all-reduce compatible gradient compression algorithms which significantly reduce the communication overhead while maintaining the performance of vanilla SGD. We empirically evaluate the performance of the compression methods by training deep neural networks on the CIFAR10 dataset.

Geometric median (GM) is a classical method in statistics for achieving a robust estimation of the uncorrupted data; under gross corruption, it achieves the optimal breakdown point of 0.5. However, its computational complexity makes it infeasible for robustifying stochastic gradient descent (SGD) for high-dimensional optimization problems. In th…

😂Distributed optimizer implemented with TensorFlow MPI operation

GPU memory-efficient training for PyTorch - 90%+ memory savings through gradient compression

无线感知联邦学习仿真：信道感知调度、梯度压缩、差分隐私与可复现实验评测

Step-Ahead Partial Error Feedback (SA-PEF) for communication-efficient federated learning. PyTorch + Flower implementation.