Echo Uses Error Compensation for 4-Bit Tensor-Parallel LLM Communication

Summary

Tensor parallelism places an all-reduce between dependent matrix multiplications, making communication difficult to hide as model computation outpaces interconnect bandwidth. Echo replaces 16-bit communication with 4-bit transfers, which can reduce the communicated data by nearly four times, while summing in high precision. Each value is quantized at most twice per call regardless of the number of GPUs. Instead of treating rounding errors independently, the method feeds the error revealed during summation into later calls so previous errors can cancel. The authors prove that accumulated error remains bounded independently of network depth. On A100 GPUs, Echo is 2.09 times faster than the 16-bit NCCL all-reduce for 1 GB messages. With an idle interconnect, it improves time to first token by up to 1.16 times and training-step time by up to 1.09 times; under increasing background traffic, the gains reach 1.44 times and 1.38 times. On Qwen3-8B, average accuracy across seven benchmarks stays within 0.4 points of 16-bit inference, while 10B-token pretraining ends with a loss less than 0.9% above 16-bit training. Echo does not require FP8 or FP4 hardware.