📅 Aug 15, 2026
⏱️ 6 min read
Megatron-LM: Tensor Parallelism vs. 1F1B Pipeline Parallelism Communication Overhead
Comprehensive breakdown of All-Reduce vs P2P communication volume in Megatron-LM 3D Parallelism, comparing ring-allreduce latency on NVLink vs InfiniBand.
#Megatron-LM
#Distributed Training
#Tensor Parallelism
#NVLink
#RDMA
📖 Book Cross-Ref: Chapter 2: 3D Parallelism & Training Infrastructure
By Fuheng Wu
Read Article →