1. Tensor Parallelism (TP) Communication Cost
In Megatron-LM Tensor Parallelism (Megatron-TP), each Transformer layer consists of:
1. ColumnParallelLinear in Self-Attention ($Q, K, V$ projections) followed by RowParallelLinear (Output projection).
2. ColumnParallelLinear in MLP (Gate/Up projections) followed by RowParallelLinear (Down projection).
An All-Reduce collective communication occurs at the output of every RowParallelLinear layer. For hidden dimension $h$, sequence length $s$, and micro-batch size $b$, the total data transferred per Transformer layer in TP of size $p$ is:
$$ \text{Comm}_{\text{TP}} = 4 \times \left( \frac{2(p-1)}{p} \right) \cdot b \cdot s \cdot h \cdot \text{sizeof}(\text{dtype}) $$
Because of the high communication frequency (twice per layer per forward and backward pass), Tensor Parallelism must reside within a single node connected by NVLink/NVSwitch (e.g. 900 GB/s per GPU on H100).
2. 1F1B Pipeline Parallelism (PP) Schedule
For scale-out across multiple server nodes across InfiniBand (IB) / RoCE networks, Pipeline Parallelism (PP) partitions layers sequentially.
In the One-Forward-One-Backward (1F1B) scheduling scheme with $m$ micro-batches and $p$ pipeline stages:
$$ \text{Bubble Fraction} F = \frac{p - 1}{m} $$
When $m \ge 4p$, pipeline bubble overhead drops below 20%.
Stage 1: [F1][F2][F3][F4][B1][F5][B2][F6][B3]...
Stage 2: [F1][F2][F3][F4][B1][F5][B2]...
Stage 3: [F1][F2][F3][F4][B1]...
Stage 4: [F1][F2][F3][B1]...
3. Summary Comparison Table
| Parallelism Strategy | Comm Pattern | Bandwidth Requirement | Scaling Boundary |
|---|---|---|---|
| Tensor Parallel (TP) | Ring All-Reduce | $\ge 400\text{--}900\text{ GB/s}$ | Intra-node (NVLink, 8 GPUs) |
| Pipeline Parallel (PP) | P2P Point-to-Point | $\ge 50\text{--}100\text{ GB/s}$ | Inter-node (RDMA/RoCE) |
| Data Parallel (ZeRO-3) | All-Gather + Reduce-Scatter | $\ge 100\text{--}200\text{ GB/s}$ | Inter-node cluster |
For more mathematical proofs and NCCL performance traces, refer to Chapter 2 of Distributed AI Systems.