Part 11 served a model that fit on one machine. Training one that doesn’t is where the parallelism vocabulary comes from: data, tensor and pipeline parallelism, ZeRO, FSDP. Each of those terms answers the same question, which is where the bytes go.
I measured what fits on a laptop: one training step on its GPU, gradients cast to 16 bits and up to eight processes on its CPU splitting one model between them. The cluster-sized numbers below are arithmetic on the same rules.
Count the bytes before the GPUs
Training with Adam keeps four numbers per parameter: the weight, its gradient and the optimizer’s two running averages from Part 9. In fp32 that is 4 bytes each, 16 bytes a parameter. I built an 85.2-million-parameter GPT, took one step and added up the tensors. They came to 1.36 GB, 16.0 bytes a parameter. The GPU’s allocator agreed within 2%.
Mixed precision leaves that number at 16. Under PyTorch’s autocast with bf16, the same model kept 16.0 bytes a parameter. The weights stay in fp32; autocast runs only the operations on its list in 16 bits, the matrix multiplications among them, and keeps norms, softmax and the loss in fp32. The usual large-scale recipe keeps a bf16 working copy plus the fp32 master copy the optimizer updates and comes to the same 16, or 18 in configurations of Megatron, NVIDIA’s training library, that keep the gradients in fp32. Only pure bf16, weights and optimizer state included, halved it to 8.0. That recipe is a numerical risk, since bf16 stores 7 bits of mantissa and small updates round away.
So a 7-billion-parameter model needs 112 GB for its training state before it reads a single token. No 80 GB GPU holds that.
Activations grow with the batch
The other half is what the forward pass keeps for backward: Part 8’s graph, every intermediate the gradient will need. I measured it on the same model at 1,024 tokens a sequence. One sequence kept 1.29 GB, four kept 4.86, close to linear. Half of that, 2.42 GB, was attention matrices: one 1,024 × 1,024 matrix per head, per layer and per sequence, in fp32. Mixed precision left them alone. On this GPU PyTorch’s attention ran unfused, one operation at a time, and saved that matrix in fp32 even under autocast. Autocast cut the rest by 34%, from 2.44 GB to 1.61, and added 170 MB of bf16 weight copies, so the four-sequence total only fell to 4.21 GB.
Activation recompute, Part 8’s activation checkpointing, keeps only each block’s input and runs the block again during backward. It cut the fp32 figure to 178 MB, a 27th of the 4.86 GB. You pay with a second forward pass.
Korthikanti and others give the 16-bit activations of one layer as roughly s·b·h·(34 + 5as/h) bytes. Here s is the sequence length, b the batch, h the width and a the number of heads; the second term is the attention matrices. Over this model’s 12 layers and four sequences it comes to 4.30 GB. That’s near the 4.21 measured, though the terms differ. Three-fifths of the attention term is dropout, which this model doesn’t use. What’s left is the softmax output, 1.21 GB in 16 bits and exactly half the 2.42 this GPU kept in fp32. Scale it to a 7B model at 8,192 tokens and you get 380 GB for one sequence, about 170 without dropout. A fused attention kernel that never stores the attention matrix brings the 380 GB down to 37, and full recompute to about 2.
bf16 and fp16 trade range for precision
Mixed precision has two 16-bit formats, and the difference shows up in the gradients. In fp16 you get 10 stored bits of mantissa and nothing below about 6e-8. The bf16 format keeps fp32’s exponent range and stores only 7 bits of mantissa.
I took one batch of real gradients from the 85M model and cast them. In fp16, 23,046 of 85.2 million became zero, and 20% landed below 6e-5, in the subnormal range, where fp16 gives up mantissa bits to reach smaller numbers. Multiplied by 65,536 before the cast, which is what multiplying the loss does to every gradient, none was lost and none overflowed. I scaled finished gradients. In real fp16 training the gradients flowing backward between layers can underflow too, and scaling the loss protects them as well.
That multiplication is loss scaling. GradScaler does it and divides the scale back out before the optimizer step; when a step produces an inf or a NaN, it skips the step and halves the scale. In bf16 no gradient became zero at any scale; its fp32 range is why bf16 needs no scaler.
Data parallelism copies everything; ZeRO splits it
Data parallelism is the obvious way to add GPUs. You give every GPU the whole model and a slice of the batch, and the GPUs average their gradients before the step with an all-reduce, a collective that leaves every GPU holding the same sum. PyTorch calls it DDP. It divides the compute and leaves every byte of state copied. ZeRO, and PyTorch’s FSDP after it, keep the same arithmetic and shard the state. ZeRO’s three stages shard the optimizer state, then the gradients too, then the weights; FSDP is the third. Each GPU gathers what it doesn’t own only while a layer needs it.
I ran DDP, ZeRO-1 and FSDP with up to eight processes on this laptop’s CPU, a 4.8-million-parameter model and the same global batch of 64 sequences each time. Every fp32 variant at every process count ended at the same loss on the third step, 3.60987. These are the same computation, laid out differently.
state per process
weights 19.1 MB gradients 19.1 MB Adam 38.3 MB
traffic per process, one step
- state per process
- 76.5 MB
- of a full copy
- 100.0%
- handed to collectives
- 0.0 MB
- sent on a ring
- 0.0 MB
- loss at step 3
- 3.60987
Start on DDP and drag the process count up: each process keeps the full 76.5 MB of weights, gradients and Adam state however many there are, which is what copying everything means. ZeRO-1, PyTorch’s ZeroRedundancyOptimizer, shards only the Adam state, and at eight processes keeps 43.5 MB. FSDP shards everything and keeps 9.6 MB, an eighth.
The bytes come back as traffic. DDP all-reduces the gradients once a step, 19.1 MB. FSDP all-gathers each block’s weights before forward and again before backward, so every process collects the slices it doesn’t own. Then it reduce-scatters the gradients, and each process ends up with the sum of its own slice only. That’s about 57 MB handed to its collectives, three times DDP’s figure. The wire sees a smaller gap. A ring all-reduce, where each process passes chunks only to its neighbor, is a reduce-scatter followed by an all-gather, so each process sends nearly twice its input. At eight processes a ring would send 33.5 MB per process each step under DDP and 50 under FSDP, 1.5 times as much. The FSDP2 docstring names the trade in one sentence:
# torch 2.14.1, torch/distributed/fsdp/_fully_shard/_fully_shard.py:110-112"""Apply fully sharded data parallelism (FSDP) to ``module``, where FSDPshards module parameters, gradients, and optimizer states across dataparallel workers to save memory at the cost of communication."""Pick FSDP with bf16 parameters and you halve the traffic while the state stays at 9.6 MB. MixedPrecisionPolicy casts the copy it gathers; its docstring adds that “the optimizer step uses the sharded parameter in the original dtype” (_fsdp_api.py:30-31).
ZeRO-1 has a quirk you should know about. The ZeRO paper keeps stage 1 at plain data parallelism’s traffic. PyTorch’s version all-reduces the gradients and then broadcasts every updated shard (zero_redundancy_optimizer.py:1151), which also comes to 1.5 times. It hands out whole tensors, so at eight processes the busiest one held 5.2 MB of Adam state. An even split would have given each 4.8.
With N GPUs, a ring all-reduce sends 2(N − 1)/N times the gradient size from each GPU. That approaches twice the gradient size and stops growing as GPUs are added. For 7B of bf16 gradients that’s 14 GB at two GPUs, 24.5 at eight and 28 at a thousand. Each GPU added is another hop around the ring, with its own latency. At a fixed global batch, adding GPUs also shrinks each one’s compute, so the wire takes a larger share of every step; grow the batch with them and the share holds.
The bytes that cross between machines are the slow ones. Inside an H100 server NVLink moves 450 GB/s each way per GPU; across servers a 400 Gb/s network card moves 50 GB/s, a ninth of that.
Tensor and pipeline parallelism split what ZeRO can’t
ZeRO’s data-parallel stages shard state. None of them shards activations: each GPU still runs whole layers on its slice of the batch. When one sequence’s activations don’t fit, or the communication of sharding gets too expensive, the model itself is cut.
Tensor parallelism splits each weight matrix across GPUs. Every GPU computes part of every layer. Megatron pays two all-reduces of activations in the forward pass of each layer and two in the backward. That much traffic needs the fast links inside one server, so Megatron’s authors keep tensor parallelism within the GPUs of a server, eight on their DGX A100 servers.
Pipeline parallelism gives each GPU a run of consecutive layers and passes activations along. It splits each batch into micro-batches, so different stages can work on different ones at once. Its cost is idle time while the pipeline fills and drains. With p stages and m micro-batches, GPipe’s schedule, which runs every forward before any backward, leaves (p − 1)/(m + p − 1) of the time unused. That’s 43% with four stages and four micro-batches, 10% with eight and sixty-four.
Pipelining divides the state. It divides the activations less than you’d hope. The usual schedule, one forward then one backward (1F1B), has the same idle time and fewer micro-batches in flight. Even so, its first stage holds p micro-batches for its share of the layers, a whole model’s activations for one micro-batch.
- GPUs
- 1
- state per GPU
- 112 GB
- activations
- 37 GB
- total per GPU
- 149 GB
- on 80 GB
- does not fit
Pick the 405B model. Its 6.48 TB of training state needs 81 GPUs of 80 GB just to exist, and plain data parallelism puts all of it on every one. Llama 3 trained that model on 16,384 GPUs with 8,192-token sequences: tensor parallelism of 8 inside each server, 16 pipeline stages and 128 data-parallel groups on top, which multiply out to exactly that count. Sequence parallelism split the parts tensor parallelism leaves whole along the sequence. Set those three and each GPU needs 122 GB under DDP. Tensor and pipeline parallelism take the state from 6.48 TB to about 51 GB a GPU, and the activations add just under 72. Sharding the optimizer state across the 128 brings the total to 85 GB. Shard the gradients too and you get 79. That’s just under the 80 GB line. Korthikanti counted a GPT-3 layer, and Llama’s layers differ from it by about a tenth, so at 79 GB the details decide whether you fit. The card itself holds 80 GiB, about 86 GB, and CUDA and communication buffers take a few of those. The 128 groups are there for compute, because at 400 trillion operations a second per GPU, 81 GPUs would need about 37 years for the 3.8e25 operations of the 405B run.
What the ML engineer watches
Four habits; three of them need nothing but a calculator.
Multiply before you reserve. Parameters times 16 bytes is the training state with Adam, mixed precision or not. Divide by the GPUs only if something shards it.
Size the activations from the batch. They grow with batch times sequence length, and an unfused attention adds one matrix per head that grows with the square of the length. Use a fused kernel first. Mixed precision cut the rest by a third here, and recompute trades a second forward for most of what’s left.
Pick the split by the bottleneck. State doesn’t fit: shard it with ZeRO or FSDP. If one sequence’s activations don’t fit, you recompute them or split each layer with tensor parallelism inside a server, where the links are fast. If the model still outgrows a server, pipeline stages across servers divide its training state, and leave the first stage holding about a whole model’s worth of activations. Everything fits but it’s slow: add data parallelism on top.
Count the bytes on the wire. Sharding trades memory for traffic, 1.5 times plain data parallelism here. Per GPU that traffic stops growing as you add GPUs; at a fixed global batch each GPU’s compute keeps shrinking. Measure the link between machines before you plan for more of them.
Most of the sizing is arithmetic: one multiplication for the state, a ring formula for the traffic and a product of three numbers for the GPU count. That closes the series. It started in Part 1 with a tokenizer turning text into ids, and it ends with 16,384 GPUs splitting one model between them.