ML fundamentals

At scale the memory decides: mixed precision and the three parallelisms

Where the bytes of a training run go, measured on a laptop: mixed precision still kept 16 bytes a parameter, half the activations were attention matrices and FSDP cut the state per process eightfold for 1.5 times the traffic of plain data parallelism.

New to the series? Start here — what it assumes, and what it doesn't.

Part 11 served a model that fit on one machine. Training one that doesn’t is where the parallelism vocabulary comes from: data, tensor and pipeline parallelism, ZeRO, FSDP. Each of those terms answers the same question, which is where the bytes go.

I measured what fits on a laptop: one training step on its GPU, gradients cast to 16 bits and up to eight processes on its CPU splitting one model between them. The cluster-sized numbers below are arithmetic on the same rules.

Count the bytes before the GPUs

Training with Adam keeps four numbers per parameter: the weight, its gradient and the optimizer’s two running averages from Part 9. In fp32 that is 4 bytes each, 16 bytes a parameter. I built an 85.2-million-parameter GPT, took one step and added up the tensors. They came to 1.36 GB, 16.0 bytes a parameter. The GPU’s allocator agreed within 2%.

Mixed precision leaves that number at 16. Under PyTorch’s autocast with bf16, the same model kept 16.0 bytes a parameter. The weights stay in fp32; autocast runs only the operations on its list in 16 bits, the matrix multiplications among them, and keeps norms, softmax and the loss in fp32. The usual large-scale recipe keeps a bf16 working copy plus the fp32 master copy the optimizer updates and comes to the same 16, or 18 in configurations of Megatron, NVIDIA’s training library, that keep the gradients in fp32. Only pure bf16, weights and optimizer state included, halved it to 8.0. That recipe is a numerical risk, since bf16 stores 7 bits of mantissa and small updates round away.

So a 7-billion-parameter model needs 112 GB for its training state before it reads a single token. No 80 GB GPU holds that.

Activations grow with the batch

The other half is what the forward pass keeps for backward: Part 8’s graph, every intermediate the gradient will need. I measured it on the same model at 1,024 tokens a sequence. One sequence kept 1.29 GB, four kept 4.86, close to linear. Half of that, 2.42 GB, was attention matrices: one 1,024 × 1,024 matrix per head, per layer and per sequence, in fp32. Mixed precision left them alone. On this GPU PyTorch’s attention ran unfused, one operation at a time, and saved that matrix in fp32 even under autocast. Autocast cut the rest by 34%, from 2.44 GB to 1.61, and added 170 MB of bf16 weight copies, so the four-sequence total only fell to 4.21 GB.

Activation recompute, Part 8’s activation checkpointing, keeps only each block’s input and runs the block again during backward. It cut the fp32 figure to 178 MB, a 27th of the 4.86 GB. You pay with a second forward pass.

Korthikanti and others give the 16-bit activations of one layer as roughly s·b·h·(34 + 5as/h) bytes. Here s is the sequence length, b the batch, h the width and a the number of heads; the second term is the attention matrices. Over this model’s 12 layers and four sequences it comes to 4.30 GB. That’s near the 4.21 measured, though the terms differ. Three-fifths of the attention term is dropout, which this model doesn’t use. What’s left is the softmax output, 1.21 GB in 16 bits and exactly half the 2.42 this GPU kept in fp32. Scale it to a 7B model at 8,192 tokens and you get 380 GB for one sequence, about 170 without dropout. A fused attention kernel that never stores the attention matrix brings the 380 GB down to 37, and full recompute to about 2.

bf16 and fp16 trade range for precision

Mixed precision has two 16-bit formats, and the difference shows up in the gradients. In fp16 you get 10 stored bits of mantissa and nothing below about 6e-8. The bf16 format keeps fp32’s exponent range and stores only 7 bits of mantissa.

I took one batch of real gradients from the 85M model and cast them. In fp16, 23,046 of 85.2 million became zero, and 20% landed below 6e-5, in the subnormal range, where fp16 gives up mantissa bits to reach smaller numbers. Multiplied by 65,536 before the cast, which is what multiplying the loss does to every gradient, none was lost and none overflowed. I scaled finished gradients. In real fp16 training the gradients flowing backward between layers can underflow too, and scaling the loss protects them as well.

That multiplication is loss scaling. GradScaler does it and divides the scale back out before the optimizer step; when a step produces an inf or a NaN, it skips the step and halves the scale. In bf16 no gradient became zero at any scale; its fp32 range is why bf16 needs no scaler.

Data parallelism copies everything; ZeRO splits it

Data parallelism is the obvious way to add GPUs. You give every GPU the whole model and a slice of the batch, and the GPUs average their gradients before the step with an all-reduce, a collective that leaves every GPU holding the same sum. PyTorch calls it DDP. It divides the compute and leaves every byte of state copied. ZeRO, and PyTorch’s FSDP after it, keep the same arithmetic and shard the state. ZeRO’s three stages shard the optimizer state, then the gradients too, then the weights; FSDP is the third. Each GPU gathers what it doesn’t own only while a layer needs it.

I ran DDP, ZeRO-1 and FSDP with up to eight processes on this laptop’s CPU, a 4.8-million-parameter model and the same global batch of 64 sequences each time. Every fp32 variant at every process count ended at the same loss on the third step, 3.60987. These are the same computation, laid out differently.

One training step of the same model, split across processes on this laptop. Pick a way of splitting and drag the number of processes. The top bar is what one process keeps, against a full copy; the bottom bar is what it sends on a ring each step, against the most any setting sent.

state per process

weights 19.1 MB gradients 19.1 MB Adam 38.3 MB

traffic per process, one step

state per process
76.5 MB
of a full copy
100.0%
handed to collectives
0.0 MB
sent on a ring
0.0 MB
loss at step 3
3.60987

Start on DDP and drag the process count up: each process keeps the full 76.5 MB of weights, gradients and Adam state however many there are, which is what copying everything means. ZeRO-1, PyTorch’s ZeroRedundancyOptimizer, shards only the Adam state, and at eight processes keeps 43.5 MB. FSDP shards everything and keeps 9.6 MB, an eighth.

The bytes come back as traffic. DDP all-reduces the gradients once a step, 19.1 MB. FSDP all-gathers each block’s weights before forward and again before backward, so every process collects the slices it doesn’t own. Then it reduce-scatters the gradients, and each process ends up with the sum of its own slice only. That’s about 57 MB handed to its collectives, three times DDP’s figure. The wire sees a smaller gap. A ring all-reduce, where each process passes chunks only to its neighbor, is a reduce-scatter followed by an all-gather, so each process sends nearly twice its input. At eight processes a ring would send 33.5 MB per process each step under DDP and 50 under FSDP, 1.5 times as much. The FSDP2 docstring names the trade in one sentence:

# torch 2.14.1, torch/distributed/fsdp/_fully_shard/_fully_shard.py:110-112
"""Apply fully sharded data parallelism (FSDP) to ``module``, where FSDP
shards module parameters, gradients, and optimizer states across data
parallel workers to save memory at the cost of communication."""

Pick FSDP with bf16 parameters and you halve the traffic while the state stays at 9.6 MB. MixedPrecisionPolicy casts the copy it gathers; its docstring adds that “the optimizer step uses the sharded parameter in the original dtype” (_fsdp_api.py:30-31).

ZeRO-1 has a quirk you should know about. The ZeRO paper keeps stage 1 at plain data parallelism’s traffic. PyTorch’s version all-reduces the gradients and then broadcasts every updated shard (zero_redundancy_optimizer.py:1151), which also comes to 1.5 times. It hands out whole tensors, so at eight processes the busiest one held 5.2 MB of Adam state. An even split would have given each 4.8.

With N GPUs, a ring all-reduce sends 2(N − 1)/N times the gradient size from each GPU. That approaches twice the gradient size and stops growing as GPUs are added. For 7B of bf16 gradients that’s 14 GB at two GPUs, 24.5 at eight and 28 at a thousand. Each GPU added is another hop around the ring, with its own latency. At a fixed global batch, adding GPUs also shrinks each one’s compute, so the wire takes a larger share of every step; grow the batch with them and the share holds.

The bytes that cross between machines are the slow ones. Inside an H100 server NVLink moves 450 GB/s each way per GPU; across servers a 400 Gb/s network card moves 50 GB/s, a ninth of that.

Tensor and pipeline parallelism split what ZeRO can’t

ZeRO’s data-parallel stages shard state. None of them shards activations: each GPU still runs whole layers on its slice of the batch. When one sequence’s activations don’t fit, or the communication of sharding gets too expensive, the model itself is cut.

Tensor parallelism splits each weight matrix across GPUs. Every GPU computes part of every layer. Megatron pays two all-reduces of activations in the forward pass of each layer and two in the backward. That much traffic needs the fast links inside one server, so Megatron’s authors keep tensor parallelism within the GPUs of a server, eight on their DGX A100 servers.

Pipeline parallelism gives each GPU a run of consecutive layers and passes activations along. It splits each batch into micro-batches, so different stages can work on different ones at once. Its cost is idle time while the pipeline fills and drains. With p stages and m micro-batches, GPipe’s schedule, which runs every forward before any backward, leaves (p − 1)/(m + p − 1) of the time unused. That’s 43% with four stages and four micro-batches, 10% with eight and sixty-four.

Pipelining divides the state. It divides the activations less than you’d hope. The usual schedule, one forward then one backward (1F1B), has the same idle time and fewer micro-batches in flight. Even so, its first stage holds p micro-batches for its share of the layers, a whole model’s activations for one micro-batch.

Memory per GPU for a training run at 8,192 tokens and one sequence per GPU. Each recipe lists the bytes a parameter for weights, gradients and Adam state. Activations follow Korthikanti's rule with a fused attention kernel, and the parts of each layer outside the split matrices are split along the sequence too (sequence parallelism), so tensor parallelism divides them; pipeline stages don't, because the first stage holds about a whole model's worth. The amber is training state and the grey activations. The line marks 80 GB, a budget inside an 80 GiB (86 GB) card that leaves a few GB for CUDA and communication buffers. The numbers are arithmetic on the rules measured above.
GPUs
1
state per GPU
112 GB
activations
37 GB
total per GPU
149 GB
on 80 GB
does not fit

Pick the 405B model. Its 6.48 TB of training state needs 81 GPUs of 80 GB just to exist, and plain data parallelism puts all of it on every one. Llama 3 trained that model on 16,384 GPUs with 8,192-token sequences: tensor parallelism of 8 inside each server, 16 pipeline stages and 128 data-parallel groups on top, which multiply out to exactly that count. Sequence parallelism split the parts tensor parallelism leaves whole along the sequence. Set those three and each GPU needs 122 GB under DDP. Tensor and pipeline parallelism take the state from 6.48 TB to about 51 GB a GPU, and the activations add just under 72. Sharding the optimizer state across the 128 brings the total to 85 GB. Shard the gradients too and you get 79. That’s just under the 80 GB line. Korthikanti counted a GPT-3 layer, and Llama’s layers differ from it by about a tenth, so at 79 GB the details decide whether you fit. The card itself holds 80 GiB, about 86 GB, and CUDA and communication buffers take a few of those. The 128 groups are there for compute, because at 400 trillion operations a second per GPU, 81 GPUs would need about 37 years for the 3.8e25 operations of the 405B run.

What the ML engineer watches

Four habits; three of them need nothing but a calculator.

Multiply before you reserve. Parameters times 16 bytes is the training state with Adam, mixed precision or not. Divide by the GPUs only if something shards it.

Size the activations from the batch. They grow with batch times sequence length, and an unfused attention adds one matrix per head that grows with the square of the length. Use a fused kernel first. Mixed precision cut the rest by a third here, and recompute trades a second forward for most of what’s left.

Pick the split by the bottleneck. State doesn’t fit: shard it with ZeRO or FSDP. If one sequence’s activations don’t fit, you recompute them or split each layer with tensor parallelism inside a server, where the links are fast. If the model still outgrows a server, pipeline stages across servers divide its training state, and leave the first stage holding about a whole model’s worth of activations. Everything fits but it’s slow: add data parallelism on top.

Count the bytes on the wire. Sharding trades memory for traffic, 1.5 times plain data parallelism here. Per GPU that traffic stops growing as you add GPUs; at a fixed global batch each GPU’s compute keeps shrinking. Measure the link between machines before you plan for more of them.

Most of the sizing is arithmetic: one multiplication for the state, a ring formula for the traffic and a product of three numbers for the GPU count. That closes the series. It started in Part 1 with a tokenizer turning text into ids, and it ends with 16,384 GPUs splitting one model between them.

ML fundamentalsFrom code to weights