From code to weights:
ML fundamentals for engineers
You already know what a lookup table is, what a pipeline costs, what a type system buys you. Moving into ML engineering does not replace that knowledge; it relocates it. Each part takes one concept, explains it from the engineering side, reads one real file, and names the single thing that does not behave the way software behaves.
Next · Part 12 on
Start here What this series assumes And what it deliberately does not, and where to go first if something is missing.Read in order
ML fundamentals
A model doesn't read text: what a tokenizer decides for you
The tokenizer ships inside the model, not in front of it. Reading tiktoken's BPE for what it fixes before any weight exists: vocabulary, cost, context limit.
ML fundamentals
An embedding is a lookup table, and everything else is how you fill it
A row of a matrix indexed by id. The science is how it gets filled; the engineering is that there is no schema migration: a new embedder invalidates your index.
ML fundamentals
Shapes are the type system of ML, and nobody checks them for you
[B, T, d] is a function signature. The runtime never verifies it, so a wrong axis raises only when sizes differ; when they collide, it broadcasts, returns plausible numbers, and ships.
ML fundamentals
Attention is a learned weighted average, and the cost is in the square
A soft dictionary whose weights the model computes from its own input. The idea is cheap; the weight matrix is T by T, so the cost comes from the data, not the code.
ML fundamentals
A transformer block is six tensors and a bus: read it like a pipeline
Six tensors on a residual stream, read line by line with shapes in the margin. Two thirds of each block's weights sit in the feed-forward; as the context grows, the cost moves to attention.
ML fundamentals
GPT-2 is the transformer plus five decisions, and inference is a different program
One artifact, two programs. Training is one parallel forward pass over the sequence; generation is a sequential loop with a cache. Same weights, different bugs.
ML fundamentals
From GPT-2 to Llama: four changes and the reason for each
Rotary positions, RMSNorm, gated feed-forward, grouped-query attention. Four local edits, justified by measurement more than theory, and each one baked into the weights.
ML fundamentals
Loss and gradients without the formulas: backprop is a graph walk
Backprop is a graph walked backwards, applying the chain rule. Read as code, it explains vanishing and exploding gradients, and why a gradient bug shows up as a slightly wrong curve, not a stack trace.
ML fundamentals
The learning rate is the hyperparameter that breaks production
Adam and the schedule are engineering decisions, not mathematics. Nothing validates the config: a bad value diverges thirty hours in and takes the run with it.
ML fundamentals
Fine-tuning is the wrong tool for knowledge: three training phases and the low-rank shortcut
Pre-training and fine-tuning share a loss and differ in a mask; LoRA trains two narrow matrices beside the frozen weights. Facts taught by fine-tuning go in slowly and bring unearned confidence: mine named Moscow as the capital of a country that does not exist.
ML fundamentals
Serving is a memory problem: prefill, decode and the KV cache
Decode reads every weight for every token, so serving runs out of memory bandwidth long before compute. On my laptop, reading the weights took 84% of each one-sequence step, and a hybrid model paid 5.5 ms per extra sequence where the arithmetic predicted under 0.5. Near capacity the wait for a first token ran to seconds; at half load its 99th percentile moved more than tenfold between runs at nearly the same rate.
ML fundamentals
At scale the memory decides: mixed precision and the three parallelisms
Where the bytes of a training run go, measured on a laptop: mixed precision still kept 16 bytes a parameter, half the activations were attention matrices and FSDP cut the state per process eightfold for 1.5 times the traffic of plain data parallelism.