ML fundamentals
What a model is made of (tokenizers, embeddings, tensors, attention, training, serving), read as an engineer, for people arriving from software or data science.
← All writingFine-tuning is the wrong tool for knowledge: three training phases and the low-rank shortcut
Pre-training and fine-tuning share a loss and differ in a mask; LoRA trains two narrow matrices beside the frozen weights. Facts taught by fine-tuning go in slowly and bring unearned confidence: mine named Moscow as the capital of a country that does not exist.
The learning rate is the hyperparameter that breaks production
Adam and the schedule are engineering decisions, not mathematics. Nothing validates the config: a bad value diverges thirty hours in and takes the run with it.
Loss and gradients without the formulas: backprop is a graph walk
Backprop is a graph walked backwards, applying the chain rule. Read as code, it explains vanishing and exploding gradients, and why a gradient bug shows up as a slightly wrong curve, not a stack trace.
From GPT-2 to Llama: four changes and the reason for each
Rotary positions, RMSNorm, gated feed-forward, grouped-query attention. Four local edits, justified by measurement more than theory, and each one baked into the weights.
GPT-2 is the transformer plus five decisions, and inference is a different program
One artifact, two programs. Training is one parallel forward pass over the sequence; generation is a sequential loop with a cache. Same weights, different bugs.
A transformer block is six tensors and a bus: read it like a pipeline
Six tensors on a residual stream, read line by line with shapes in the margin. Two thirds of each block's weights sit in the feed-forward; as the context grows, the cost moves to attention.
Attention is a learned weighted average, and the cost is in the square
A soft dictionary whose weights the model computes from its own input. The idea is cheap; the weight matrix is T by T, so the cost comes from the data, not the code.
Shapes are the type system of ML, and nobody checks them for you
[B, T, d] is a function signature. The runtime never verifies it, so a wrong axis raises only when sizes differ; when they collide, it broadcasts, returns plausible numbers, and ships.
An embedding is a lookup table, and everything else is how you fill it
A row of a matrix indexed by id. The science is how it gets filled; the engineering is that there is no schema migration: a new embedder invalidates your index.
A model doesn't read text: what a tokenizer decides for you
The tokenizer ships inside the model, not in front of it. Reading tiktoken's BPE for what it fixes before any weight exists: vocabulary, cost, context limit.