Fine-tuning is the wrong tool for knowledge: three training phases and the low-rank shortcut

Part 9 trained a model from nothing. That is not how this work usually starts. You start from weights someone else trained and change them. Four words describe the job as if they were four different machines: pre-training, fine-tuning, RLHF, LoRA.

I read peft’s LoRA layer and trl’s DPO loss, then taught SmolLM2-135M fifteen capital cities: three it already knew and twelve belonging to countries that do not exist. It learned all of them. That is the problem.

Three phases, one model in training

Pre-training predicts the next token over everything you have. That is Part 8’s cross-entropy over a corpus, and it is where almost all of a model’s knowledge arrives.

Supervised fine-tuning, SFT, is the same loss on pairs of instruction and response. The usual change is which positions count: the prompt is masked out, as trl does by default for prompt-completion data. On one example of five prompt tokens and seven answer tokens, the whole sequence scored 2.9883 and the answer alone 2.4845, the same two numbers I got by hand from the logits. Only one array of labels differs: same binary, different config file.

Preference tuning, the third, has no right answer: each example is two answers and a note saying which one a person preferred.

By default all three update every weight of the model being trained, which preference tuning calls the policy, a reinforcement-learning term. What sets the third apart is how many other models it keeps around.

Preference tuning optimizes a difference

Reinforcement learning from human feedback, RLHF, trains a reward model to score answers and optimizes the policy against it. Its classic algorithm, PPO (proximal policy optimization), adds a critic that predicts how a partial answer will finally score; it also keeps a frozen reference, the model before this phase, and penalizes drifting from it. Four models for one update. Direct preference optimization, DPO, drops the reward model and the critic and keeps the reference. Here is its core, condensed from trl’s dpo_trainer.py:1445-1483:

chosen_logratios = chosen_logps - ref_chosen_logps        # how far the policy moved, on the good answer
rejected_logratios = rejected_logps - ref_rejected_logps  # and on the bad one
chosen_scores, rejected_scores = chosen_logratios, rejected_logratios   # the default divergence
delta_score = chosen_scores - rejected_scores
per_sequence_loss = -F.logsigmoid(self.beta * delta_score)

The loss wants the gap between those two movements to be large. beta, 0.1 by default, prices drifting from the reference: the larger it is, the less the policy moves.

The reference sounds like a second copy of the weights. With a LoRA policy, covered below, it isn’t. The trainable part is a small detour beside the frozen weights, and switching it off gives you the reference. Run that way, the policy’s log-probabilities matched a separately loaded frozen model to the last bit, before training and again after twenty steps. At step zero they are the same model, so the gap is zero and the loss is minus the log of a half: 0.693147, log 2 to six places.

I ran that arithmetic in a loop of my own, outside DPOTrainer. Twenty steps on two preference pairs, at a brutal rate of 0.001, took the gap from 0 to 95.4. The good answer became 2.9 nats more likely. The bad one became 92.5 nats less likely.

Part of that is built in. A log-probability can’t rise above zero, and the good answers started at -2.3 and -3.5, so 2.9 nats was nearly all their room. The bad ones were nine and twelve tokens long, with no floor. So I reran it on pairs of equal length, Paris against Lyon and 4 against 5, three times each way round. At the brutal rate the rejected side fell 31 to 47 nats every time; the preferred one moved anywhere from -1.2 to +4.2, and once ended less likely than it started.

At 0.00001, a rate someone would use, the split depends on where each answer started. With Paris and 4 preferred, they gained 0.01 to 0.02 nats and the others lost 0.16 to 0.17; with Lyon and 5 preferred, they gained 0.15 and the others lost 0.03. In these pairs the bigger move went to whichever answer started less likely. The loss asks for a difference and takes it from the cheaper side. Nothing in it says the preferred answer has to become likelier.

LoRA trains a different object

LoRA, low-rank adaptation, is a choice about what the update is, and you can make it in any of the three phases. You leave a weight matrix alone and add two small ones beside it; the pair is the adapter. Here is peft, condensed from lora/layer.py:263-1102:

self.lora_A[adapter_name] = nn.Linear(self.in_features, r, bias=False)  # d_in to r
self.lora_B[adapter_name] = nn.Linear(r, self.out_features, bias=lora_bias)  # r to d_out
self.scaling[adapter_name] = lora_alpha / r
nn.init.zeros_(self.lora_B[adapter_name].weight)
result = result + lora_B(lora_A(dropout(x))) * scaling

The rank r is how narrow the middle is. The detour is multiplied by lora_alpha / r before it’s added back: 2.0 with alpha 16 and rank 8.

The zeros on the last setup line are the trick. At step zero the product of the two matrices is zero, so the model is bit for bit the one you started with. Then training moves A and B. What they learn is a difference that multiplies out to a change of rank 8 on one of SmolLM2’s 576 by 576 matrices, rank in the linear algebra sense: how many independent directions the change has. Merged into the matrix, the adapter gave the same logits as when left beside it, to within 0.00008.

That rank is where the saving comes from. Adapting the query and value matrices of Llama-3.1-8B at rank 8 trains 3,407,872 numbers out of 8,030,261,248, four hundredths of a percent.

Drag the rank up: the total barely moves. The frozen base is almost all of it. Now switch the base to NF4 with double quantization, the 4-bit format from QLoRA, the recipe that trains LoRA over a 4-bit base. Nearly two thirds of the LoRA bar goes. Only the layers inside the blocks drop to just over four bits a weight; the embeddings and the output layer stay at sixteen.

Facts go in slowly, and confidence comes along

That is the mechanics. Here is the part people get wrong in planning meetings.

Start with twelve capitals of invented countries, all of the form “The capital of Vantressa is Molduin”, and twelve real ones in the same shape. The model completed three of the real ones word for word. Six of the other nine said “the city of Rome” and the like, which an exact match doesn’t count; all nine were dropped. Then fine-tune every weight on all fifteen at once, at 0.00001.

The three it knew sharpened fast: confidence in the first token of the right answer went from 22% to 97% in twenty steps. By then the twelve new ones were at 2%, with one answered correctly. By step forty it answered all twelve. Part of the slowness is where they started. On that first token the new facts began 8.6 nats away against 1.5, more than five times further, and each needed two or three tokens right where the known ones needed one. Gekhman and colleagues report the same pattern at scale: new knowledge fits significantly slower, and hallucination rises as it lands.

Answered means it completed the sentence it was trained on. Asked “What is the capital of Vantressa?”, it got none of the twelve. Asked which country Molduin is the capital of, none. “Vantressa’s capital city is” got six. It had learned a sentence.

A version of the rising hallucination showed up too. Six more country names, invented or borrowed from obscure places, stayed out of training. Before it, the model answered them as it did the real ones above, at around 24% confidence: five answers began “the town of” or “the city of”, and four of those went on to spell the country’s own name back. After a hundred steps, four of the six ended on a real city and a full stop. The capital of Volnaya is Moscow, at 82%. Quenby’s is Shillong, Dalmerie’s Edinburgh, and Rhuddlan’s Birmingham with the B missing.

Nothing told it anything about those places. Moscow was in the batch as Russia’s capital, but Volnaya also sounds Russian, and at step 10 the answer was Voronezh. Other guesses tracked the sound too: Maribor for Marovska, Cardiff for Rhuddlan. It learned that a sentence of this form ends in a city, and it often reached for one that sounded right.

Drag to step 20: the known facts are at 97.0%, the new ones at 1.9% with one of twelve recalled. Then drag across the run and watch the dotted line, the six untaught countries. It dips to 5.7% at step 10, as the model stops saying “the”, and ends at 41.1%, up from 23.7%. Two answers carry that rise: Moscow at 82%, and 89% on “Th”, the start of Threndale’s own name, which it finishes as “Thwapth”. The other four average 19%, down from 24%.

The patch nobody can read

A full fine-tune is a diff, so I counted it. Of 134,515,008 parameters, 134,513,152 changed. None moved enough to look at: the median change was 0.0002 and the largest 0.0012, about the rate times the hundred steps. Part 9’s Adam moves a weight with a steady gradient by about the rate each step. The change to one attention matrix used 572 of its 576 possible directions. Measured by the sum of squared changes, the top eight carry 71% of it and 27 carry 90%; LoRA’s eight carry all of it by construction. Subtracting the diff gives the original weights back, the one thing it shares with a patch you could review.

An unreadable patch only matters if it changed something you cared about, so I measured that. Eight versions of the same fine-tune: three learning rates for the full update and five for a LoRA adapter of rank 8 on the query and value matrices. Each was scored by how far its loss rose on six posts from this blog that none trained on.

The full fine-tune learned all twelve facts at every rate. LoRA learned none until 0.0003, thirty times the rate where the full one already worked. At the lowest rate I tried that does the job for each, the full update raised the loss on those posts by 0.005 nats and LoRA by 0.111. Over twenty times as much, a strange bill for the cheap option.

Now hold the rate fixed. At 0.001 the full update raised the loss by 4.419 against LoRA’s 0.287, fifteen times worse. This adapter does limit the damage. It just needs a rate at which there is damage to limit.

Be fair about eight runs on one seed. This is a 135M model, fifteen facts, a hundred steps and one adapter configuration, on a coarse grid of rates. An adapter on every linear layer might learn at a lower rate, and a larger model or a real dataset could order these differently. The part I’d expect to transfer is the method: compare at matched outcome as well as at matched settings, and measure the collateral on something the fine-tune was not about.

What the ML engineer watches

Four habits, and the first is a decision you make before any of this.

Ask what you’re trying to change. Missing facts belong in the context window, from retrieval, where you can see and remove them. Behaviour, format and refusals are what fine-tuning is good at.

Score the collateral as well as the target. Every run that learned the twelve facts ended above 99% on them, including the full update at 0.001, whose loss on the six posts rose by more than four nats. Part 8 asked for one held-out score; this phase needs two, one on general text and one on the thing you actually sell.

Treat the rate as the damage knob. Across eight runs it moved the collateral by three orders of magnitude; full against low-rank moved it by a factor of fifteen at the one rate where both learned. Sweep it as Part 9 did, and pick the rate by the collateral number.

Keep the base, the adapter and the merge apart. The Llama adapter at rank 8 is 6.8 MB beside 16 GB of frozen base, and it turns off in one call. That made the reference model above free in memory. Compute still pays: every step runs the model again with the adapter off. Merge when you serve one adapter on an unquantized base. Keep the pieces while deciding, or when one base serves many adapters.

None of this is modelling work either. A mask over labels and a sigmoid on a difference; the low-rank part is two skinny matrices, one of them zeroed. The arithmetic is small enough to read in an afternoon, and it still ends in a diff of a hundred million numbers that nobody can review.