The archive · 19 pieces
Writing
Long essays, working notes, and series on agent reliability, evaluation, and the EU AI Act.
Fine-tuning is the wrong tool for knowledge: three training phases and the low-rank shortcut
Pre-training and fine-tuning share a loss and differ in a mask; LoRA trains two narrow matrices beside the frozen weights. Facts taught by fine-tuning go in slowly and bring unearned confidence: mine named Moscow as the capital of a country that does not exist.
The learning rate is the hyperparameter that breaks production
Adam and the schedule are engineering decisions, not mathematics. Nothing validates the config: a bad value diverges thirty hours in and takes the run with it.
Loss and gradients without the formulas: backprop is a graph walk
Backprop is a graph walked backwards, applying the chain rule. Read as code, it explains vanishing and exploding gradients, and why a gradient bug shows up as a slightly wrong curve, not a stack trace.
From GPT-2 to Llama: four changes and the reason for each
Rotary positions, RMSNorm, gated feed-forward, grouped-query attention. Four local edits, justified by measurement more than theory, and each one baked into the weights.
GPT-2 is the transformer plus five decisions, and inference is a different program
One artifact, two programs. Training is one parallel forward pass over the sequence; generation is a sequential loop with a cache. Same weights, different bugs.
A transformer block is six tensors and a bus: read it like a pipeline
Six tensors on a residual stream, read line by line with shapes in the margin. Two thirds of each block's weights sit in the feed-forward; as the context grows, the cost moves to attention.
Attention is a learned weighted average, and the cost is in the square
A soft dictionary whose weights the model computes from its own input. The idea is cheap; the weight matrix is T by T, so the cost comes from the data, not the code.
Shapes are the type system of ML, and nobody checks them for you
[B, T, d] is a function signature. The runtime never verifies it, so a wrong axis raises only when sizes differ; when they collide, it broadcasts, returns plausible numbers, and ships.
An embedding is a lookup table, and everything else is how you fill it
A row of a matrix indexed by id. The science is how it gets filled; the engineering is that there is no schema migration: a new embedder invalidates your index.
A model doesn't read text: what a tokenizer decides for you
The tokenizer ships inside the model, not in front of it. Reading tiktoken's BPE for what it fixes before any weight exists: vocabulary, cost, context limit.
Who calibrates the judge? The gap nothing fills
Six posts, five strategies, five tools. Two measure judge-vs-human agreement; none gates a CI build on it with a published confidence band. That's the gap.
Deterministic where you can, judge where you must
Tool calls deserve deterministic comparison. Goal completion needs LLM-based assessment. Some tools draw the line; others don't.
The negative rubric: telling a judge what NOT to evaluate
TruLens uses rubrics that explicitly tell the judge what falls outside scope. The pattern is unusually explicit.
When judges show their uncertainty: DeepEval and ARES
DeepEval weights scores by token probabilities. ARES applies Prediction-Powered Inference for formal confidence intervals. Two routes to the same goal.
From rubric to graph: how DeepEval splits a judgment
DeepEval encodes evaluation as a graph traversal. Each node is a simpler decision than the overall judgment.
Article 72 named the loop. Your dashboards aren't closing it.
Article 72 demands lifetime performance data fed back into risk management. Dashboards don't close that loop, and the template is set to become guidance.
Binary verdicts: how Ragas keeps judges honest
Ragas decomposes complex judgments into atomic yes/no questions. The trade-off: you trade information for variance reduction.
Variance is the problem with LLM-as-judge, and tools don't agree on how to fix it
Reading the source code of seven OSS LLM-as-judge tools: variance manifests across four axes, and the seven approaches don't share a framework; they share a problem.
Article 15 named the requirements. The toolchain hasn't caught up.
Article 15 names five AI-specific vulnerabilities and a lifecycle requirement. Today's toolchain covers a fraction. Where the gap lives, and what to do meanwhile.