Who calibrates the judge? The gap nothing fills

Six posts. Five strategies walked through. Five tools out of the seven I read for the underlying research. One question the series kept gesturing at and never quite answered: how reliable is the judge, really, on the task it’s running, with the rubric it’s been given?

I’m building AgentAnvil, a contract-driven testing framework for LLM-based agents, and reading other evaluators is how I work out what those contracts should constrain. The strategies in this series narrow what the judge can say (Parts 2-3), measure how sure it sounds (Part 4), pin down what it should ignore (Part 5), or skip it where deterministic checks suffice (Part 6). Each is honest, each works, each leaves something behind. I assumed none of them shipped the primitive that measures how often the judge agrees with a human on your task. A second pass through the source forced a correction. Two of the five ship it. What none of them ships is the workflow that gates a CI build on that agreement, with a published confidence band the build can read.

This post is the series closer. It maps the gap: narrower than I thought, still real.

The intuition

A judge is a measurement instrument. Run a thermometer on the same liquid twice and you read two numbers; the spread between them is the instrument’s variance. The thermometer is useful because somewhere upstream a manufacturer characterised its error against a reference standard: boiling water, freezing point, NIST-traceable cells. The thermometer carries an implicit margin of error because that calibration step was paid for.

An LLM judge has no such step out of the box. You write a rubric, you pick a model, you point it at agent traces, you get scores. The score is what the framework returns; the framework rarely tells the CI gate what the score’s margin of error is on your task. To know that, you’d run the same items past one or more human raters, compute agreement against the judge, and treat the residual disagreement as the instrument’s error against the reference (bias plus whatever noise one pass carries); its run-to-run noise alone is measured separately, by re-judging the same items. Cohen’s κ (Cohen, 1960; see Landis & Koch, 1977 for the magnitude bins) is the standard primitive. That measurement is the calibration step.

When I started this series I assumed the primitive itself was missing across the OSS tools. It isn’t, quite. Re-grepping the five trees with calibration vocabulary in mind, two of them ship judge-vs-gold κ (Ragas through validate_alignment(), TruLens through GroundTruthAggregator(true_labels=...)), one (ARES) ships a coarser version, and Inspect AI ships an inter-judge α. What none of them ships is the regating workflow around those primitives: a published confidence band on the κ that a CI build consumes as a fail-the-build threshold, with a re-calibration trigger when the judge model or rubric changes. The primitives are there. The gate around them isn’t. That’s the gap this post is about.

The gap, mapped

Here’s the strategy-by-tool matrix at the end of the series, extended with two rows the walkthroughs didn’t reach.

The first five rows record what each post unpacked. Ragas ships scope constraint cleanly. DeepEval ships decomposition and a token-probability nod at calibration. ARES ships the PPI bound. TruLens ships the negative rubric. Inspect AI ships the deterministic-vs-graded partition. Six cells filled across the five rows: one per strategy, except probabilistic calibration, which DeepEval and ARES share.

The sixth row was the surprise. Ragas’s validate_alignment() in metrics/base.py takes your judge and a held-out human-labeled dataset and returns Cohen’s κ: discrete metrics through cohen_kappa_score, ranking metrics through weighted κ, numeric metrics through Pearson r. The first-party how-to (“How to Align an LLM as a Judge”) walks the same idea with raw percent agreement; validate_alignment() is where κ lives. TruLens goes further. GroundTruthAggregator(true_labels=...) in feedback/groundtruth.py ships cohens_kappa, ece (Expected Calibration Error), brier_score, matthews_correlation, plus rank-correlation primitives, exposed at the top of trulens.feedback and exercised by dedicated tests. ARES emits raw judge-vs-gold accuracy alongside its PPI bound. The primitives are not exotic and not absent: two of the five ship judge-vs-gold κ, ARES ships a coarser version, and Inspect AI ships krippendorff_alpha(), built for agreement among judges rather than against gold.

The seventh row is the one still empty everywhere. None of the five wraps those primitives into a CI-time gate keyed off a published confidence band: “fail the build when κ drops below 0.7 with a 95% lower bound under 0.6.” Ragas returns a κ point estimate; the bootstrap is yours. TruLens returns the same, but builds the most around it: its benchmark package ships AlignmentReport, CriteriaABTest (rubric changes) and CrossModelAlignment (judge-model changes) against a golden set, and Run.compare() puts a bootstrap CI on score deltas between runs. These are manual tools. No tool of the five publishes a confidence band on κ or wires one into a build gate.

ARES is the closest, and worth disambiguating. Its PPI machinery wraps a CI around the agent’s aggregate score, which sounds like it ought to fill the regating row. But the interval is over the population statistic, not over agreement with the held-out humans, and a regating workflow needs the latter. PPI tells you the aggregate score sits inside a band (Angelopoulos et al., 2023); it doesn’t tell the build to fail when judge-human agreement drifts.

That’s the gap. Narrower than I framed it at the top of this post, but real.

What the gap costs

Imagine the gate. A pull request modifies an agent’s plan generator. CI runs the eval suite: twenty rubrics, a thousand items each. Two rubrics drop by 3 points. Is that real?

That question has two halves. Whether the drop exceeds noise is answerable without labels: a paired bootstrap or permutation test over the per-item deltas on those 1,000 items, with both runs judged fresh so the judge’s rerun variance is inside the deltas. Whether a drop on the judge’s scale is one a human would see is the half only labels answer. A judge that misclassifies at random shrinks real deltas toward zero; one biased by the change itself (say, a shift in output style) can invent or hide them.

The widget lets you set how reliable your judge is against a hidden gold standard, then watch what the confidence band on Cohen’s κ looks like at different sample sizes. It’s not a calibration tool; it’s the cost demo for not having one. The point: even when you know your judge is 80% accurate, the band you can publish is wide unless you’ve annotated hundreds of items. Below about 100 human-labeled items, the band is too wide to certify the judge’s validity.

Most teams are below 100 human-labeled items per rubric. Most teams are at zero. The series has been documenting tools that operate cleanly in that regime: strategies that need no labels at all. The gap is what those strategies can’t substitute for.

Why no one ships the gate

The primitives are the easy part. The regating workflow around them is hard in a specific way that doesn’t map to a general OSS evaluator.

A framework can ship a judge, rubric templates, decomposition graphs, retry logic, structured output parsers: all the things you read in Parts 2-6. It can also wrap sklearn.metrics.cohen_kappa_score once the labels exist, which is exactly what TruLens’s GroundTruthAggregator.cohens_kappa does in feedback/groundtruth.py. What a general framework can’t ship is the threshold. The CI gate that fails when κ < 0.7 needs to know which 0.7 is yours, on which rubric, with what tolerance for noisy jitters that aren’t real regressions. None of that is universal across tasks, and none of it can be defaulted without being wrong somewhere important.

The widget shows what producing the labels actually costs. Pick the number of items, the raters, the per-item annotation time. The output is the bill: cheap for one rubric, but it multiplies by every rubric in the suite and recurs on every judge or rubric change. That bill, plus the per-team threshold question above, is why the regating hook hasn’t been packaged. ARES’s design pressure (“annotate a small set, use it efficiently”) is the literature’s response to the labelling cost. ARES applies it to RAG evaluation only.

What’s left to integrate is bounded and specific: a bootstrap CI wrapper around the κ Ragas and TruLens already compute, a config flag that says “these are my thresholds; gate on them”, and a hook that fires re-calibration when the judge model or the rubric version changes. The pieces are off-the-shelf: sklearn.metrics.cohen_kappa_score for the point estimate, the irrCAC family (Gwet’s inter-rater agreement package) for analytic confidence bands on κ, AC1 and Krippendorff’s α, statsmodels.stats.inter_rater for multi-rater extensions. The wrapper is the work no tool has done end to end.

What you can do today

Three moves you can make this quarter without changing your tool.

First: budget a calibration sample. Start with 50 items per rubric, labelled by one careful human; your judge supplies the second set of labels, and the pair gives you a Cohen’s κ point estimate. Scale to 100-200 once the rubric stabilises, because below ~100 items the band on the κ is too wide to certify the judge’s validity (the widget above demonstrates this). Whether your tool is Ragas (validate_alignment()), TruLens (GroundTruthAggregator), or something else, this is the one place the labels are unavoidable. Re-run the calibration once a quarter, and again whenever the judge model or the rubric changes.

Second: bootstrap the CI yourself and write the threshold into the gate. Neither Ragas nor TruLens returns a confidence band on κ; you compute it with a 1,000-resample bootstrap loop, or with irrCAC’s built-in confidence-interval helpers (kappa2.table in R, irrCAC.table.CAC(table).cohen() in Python). Then fail the build with a calibration-required error when the lower bound slips under your threshold, not a quality regression. The two failure modes feel similar from the outside; they require different fixes (rubric edit vs. agent fix). Disambiguating them is what an unwrapped judge cannot do for you.

Third: be honest in the eval report about what the judge can and cannot tell you. “Faithfulness dropped 3 points” is one kind of claim. “The judge scored faithfulness 3 points lower this run (paired bootstrap 95% CI: 1.6-4.4); on a 200-item calibration set it agrees with humans at κ=0.72 (95% CI: 0.62-0.82), so the drop is real on the judge’s scale, and how far it reflects a human-perceived regression is bounded by that agreement” is another. The second is what a quality gate would have to publish to actually gate quality.

None of these requires your OSS tool to ship more than it does today. They require accepting that the regating wrapper is yours, and treating it as a budgeted, repeatable engineering activity rather than a research project.

The series, end to end

The strategies in this series aren’t competing approaches. They are complementary moves against the same problem: five different cuts at narrowing what an LLM judge can mess up. A mature evaluation harness combines them: deterministic where it can (Part 6), scope-constrained where it can’t (Part 2), structurally decomposed where the judgment is complex (Part 3), rubric-anchored where it overlaps with siblings (Part 5), probabilistically calibrated where the output needs an interval (Part 4). The architecture is plural.

Visually the five stack like this. Toggle a strategy off and the bar exposes the share it was bounding; the right-most slice never moves: the regating residual no toggle reaches. (The slice widths are illustrative.)

I’m working on this gap in AgentAnvil. Whether AgentAnvil ships the closure or some other tool does is not the interesting part; the gap exists, it has a name, and the strategies in this series do not fill it.

Rule of thumb: a judge is a measurement instrument, and a measurement without a published, gated calibration band isn’t one a build can defend. Five tools in this series narrow the variance; none of them ships the gate.