What this
series assumes
This series reads the source of seven open-source tools that use an LLM to score another LLM. It assumes you have shipped software and called a model API. It does not assume you evaluate anything today, or that you have used a single one of these tools.
What it
assumes
Engineering, not statistics.
If you can read Python and you have ever argued about what a flaky test means, you have the prerequisites.
- Python you can read. The parts quote real source, with file and line. You are reading it, not writing it.
- You have called a model API. That a prompt goes in and text comes out, and that the same prompt twice does not give the same text.
- What a test suite is for. Why a build gets gated, and what it costs when the gate is wrong in either direction.
- That variance is a property, not a bug report. No formal statistics: where a confidence interval appears, it is built in the part that needs it.
That last one is the whole subject. Every tool here is fighting the same thing — a scorer whose answer moves when nothing else did — and the interesting part is that they do not agree on how.
What it does
not assume
Not a tool tutorial.
No prior use of Ragas, DeepEval, TruLens or ARES. No RAG pipeline in production. No background in inter-rater agreement or Prediction-Powered Inference — Part 4 builds what it needs.
What the series is not is a comparison table telling you which tool to adopt. It reads the code to recover the strategy each one encodes, because the strategies outlive the tools and the tables go stale in a quarter.
Versions move. Every claim names the file and line it came from, so you can check whether it still holds in the release you are on.
If something
is missing
Read the tools, not the posts about them.
The four repositories the series spends most of its time inside, and the paper that started the practice. None of them is mine.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena The reference that made the practice mainstream, and the one that first measured a judge against humans. Read it if you read only one thing here.
- Ragas Part 2. The binary-verdict strategy: decompose a judgment into atomic yes/no questions.
- DeepEval Parts 3 and 4. Evaluation as a graph traversal, and scores weighted by token probabilities.
- TruLens Part 5. The negative rubric: telling a judge, explicitly, what falls outside scope.
- ARES Part 4. Prediction-Powered Inference, for a confidence interval around a judge’s verdict.
How to
read it
The argument compounds.
Part 1 first, then any order. It sets the four axes of variance the rest of the series keeps referring back to. Parts 2 to 6 are one strategy each and stand alone.
Part 7 is the point. Six posts establish what the tools do; the last one names what none of them does. If you only have time for two, read 1 and 7.
Every claim has a citation to code. File and line, not documentation. Where a tool says one thing and does another, the series says so and shows the line.