Practical LLM Evaluation

1Evaluation is a decision, not a score

A red light in continuous integration doesn't say "the code scores 7.3 out of 10", it says "we don't deploy". The authors want the same for an LLM: "evaluation is not a number; it is a decision system" (lesson 4). It has four parts. The evidence is what you measure, the uncertainty says how far you can trust it, the threshold is the boundary fixed in advance, the action is what you do when the boundary is crossed.

Example: ship if routing accuracy reaches 90% overall and 88% on every critical intent. Without a threshold, "metrics are not actionable, and releases become debates" (lesson 4). The numbers are illustrations: "the exact numbers are product-specific" (lesson 8). The book keeps two or three decisive metrics per axis (quality, safety, cost, reliability); the rest explains why they moved.

2The right evaluator is the cheapest one that holds

Before paying for an LLM judge, ask whether a line of code can settle it. The book stacks four evaluators. Deterministic validators check what is true or false, like JSON that parses. Reference-based metrics compare against a label, like a ticket's category. The LLM judge, a model asked to grade another's outputs, covers the subjective. Humans keep the risky cases. The rule: "choose the cheapest evaluator that is reliable for the failure mode, then escalate only when needed" (lesson 11).

"The refund has been processed" is not a style problem. The validator reads the trace (the log of tool calls) and looks for a successful billing call before that sentence.

3The LLM judge is a noisy instrument, versioned like code

A scale whose tare changes between two weighings tells you nothing about the flour. A judge's outputs are likewise "noisy measurements, not ground truth" (lesson 11). The book lists the flaws without putting numbers on them: inconsistency from one run to the next, sensitivity to the rubric (the grading grid given to the judge), a taste for long answers.

The remedy comes down to concrete habits. A fixed rubric, one dimension at a time, a simple scale, evidence quoted by the judge. Calibration, meaning comparing its grades with those of humans, with the advice to calibrate "against a small human-rated set" (lesson 11). Then version the model, the prompt and the rubric. If the judge changes, the old and the new one grade the same calibration set before anyone compares curves (lessons 17 and 58). One last trap, stated for vision-model judges but valid for text: a judge that hallucinates like the evaluated model "will confirm its hallucinations rather than catch them" (lesson 58). Hence a judge from another model family, or one calibrated on human judgments, with calibration tracked over time.

An honest limit: no bias rate, no quantified calibration protocol. Kappa (agreement between raters, corrected for chance) shows up in a single paragraph of the whole text, in passing during data annotation (lesson 70). The only threshold given is for a Krippendorff's alpha, at least 0.67, "subject to domain review".

4Don't react to noise: interval and minimal change

8.10 then 8.18 proves nothing if two identical runs already differ by 0.2. The book gives every noisy metric a confidence interval, the range where the true value falls with a given probability. An accessible method is the bootstrap: you draw cases at random, with replacement, from those evaluated, recompute the mean, repeat, and the range comes out of the spread.

Before crying regression (a drop that justifies acting), the book asks four questions. Does the change exceed the run-to-run variation, and the minimal useful change, the smallest gap that would alter a decision? Is it concentrated in a critical slice? Does it cross a threshold? The real question is not "did the number move" but "did it move enough to change the decision" (lesson 13).

5The closed loop: every incident becomes a test case

Offline, a set of cases freezes the configuration and blocks a version that regresses. A canary (a release to a small share of traffic) then keeps watch. Online, you log, measure drift, and sample for review. Every incident goes back into the test suite: it becomes, the book says, "a memory of what broke before" (lesson 15). Without traces, "incidents remain anecdotes" (lesson 17).

Two clarifications helped me. The development set, which you keep tweaking, stays separate from the hold-out (the reserve set), which serves release decisions and on which you never tune anything, or you end up optimizing against your own exams. A gate that is too jumpy causes "rollback fatigue" (lesson 14) and the team raises the thresholds without analysis.

6Output contract, bounded loop, claim tied to evidence

An online form rejects a made-up date before opening a file. The book demands the same of the model's output through a contract: expected fields and types, often in JSON. It sees in it "a control layer, not a quality metric" (lesson 32). It guarantees a usable output, not a correct one. Around it comes the reject-and-repair loop, in three rules: repair only when the repair is "strictly syntactic and safe", switch to a fallback when the meaning is ambiguous, cap the attempts. Every retry lengthens tail latency and cost (lesson 32).

# my own, with parse_json_object and contract_error from the book
raw = call_model(prompt)
for attempt in range(MAX_ATTEMPTS):
    obj = parse_json_object(raw)
    error = "invalid_json" if obj is None else contract_error(obj)
    if error is None:
        return obj
    if not syntactically_repairable(raw):   # trailing comma: yes. Doubtful meaning: no.
        break
    raw = repair(raw)
return fallback_on(error)                   # safe template, error code logged

The neighboring control is the strongest. "Refund confirmation is not a writing task": it is a claim that must rest on executed evidence (lesson 33). The gate ties the sentence to an indicator drawn from the trace, not to what the model says about itself. If the indicator is missing, you block the reply, fall back to a safe template and open an incident.

7Evaluating an agent step by step

An agent that cancels an order can read a wrong status, pick the wrong tool, retry without progress, then cancel anyway. So the book splits evaluation into eight steps: input and state validation, action choice, execution, recovery, loop detection, completion, safety, latency and cost (lesson 102). The text is dry about it: saying "the agent failed" "is not a useful root-cause analysis" (lesson 102).

Four ideas come out of it.

• The final state, not the trajectory: "the key evaluation point is not whether the agent exactly follows one reference trajectory" (lesson 102).
• A tool failure gets classified (timeout, permissions, schema, malformed response) before you pick the remedy among six: bounded retry, another tool, ask the user, rollback, human, safe stop (lesson 102). The book only measures that the chosen remedy matches the failure type (lesson 103). Replanning (making a new plan after the failure) is graded separately.
• A loop is spotted by repeated calls with the same arguments and no progress, or by an exceeded step budget.
• Completion is not a Boolean: full success, partial, safe failure, unsafe failure, resolved by a human (lesson 105). Pushing completion can raise safety violations.