LLM-as-a-judge: evaluate the evaluator
Separate answer preference from correctness and reveal how order effects can alter a reported win rate.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · Zheng et al. · arXiv:2306.05685
Original editorial analysis by Token Lab · Source checked 2026-10-25 · Paper claims and synthetic examples are separated.
01 · What the paper establishes
The paper studies strong language models as judges and introduces MT-Bench and Chatbot Arena evaluations. It analyzes position, verbosity and self-enhancement biases and compares judgments with human preferences. Agreement in that setting is not a universal correctness guarantee.
Primary source: paper abstract ↗Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
02 · The mechanism
Text / 原始公式
Tie-adjusted win rate = (wins + 0.5×ties)/comparisons Order sensitivity = mean[decision(A,B) ≠ mapped decision(B,A)] Always map judgments back to candidate identity.
A judge is another measurement instrument with its own failure modes. A preference may reward style rather than truth, and a fluent justification does not establish a valid rubric. Swapping answer order is a useful control: remap labels to the same candidate before checking agreement. Repeated judgments of the same item are correlated observations.
Notation and intuition are teaching explanations. Consult the original method for complete assumptions and implementation details.
Mechanism map & worked derivation
| Symbol | Meaning |
|---|---|
| w, t, n | Wins, ties and number of comparisons |
| mapped(B,A) | Swapped judgment remapped to original identity |
| item ID | Unit required to detect paired inconsistency |
- Take 10 items with A winning 6, tying 2 and losing 2. The tie-adjusted win rate is (6+0.5×2)/10=0.7. Keep the tie convention explicit.
- Equal marginal rates need not mean consistent decisions. On four items, original winners [A,A,B,B] and identity-remapped swapped winners [B,B,A,A] both give A 50%, yet every item flips.
- Compute consistency from item-level pairs before averaging. Then analyze whether disagreements concentrate in math, long answers, missing evidence or a particular judge prompt.
Save raw labels, parsing failures and ties. Do not silently drop difficult cases or treat agreement as factual correctness.
Change one variable. Explain what moves.
Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.
04 · Work through an example
Original illustration: 100 paired items are judged twice. In A-first presentation, A wins 70; in swapped presentation, A wins 50 after identity remapping. The balanced marginal rate is 60%, but these totals alone cannot identify how many individual decisions flipped. Keep per-item outcomes to measure actual inconsistency. The slider changes only the second marginal count.
05 · Design your experiment
Create a blinded rubric and a human-labeled audit subset. Randomize answer order, include swapped pairs, fix judge version and preserve raw outputs. Track ties, parse failures, verbosity differences and correctness disagreements. Bootstrap by item rather than by repeated judge call. Evaluate whether observed differences change a real deployment decision.
State one hypothesis; save the configuration and raw results; identify a competing explanation; report the narrowest supported conclusion. Use the linked lab as a local starting point, not a full paper reproduction.
Open related lesson / lab →06 · Limits & reviewer questions
Editorial caution: human agreement is not the same as factual truth, and humans also disagree. A judge validated for one domain may fail in another. Averaging swapped orders can reduce an order imbalance but does not repair a systematically wrong rubric.
Why can two identical marginal win rates still hide severe inconsistency?
Different items can flip in opposite directions and cancel in the total. Only paired item-level decisions reveal that pattern; aggregate win counts discard the necessary information.
07 · Read the original
Bias analysis and human-agreement experiments; inspect prompt, judge and voting protocols.
Year is the first arXiv submission year. Unversioned links follow revisions; record the exact version in your reproduction report. DeepSeek-R1 here is explicitly pinned to v1.
Continue this topic
Calibration: being right is different from knowing how often
Adjust temperature without changing class ranking, and learn why one confidence score cannot establish calibration.
Guo et al. · arXiv:1706.04599
Deep RL that Matters: a score is a distribution
Separate seed variation, implementation choices and model selection before declaring an improvement.
Henderson et al. · arXiv:1709.06560
Show Your Work: compare search budgets, not just winners
A better winning score can come from more attempts. Plot the selection process as part of the result.
Dodge et al. · arXiv:1909.03004