混合裁判一致性
Agreement and order consistency measure different properties.
本地计算 / 模拟本地教学实验，不调用外部 API。合成示例不能替代生产环境测量。
学习指南与本课目录
LEARNING CONTRACT理解 → 实现 → 研究
目标：解释本课机制、复现一个例子、测试一个失败案例。读完不自动代表掌握。
- 理解层：写明假设、形状/单位和一个反例。
- 实现层：复算数值例子并检查边界。
- 研究层：在匹配预算下比较一个替代方法。
02 · 预测与配置
技术场景说明、参数名称与参考答案目前为英文。
03 · 检查证据
方法与参考资料
Build an exportable rubric and analyze supplied judge labels. No LLM is called. In swapped trials A means original B and vice versa; normalize before computing consistency. Fixture labels are synthetic. Rubric wording is an original starting point, not a validated scale.
Text / 原始公式
agreement=matches/n; consistency=matches(original, remapped swapped)/n
High agreement on a small biased sample does not validate a judge for a new domain.
Judging LLM-as-a-Judge · MT-Bench ↗练习题
judge-1 / 01What does “Mixed judge agreement” test?
Agreement and order consistency measure different properties.
Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗
打开实验 →judge-1 / 02Which computation produces the result in “Mixed judge agreement”?
Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗
打开实验 →judge-1 / 03What is the baseline configuration for “Mixed judge agreement”?
{"task":"Compare two answers for factual correctness using supplied evidence.","ratings":"[\n {\n \"human\": \"A\",\n \"judge\": \"A\",\n \"swapped\": \"B\"\n },\n {\n \"human\": \"B\",\n \"judge\": \"A\",\n \"swapped\": \"A\"\n },\n {\n \"human\": \"tie\",\n \"judge\": \"tie\",\n \"swapped\": \"tie\"\n },\n {\n \"human\": \"B\",\n \"judge\": \"B\",\n \"swapped\": \"A\"\n }\n]"}
Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗
打开实验 →judge-1 / 04What numerical result should you check for “Mixed judge agreement”?
{"Human agreement":0.75,"Order consistency":0.75,"A wins (original)":0.5}
Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗
打开实验 →judge-1 / 05How would you run a controlled comparison for “Mixed judge agreement”?
Keep the supplied fixture and other settings fixed; change ratings. Compare human agreement and order consistency; retain evidence spans in real evaluations.
Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗
打开实验 →judge-1 / 06What failure could invalidate a conclusion from “Mixed judge agreement”?
High agreement on a small biased sample does not validate a judge for a new domain.
Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗
打开实验 →judge-1 / 07Is “Mixed judge agreement” a benchmark of a deployed language model?
No. Build an exportable rubric and analyze supplied judge labels. No LLM is called. In swapped trials A means original B and vice versa; normalize before computing consistency. Fixture labels are synthetic. Rubric wording is an original starting point, not a validated scale.
Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗
打开实验 →judge-1 / 08Which evidence should you report for “Mixed judge agreement”?
Compare human agreement and order consistency; retain evidence spans in real evaluations. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.
Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗
打开实验 →judge-1 / 09How can another researcher reproduce “Mixed judge agreement”?
Open judge-1, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.
Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗
打开实验 →judge-1 / 10What is the next validation step after “Mixed judge agreement”?
High agreement on a small biased sample does not validate a judge for a new domain. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.
Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗
打开实验 →假设、参考实现与验收
比较次数必须为正。保留平局，把交换标签映射回原身份，并单独报告解析失败。合成评分不证明真实裁判可靠。
已复核引擎公式、场景预设及数值保护。固定种子的合成输入不是生产基准。
运行前：预测一个边界情况
Agreement and order consistency measure different properties.
提示：隔离假设
固定基线输入，只改变一个参数或边界；同时比较数值与声明单位。单例通过仅验证该样例。
最小可运行算例
将算例及所链接模块保存在同一目录。使用 Node.js 22 或更高版本，然后运行 node judge-1.mjs. 无需第三方运行依赖。
import {calculate,network} from './academy-core.mjs';
const params={
"task": "Compare two answers for factual correctness using supplied evidence.",
"ratings": "[\n {\n \"human\": \"A\",\n \"judge\": \"A\",\n \"swapped\": \"B\"\n },\n {\n \"human\": \"B\",\n \"judge\": \"A\",\n \"swapped\": \"A\"\n },\n {\n \"human\": \"tie\",\n \"judge\": \"tie\",\n \"swapped\": \"tie\"\n },\n {\n \"human\": \"B\",\n \"judge\": \"B\",\n \"swapped\": \"A\"\n }\n]"
};
const result=calculate('judge',params);
console.log(result.metrics);测试与迁移
Agreement and order consistency measure different properties.
在下方笔记记录输入、代码版本、期望值和实测值；再构造违反上述假设的反例，解释原结论为何不再成立。
下载完整代码包（.tar.gz） · 完整参考代码与回归检查预测、检验与证据记录
展开答案前先预测一个结果；练习后记录差异和迁移测试。记录仅存本地，在不同语言版本间共用。
打开记录／导出／清除 →数学复核范围逐篇记录，不代表完成论文实证复现；可在笔记中记录疑似错误。 复核范围