AI evaluation: from measurements to decisions, research and market power
A field guide to traditional ML, LLM and agent evaluation: statistics, Arena, engineering, research frontiers, companies and acquisition evidence.
Jump to the runnable reference and tests
来源与版本详情
Contents · chapters, research and reference code
本文目录
评测:判断系统能否发布
以下原创案例贯穿一个发票助手:它读取租户范围内的资料并提出付款建议,实际付款需要单独且精确的授权。场景与实验均为合成教学材料,不代表真实部署。
先明确决策:这个发票 Agent 能否上线?
本例助手回答发票问题、读取租户资料并建议付款,只有独立授权允许它执行精确操作。新模型文笔更好、新检索返回更多材料,并不说明二者组合适合上线。先定义目标用户与任务总体:一次客户业务案例是一个统计单位,六次重试仍是一案,但每次都计入成本和风险。
验收合同应固定模型、提示词、语料快照、工具版本、调用预算、超时、评分规则及排除条件。支持充分的回答或正确拒答、时限和权限边界分别报告,不能让流畅度抵消越权。超时评分器记为“未知”,不要偷偷删掉;模型未完成任务则按预定规则记失败。证据不足时暂不上线也是有效决策。
阶段一:把业务任务建成可复现的环境
样本应覆盖正常发票、缺失政策、版本冲突、过期余额、外租户材料、收款人变化、撤回审批和重复提交。每案记录初始状态、允许的终态以及禁止的副作用;“准备付款”不能以“已经付款”为成功。教学基线使用合成文档和模拟工具,不会真的付款。
按客户、文档家族或时间拆分开发、验证和最终保留集。同一发票的十个改写通常不是十个独立样本。将正确证据直接交给生成器的 oracle-context 条件可隔离生成问题;缺失证据条件检查拒答。生产分布若有90%简单题、10%复杂题,不能用各半的诊断集直接估计整体表现:应报告分层结果和明确权重。
阶段二:把观察转化为可信判断
能精确检查的事实使用确定性评分器:JSON结构、许可文档、付款参数、账本终态、重复效果与审批消费次数。语义支持关系才交给人工或校准过的模型评分器。工具调用记录只证明尝试,业务状态才证明结果;一条真实引用也可能完全不支持回答。
评分器需要独立复核样本,分别统计误放和误杀。例如1000个合成样本中有50个错误,一个总判通过的评分器虽有95%准确率,却漏掉全部错误。改变回答顺序、长度和诱导性文字,检查评分是否稳定。先冻结评分器版本,再比较候选系统;评分器失联记未知,并报告覆盖率和类别错误。
阶段三:控制变量并解释不确定性
基线与候选使用同一案例、动作预算和资源上限。重复运行用于估计随机性,但任务与试次必须分开。固定检索比较模型,固定模型比较检索,再测二者组合,才能看出交互作用;额外上下文和重试必须单独记账。外部API的种子不保证跨版本完全确定。
合成例:200个独立案例中共同通过150,只有候选通过20,只有基线通过10,共同失败20。基线80%,候选85%,提高5个百分点;但30个结果相反的配对决定了差值的不确定性。正态教学近似标准误约2.72个百分点,95%区间略跨零,不能仅凭85大于80宣布稳定提升。有客户内相关性时应以客户为单位重采样。
阶段四:让发布门禁能够拒绝更高分
看结果前定义最小有用提升、不确定性容忍度、延迟成本上限和硬性安全约束。更高平均分若伴随越权付款,不能上线。100个对抗案例零失败是有用证据,却不等于零风险;独立伯努利假设下单侧95%失败率上界约2.95%,且非随机对抗集不能直接代表线上发生率。
先影子运行,确保建议不能提交;再限定金丝雀人群,并保留原有外部权限控制。比较同期组,预先设定观察周期或使用有效序贯方法,避免看到显著就停。回滚不仅是换回模型:已经提交的业务操作必须对账,不能重新执行。每个阻断条件应有负责人和恢复依据。
How the four disciplines fit together
Security determines which actions are permitted. Infrastructure executes work within capacity and failure constraints. Observability preserves evidence of what happened. Evaluation tests whether a change improves the outcome under a declared contract. None substitutes for the others.
机制、研究与市场背景——以下参考材料仍为英文
The following chapters provide the mathematical examples, experiments, research papers and dated company evidence behind the operational story. Return to the lifecycle when deciding where each mechanism belongs.
01 · Evaluation begins with a decision, not a leaderboard
Imagine that an engineering team is replacing the model behind a customer-support assistant. The new model wins more public head-to-head comparisons and writes more polished answers. Yet its first production week produces more incorrect refunds. Both observations can be true. A public preference ranking measures how a particular population judges particular responses; a refund workflow requires correct policy interpretation, authorization and state transitions. The mistake is not necessarily in either measurement. It is in treating one measurement as a substitute for another.
Definition. An AI evaluation is a specified procedure for gathering evidence about a system's behavior under stated conditions and using that evidence to support a decision. A benchmark is a reusable collection of tasks and measurement rules. A metric summarizes observations. A grader turns an output, action or environment state into a judgment. A harness controls execution and records the conditions. These objects are related, but none is interchangeable with the others.
The decision should be written before collecting scores: “Release candidate B only if it improves successful resolution on our target tasks without increasing unauthorized actions, while meeting our latency and cost budgets.” This statement identifies the target population, the benefit, the constraints and the deployment boundary. It also exposes missing knowledge: what counts as successful resolution, how frequently dangerous cases occur, and how confidently we can measure the difference.
Before this lesson, you should understand a sample mean as an estimate of a population property, a conditional probability as a statement restricted to specified conditions, and a train/test split as a barrier against adapting to the answers. You do not need to have memorized advanced statistics. You do need to recognize that running the same task repeatedly measures execution variability, while collecting new tasks measures a different source of uncertainty.
By the end, you should be able to design a release gate, interpret an Arena-style comparison, calibrate an automated judge, build an auditable evaluation pipeline, and explain why a high score can coexist with an unsuitable product. The examples in this chapter are original teaching examples unless explicitly attributed; they are not measurements of commercial models.
Choose a reading route
First encounter: read the definitions and worked examples, then predict the local experiment result before changing its input. Engineering implementation: focus on the data contract, architecture and Python reference, then write one failure test of your own. Research reading: inspect the cited paper versions, challenge their evaluation assumptions, and choose one open problem with a falsifiable baseline.
02 · Traditional ML, LLMs and agents: what changes?
Traditional supervised evaluation often starts with a fixed input, a reference label and a loss function. A classifier receives an image and predicts a class; a forecasting model predicts a number. Accuracy, precision, recall, calibration and error by subgroup can be useful because the target is comparatively explicit. Even here, labels can be wrong, class distributions can shift, and the cost of a false negative can differ from that of a false positive. Traditional evaluation was never automatically simple or unbiased.
For a language model, several outputs may be acceptable. A concise explanation and a longer explanation can both be correct; exact string matching can reject one for irrelevant reasons. Meanwhile, an apparently correct conclusion may contain fabricated evidence. Evaluation therefore separates dimensions such as factual correctness, instruction following, evidence support, completeness and usefulness. Combining them into one number is a policy choice that requires explicit weights.
An agent adds actions and environmental state. It can finish a task through different trajectories, use extra tools, encounter failures, recover, or damage something unrelated to the task. A final paragraph saying “Done” is not evidence that the task was completed. A database query, repository diff or sandbox state may be the appropriate verifier. The evaluated object is now the model plus its prompt, tool definitions, memory, permissions, retry logic and surrounding application.
Three distinctions prevent many misleading comparisons. First, model evaluation isolates a model as far as possible; system evaluation tests the assembled product. Second, capability evaluation asks what can be accomplished under a budget; reliability evaluation asks how consistently the behavior occurs. Third, offline evaluation estimates performance on controlled tasks, whereas online experimentation estimates effects on users under an actual deployment.
None replaces the others. A model can be capable but unreliable, a reliable product can depend on expensive scaffolding, and a statistically positive online result can conceal a severe safety regression. Keep the axes visible until the decision requires an explicit tradeoff.
03 · Write an evaluation contract
A useful evaluation contract names the task distribution, unit of analysis, execution budget, grading rubric, exclusion policy and release rule. For a support agent, a unit might be one customer case, not one model call. For a coding assistant, it might be one repository issue, not every generated patch. If a case produces ten retries, treating those retries as ten independent customers exaggerates the amount of evidence.
Make the task set versioned and immutable during a comparison. Record the repository snapshot or document corpus, model identifier, prompt hash, tool schema, inference settings, maximum actions, wall-clock timeout and grader version. Record unavailable dependencies and API errors as explicit outcomes. Quietly deleting inconvenient failures changes the question being answered.
Use three task collections with distinct roles. A development collection helps engineers investigate errors. A validation collection supports regular tuning and model selection. A held-out test collection supports a final estimate and should not become a daily feedback channel. When the team has repeatedly inspected the held-out answers, it has effectively converted that set into development data. Refreshing a test set is not a failure; pretending it remains untouched is.
A contract also specifies the aggregation rule. Suppose the task mix is 80% simple retrieval and 20% difficult account changes. Averaging those tasks equally may answer a different question from weighting them by production traffic. Conversely, production-weighting alone may hide a rare but unacceptable unauthorized transfer. Report an overall estimate and decision-critical slices, then apply hard constraints separately where appropriate.
Finally, define inconclusive outcomes. “Insufficient evidence to release” is a legitimate result. A good process does not force every experiment into a winner and a loser. It preserves uncertainty until the expected value of further testing is lower than the cost of waiting.
04 · Uncertainty, paired comparisons and repeated trials
A score is an estimate, not a physical constant. If candidate A succeeds on 80 of 100 tasks, the observed success rate is 80%, but a different sample could produce another value. A confidence interval communicates sampling uncertainty under a specified model. It does not account automatically for mislabeled tasks, a biased sample, contaminated training data or a broken grader.
Compare two systems on the same tasks whenever possible. For each task, calculate the difference between candidate and baseline. Paired comparisons remove some variation due to task difficulty: a hard task is hard for both systems. Bootstrap whole tasks or higher-level clusters such as users or repositories when those are the independent units. Do not independently resample individual tool calls from the same trajectory.
Repeated trials answer a second question: how much does behavior vary when the task is held fixed? Keep the task identifier and trial identifier separate. A system that solves each of ten tasks in one lucky run out of ten has different production behavior from one that reliably solves nine tasks and always fails the tenth. Their pooled success rates can conceal these patterns.
Pass@k describes the chance that at least one of k attempts succeeds under the evaluation's sampling and selection assumptions. It is relevant when a user can actually generate and identify a good candidate. It does not mean the first answer is reliable. A requirement that all k repeated attempts succeed is a different reliability criterion, sometimes written pass^k. Under independent identical success probability p, these teaching models become 1−(1−p)^k and p^k respectively; independence is often unrealistic in deployed systems.
For rare failures, zero observations is not proof of zero risk. Under independent Bernoulli trials, the rough “rule of three” gives a 95% upper bound near 3/n after zero failures in n trials. With 300 clean trials, a failure rate around 1% is still compatible with this approximation. Correlated tasks or an unrepresentative test population weaken that reassurance further.
At least one success is not dependable repetition
With p = 0.8 and k = 3 independent attempts, these values are 0.992 and 0.512. They answer different product questions. For sampled code completions, a common finite-sample pass@k estimator with n generated candidates and c passing candidates is one minus the ratio of combinations shown below. It estimates oracle availability; an imperfect selector can perform worse.
When fewer than k failures exist, the numerator is zero. For n = 10, c = 2 and k = 3, the estimate is 1 − 56/120 = 0.5333. Keep failed, truncated and unparseable attempts in the declared denominator. This estimator does not remove task selection bias.
Source for the sampled-completion estimator: Evaluating Large Language Models Trained on Codehttps://arxiv.org/abs/2107.03374v2, first submitted July 7, 2021; v2 July 14, 2021. Versioned PDFhttps://arxiv.org/pdf/2107.03374v2. The numerical examples here are original and do not reproduce its model scores.
05 · Build graders, then evaluate the graders
Choose the strongest feasible evidence for each requirement. If a generated file must parse, execute a parser. If a balance must remain unchanged, inspect the balance. If an answer must cite an accessible source, verify the source and its relationship to the claim. A language-model judge is useful for semantic judgments that cannot be captured cheaply by deterministic checks, but it should not replace a stronger available verifier.
A rubric should define observable differences between scores. “Excellent” and “poor” are adjectives, not measurement rules. For an evidence-grounded answer, a rubric might distinguish unsupported central claims, partially supported claims with missing qualifications, and a complete answer whose important claims are supported by the supplied documents. Include examples near the decision boundary, not only obvious successes and failures.
Calibrate the judge on an independently annotated set. Measure agreement and inspect the disagreement matrix, especially false passes on consequential errors. An overall agreement score can look strong when almost all examples pass. Include adversarial examples, long fluent wrong answers, concise correct answers, irrelevant citations and attempts to instruct the judge from inside the candidate output.
The foundational MT-Bench and LLM-as-a-judge paperhttps://arxiv.org/abs/2306.05685v4 studies position, verbosity and self-enhancement biases. Its reported human agreement belongs to its tested models and tasks, not every future judge. The paper first appeared on 9 June 2023; the linked v4 is dated 24 December 2023. Read the PDFhttps://arxiv.org/pdf/2306.05685v4.
In your own protocol, randomize answer order, reverse a subset of pairs, blind provider names, constrain the rubric, and audit failures manually. Preserve judge versions and outputs so a score can be reproduced. Treat a judge upgrade as a measurement-system change: regrade a bridge set before interpreting a historical trend as model improvement.
A passing schema can still contain a false answer
The complete Python reference below includes a contract grader for an answer, its source identifiers and an answer/abstain status. It rejects malformed JSON and references outside the allowed source set. A deliberate counterexample changes “30 days” to “300 days” while preserving the schema and source ID: the contract grader still passes. This test is intentional. Structural validity, citation existence and semantic support are different properties; a real application needs a separately calibrated check against source content.
Keep grader failure distinct from candidate failure. An unavailable judge should produce an unknown evaluation state and a retry/review policy, not a fabricated zero or perfect score. Never execute a candidate response as Python merely to parse its structure.
06 · Arena: preference measurement and its boundaries
Arena, formerly LMArena and Chatbot Arena, made pairwise human preference a prominent part of model evaluation. In a blind comparison, a person sees responses to a prompt and expresses a preference before model identities are revealed. The statistical problem is to infer relative strengths from a sparse network of comparisons, not simply to count total wins across unequal opponents.
The Bradley–Terry model represents the probability that model A is preferred to B as a logistic function of their strength difference. The strengths are relative: adding the same constant to every strength leaves pairwise probabilities unchanged. A model with a higher estimated strength is more likely to win under the observed comparison process. This is not an absolute percentage of “intelligence.”
The original Chatbot Arena paperhttps://proceedings.mlr.press/v235/chiang24b.html is an ICML 2024 publication, with its official proceedings PDFhttps://raw.githubusercontent.com/mlresearch/v235/main/assets/chiang24b/chiang24b.pdf. The research establishes a concrete pairwise evaluation framework; it does not establish that public preference is sufficient for every deployment.
Read rankings with their uncertainty, category, date and methodology. Differences can be too small to distinguish confidently. The population of people submitting prompts is not necessarily your customer population. Fluent presentation can affect preference. Different categories can reverse a ranking. Connected comparison coverage matters: a model tested against a narrow or atypical opponent set requires careful estimation.
Arena's 14 July 2026 factuality announcementhttps://arena.ai/blog/factuality-in-arena describes a separate, initially optional Text/Search view combining preference with claim-level factuality audits. Its stated default factuality weight is 25%. This is a useful illustration of an evolving construct: changing the mixture changes what the ranking means. Do not splice the adjusted score into an older preference-only time series without explaining the break.
Use Arena as external evidence and a source of candidate models. Then run a matched evaluation on your own tasks. The engineering question is not “Who is number one?” but “Does this candidate improve our outcome under our constraints?”
Work through the pairwise model
The strengths theta are dimensionless log-odds parameters in this unscaled formulation. A difference of ln(3) gives a 75% predicted preference probability, not a 75% task accuracy. Fix one strength or constrain their sum to remove the arbitrary shared offset. Estimate strengths from the comparison graph, and inspect task and voter dependence before trusting naive independent-vote uncertainty. Ties require an explicit modeling policy rather than silently counting every tie as a decisive win.
07 · Current frontier: measuring validity, contamination and alignment
The research frontier increasingly concerns whether measurements justify their claims. Measuring what Mattershttps://arxiv.org/abs/2511.04703v1, submitted 3 November 2025 and formally published in the NeurIPS 2025 Datasets and Benchmarks trackhttps://proceedings.neurips.cc/paper_files/paper/2025/hash/1967e0fc3aa6cbbace562f5cb8e3954e-Abstract-Datasets_and_Benchmarks_Track.html, reviews 445 benchmarks with 29 expert reviewers. Its central concern is construct validity: whether the task and metric represent the phenomenon claimed. PDFhttps://arxiv.org/pdf/2511.04703v1. A practical implication is to draw the chain from real-world requirement to task, observation and score, then challenge each link.
Contamination is another threat. On 23 February 2026, OpenAI reported problems with SWE-bench Verifiedhttps://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/, including flawed tests and contamination. Its 59.4% flawed-test finding applies to an audited difficult subset of 138 problems, not automatically the entire benchmark. This is a provider-authored audit, and its scope matters. The lesson is to inspect failed cases and grading assumptions, not to assume every low score is a model defect.
Arena's 8 October 2026 Alignment Indexhttps://arena.ai/blog/ai-alignment-index describes three observable signals: unauthorized action, false attribution and deceptive completion. It uses judged evidence and adjusts for conversation length. This is an operational measurement of selected behaviors, not a proof of complete alignment. A narrow, clearly defined indicator can be useful precisely because its limits are visible.
These developments point toward an important discipline: evaluate the evaluation. Test whether a benchmark responds to genuine improvements, whether superficial changes can raise its score, whether labels are reproducible, and whether the score predicts a meaningful deployment outcome. A benchmark that is difficult but unrelated to the product is not automatically valuable.
08 · RAG evaluation: separate retrieval from answering
A retrieval-augmented system can fail before the language model sees the evidence. If the relevant policy paragraph is absent from the retrieved context, improving answer wording will not solve the underlying problem. Conversely, perfect retrieval does not guarantee that the model uses the evidence correctly. Evaluate the pipeline at its interfaces.
For retrieval, define relevance at the unit you actually retrieve: document, chunk or passage. Measure whether the required evidence appears within the allowed context budget. Recall@k is meaningful only with a suitable relevance set and a specified k. A large k can improve recall while increasing latency, cost and distracting context. Test reranking and chunking under the same token budget rather than comparing an inexpensive short context against a much larger one.
For generation, separate answer correctness from support. An answer can be factually correct but unsupported by the retrieved material; it can quote a source accurately while answering the wrong question. Include citation validity, entailment of important claims, completeness, abstention when evidence is insufficient, and handling of conflicting or outdated documents.
Use counterfactual tests. Remove the decisive passage and check whether the system acknowledges uncertainty. Replace a policy with a newer version and inspect whether the answer changes appropriately. Insert an irrelevant but persuasive paragraph and test resistance to distraction. Add a malicious instruction inside a retrieved document and verify that the system treats it as data rather than authority.
A fair ablation changes one component at a time: retrieval method, chunk size, reranker or generator. Keep the corpus snapshot, questions, answer rubric and budget fixed. The result should explain a mechanism—such as improved evidence recall on long policies—rather than merely report a higher aggregate number.
09 · Agent evaluation: outcomes, permissions and recovery
For an agent, evaluate both the final state and the path constraints. A travel assistant that books the right flight using an unauthorized payment method has not succeeded. A coding agent that fixes a test by deleting the test has not solved the intended problem. The verifier needs to inspect meaningful invariants, not only whether the final response sounds successful.
Construct a sandbox with explicit initial state, permitted actions and expected postconditions. Record tool inputs and outputs, environment changes, retries, timeouts and side effects. A task should define whether asking for clarification is acceptable, when human approval is required, and what should happen when a tool fails. Otherwise, the grader may punish sensible behavior or reward recklessness.
Anthropic's 9 January 2026 agent-evaluation guidehttps://www.anthropic.com/research/demystifying-evals-for-ai-agents emphasizes tasks, trials, graders, transcripts and outcomes. The engineering takeaway is to inspect trajectories when scores behave unexpectedly and to combine grader types. The guide is practitioner evidence, not a universal benchmark result.
Security evaluation needs an attacker model. Specify what the attacker controls: a retrieved page, email body, repository file or tool response. State the protected boundary and observable violation. Measure attack success and benign-task completion together; a system that refuses every request may look secure while being useless. Keep attack development separate from the final test set.
Recovery deserves its own tasks. Inject a transient timeout, stale credential, contradictory tool response or partial write, then verify whether the agent retries safely, rolls back, asks for help or stops. Do not assume that a successful rerun demonstrates a reliable repair policy. Compare against a retry-only baseline and account for extra compute.
For recursive self-improvement, freeze the evaluator outside the optimization loop. If the same system proposes changes, rewrites the tests and judges its own success, a rising score may reflect adaptation to the measurement process. Independent holdouts, resource budgets and explicit stopping rules are essential.
10 · Multimodal and physical AI need different evidence
Multimodal evaluation is not text evaluation with an image attached. A visual system may need to locate a small object, read a chart, distinguish a temporal event or combine speech with a scene. A text-only final answer can hide whether the model used the relevant modality or guessed from linguistic priors.
Design modality-specific counterfactuals. Change the number in a chart while leaving the question unchanged. Shift an event to a different video timestamp. Replace spoken words while preserving background noise. Mask the decisive region of an image. A system that gives the same answer after the decisive evidence changes may be relying on the wrong signal.
Report input resolution, frame sampling, audio duration, preprocessing and latency budgets. A model receiving every video frame is not being compared fairly with one receiving a few selected frames unless that resource difference is part of the question. Human ratings of aesthetics or naturalness should be separated from task correctness and accessibility.
Physical AI adds dynamics and safety. A robot policy evaluated only in simulation may exploit simulator artifacts or fail under sensor noise, wear, contact variation and unexpected objects. Report the distribution of environments, reset procedure, intervention rate, task completion, collision or constraint violations, and recovery behavior. A successful demonstration is not a reliability estimate.
The broader principle is to preserve the causal connection between the evidence and the claimed ability. If the claim is spatial reasoning, eliminate shortcuts that can solve the task without spatial evidence. If the claim is robust control, vary the physical conditions rather than replaying one favorable trajectory.
11 · An engineering architecture for continuous evaluation
A production evaluation system has four separable layers: a versioned task store, an execution harness, grading services and a decision/reporting layer. Keeping these boundaries explicit makes it possible to rerun old outputs through a new grader, compare a new model under an unchanged harness, and diagnose whether a regression came from the product or the measurement system.
The task store contains inputs, references, metadata, permissions and provenance. The harness creates isolated environments, enforces budgets and captures artifacts. Graders produce dimension-specific results with evidence. The reporting layer aggregates by task slice, uncertainty and cost, then applies release rules. Every result should be traceable to the exact versions that produced it.
Run inexpensive deterministic checks on every change. Run a representative regression suite in continuous integration. Schedule larger repeated-trial and adversarial evaluations when a model, prompt, retrieval index or tool changes materially. Expensive human review should focus on ambiguous or consequential cases rather than being allocated uniformly without purpose.
Connect evaluation to observability without confusing them. Observability records what happened in production: traces, errors, latency, usage and feedback. Evaluation applies a criterion to selected behavior. A trace is evidence for debugging; it is not automatically a quality label. A useful feedback loop turns appropriately redacted production failures into reviewed regression tasks, while keeping a separate holdout for unbiased assessment.
Protect the measurement pipeline itself. Candidate outputs are untrusted data and can contain instructions aimed at the judge. Sandbox executable submissions, restrict network access, prevent test leakage and log grading exceptions. A failed grader should be reported as unknown or infrastructure failure, not silently converted into a model pass or fail.
行业格局:控制点、买方与竞争
行业证据核对于 2026 年 11 月 8 日。产品文档只能证明供应商提供了相关能力,不能证明独立实测效果。竞争判断与机会分析为 Token Lab 的分析;标为合成的算例不是公司实测数据。
评测产业:谁出售证据,谁掌握决策?
核对日期为 2026 年 11 月 8 日。AI 评测包含多个市场,而不是单一排行榜生意。模型实验室购买能力反馈,应用团队购买回归工作流,企业风险团队需要可审计的验收证据,基础设施买方比较成本与性能。一家公司在一个市场强,并不意味着能替代其他市场。关键是其证据支持什么决策、适用于哪些条件,以及证据错误时谁负责。本分析区分公开产品与交易事实,以及本站对商业动机的解释。融资金额和厂商主张不是评测准确性的独立证据。
四种决策,对应四类证据产品
公开基准帮助筛选模型,私有任务数据帮助决定是否发布应用修改,安全评测探查特定威胁模型,生产结果研究则判断系统是否改善真实业务流程。它们互补:发票助手可能赢得写作偏好比赛,却使用未授权供应商记录。本站的市场判断是,应按厂商改善什么决策分类,而非是否宣传 AI 评委。合理采购首先定义总体、任务、验收标准、失败代价和更新频率,然后才能公平比较人工评审、代码检查、模型评委与托管服务。
参考架构:独立的评测与发布决策
这是一套发布控制架构,不是抓取排行榜的流程。版本仓库固定基线与候选;冻结案例和匹配的工具权限送入隔离运行器,输出原始观察,包括失败和成本。版本化评分器与人工裁定分别提供证据,再进行配对汇总。发布门槛依据预先声明的质量与风险标准,决定是否进入有限灰度。最终留出集对候选优化过程不可见,只按评测协议使用。
开发反馈与最终测试反馈必须分开。反复根据留出集选择候选,就已把它变成开发数据。候选不能修改任务、评分器代码、分数或晋级规则。保存配置和原始结果,使表面提升可以追溯到模型、检索、提示词或工具变化。灰度监控检查部署行为,却不能修复受污染的离线比较;回滚权限属于发布系统,而不是被评测的 Agent。
公司与产品定位
这是选择性行业地图,不是市场份额排名或供应商认证。覆盖范围取决于版本、集成、部署与执行模式。
Arena / LMArena
- 所有权/组织
- Arena Intelligence Inc.
- 控制点
- 人类偏好与外部评测
Arena 出售基于人类反馈的外部评测,与运营客户内部回归集不同。商业服务于 2025 年 9 月 16 日推出,提供反馈分析、代表样本和交付承诺。社区参与是独特输入,边卖服务边维持可信中立是商业挑战。偏好获胜不能认证工具权限或业务正确性。应指明 Arena 的产品与方法:当前 Agent Mode 不同于经典成对偏好排名,不能用同一统计描述覆盖所有产品。
Arena commercial evaluation launch ↗https://arena.ai/blog/ai-evaluations
Artificial Analysis
- 所有权/组织
- 独立基准评测公司
- 控制点
- 模型与基础设施选型
Artificial Analysis 测量模型、Agent、推理供应商与硬件,并提供公开信息及免费、商业 API。潜在买方需要选型和采购情报,而非应用发布关卡。其价值主张是跨质量、性能和经济维度的可比性。综合指数仍是人为构建的指标,必须保留版本、设置和日期。当前 Intelligence Index 方法以英语文本为主,多语言与其他模态需要另行证据。结果适合筛选候选,之后仍应测试客户真实工作负载。
Scale AI / SEAL / Scale Labs
- 所有权/组织
- 独立公司;Meta 持非投票少数股权
- 控制点
- 专家数据与研究评测服务
Scale 结合专家数据运营、评测服务,以及 SEAL 和 Scale Labs 的研究。潜在优势是规模化组织困难任务与专家裁决,尤其适用于简单自动标签不足的情形。Meta 少数股权投资是真实所有权事实,不证明所有 Scale 评测有偏,也不等于 Meta 买下公司。企业尽调应检查数据访问、合同隔离和发布流程。人类专业知识能改善有效性,但服务交付成本与裁决一致性仍是重要商业约束。
Galileo / Cisco
- 所有权/组织
- 2026 年 5 月 22 日完成收购
- 控制点
- 从评测到运行时护栏
Galileo 连接评测、可观测性与运行时护栏,包括小型 Luna 评测器。Cisco 于 2026 年 5 月 22 日完成收购,继续把 Galileo 写成独立公司是错误的。战略机会是把离线质量标准变成成本可承受的生产执行能力,并借助企业分销。成本和延迟主张若未在同等负载下测试,仍只是厂商说法。快速评分器也要校准并明确不确定时的动作,不能替代资源授权、精确批准或已提交副作用的恢复。
Databricks / MLflow
- 所有权/组织
- Databricks 平台与开源 MLflow
- 控制点
- 数据治理下的评测工作流
MLflow 把追踪、评测、评委、提示词与人工反馈连接到熟悉的模型开发流程。Databricks 增加托管运营,并靠近企业数据治理。本站认为,优势来自数据聚集和既有采购,开源 API 则支持可迁移性。要核对准确版本与产品版本,不能认为每个历史发布能力相同。治理良好的数据和模型注册表很有价值,但它们不能决定某个请求中的外部付款或删除是否得到授权。
Microsoft Foundry
- 所有权/组织
- 微软云平台
- 控制点
- 追踪转数据集与发布集成
Microsoft Foundry 把追踪、评测、监控与 CI/CD 质量关卡带入企业开发环境。文档描述了将 Agent 追踪转成整理后的版本化评测数据。商业优势是降低微软体系企业的集成与采购摩擦。专业厂商必须说明额外平台为何足以改善决策来抵消这份摩擦。数据集创建仍需代表性选择和敏感数据控制,自动采集追踪不证明标签正确,也不证明部署的 Agent 具有最小权限。
AWS Bedrock / Google Cloud
- 所有权/组织
- 相互独立的云平台产品
- 控制点
- 托管模型与应用评测
AWS Bedrock 提供模型与 RAG 评测、人工评测选项和模型评委任务。Google Cloud 的 Gen AI 评测文档覆盖模型与应用,包括第三方模型。二者是不同产品,只因云分销形成类似竞争压力而放在一起。既有基础设施与合同让采用更便利,跨供应商中立和可迁移性仍需尽调。无论哪一家,托管执行都不会自动选择正确任务总体或验证评委;公开结果应保留配置、预算和指标版本。
Amazon Bedrock Evaluations ↗https://aws.amazon.com/bedrock/evaluations/ · Google Cloud Gen AI evaluation overview ↗https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview
经济性:标注与可复现环境才是昂贵部分
评测成本不仅是评委模型的 token 账单,还包括任务设计、专家裁决、环境重置、推理、重复试验和随产品变化的维护。厂商只有降低可辩护决策的成本才创造价值,而不只是多产出分数。本站认为,可复用的专家流程、难获得且有代表性的任务、防污染机制和可复现 Agent 环境,比另一个通用评委更可能形成优势。但客户专属标注未必能合法或有效地跨租户复用。所谓数据网络效应必须解释同意、隔离、可迁移性,以及如何在不泄露机密任务的前提下泛化。
技术关键:质量对应三种不同的判定契约
基准测试问模型能否解决声明的任务集;偏好竞技场问抽样用户在特定展示与采样方式下更喜欢哪个回答;部署评测问配置后的系统能否在质量、安全、延迟和成本约束内完成买方流程。三者目标不同。受偏爱的答案可能缺证据,正确答案可能到达太晚,有能力的模型也会被检索或工具集成错误拖垮。采购判断可以综合这些证据,但应先区分各自证明了什么。
公平比较应让两个候选运行同一批案例,并分析配对差异。选择前固定数据集、评分规则、裁判版本和工具权限;最终留出集不能进入优化循环。LLM 裁判需要用经裁定样本校准,检查位置偏差、文风偏差以及与候选共享错误的问题。报告不确定性和任务族分组结果。如果某类高成本失败恶化,总分上升不足以证明改进。真正可辩护的产品是可重复的决策过程,而非孤立的排行榜数字。
未解决的缺口:从分数到安全决策
评委可以稳定但错误,多个评委也可能共享盲点。公开排名反映的语言、任务与表达偏好可能不同于采购者工作负载。Agent 测试还依赖 harness、工具、外部状态、预算和重置政策,因此只用模型名字作为比较单位并不完整。生产验收必须把任务结果与安全不变量结合:不能跨租户访问、不能有未授权副作用、不能暗中升级权限。本站认为,下一个有价值的层是带有明确不确定性和责任归属的证据整合;它要连接专家判断、可复现执行和业务结果,不能把自动代理指标包装成真实标准答案。
收购、资本与战略影响
资本与归属:讨论独立性必须准确
Arena 于 2026 年 1 月 6 日宣布 1.5 亿美元 A 轮。Scale 于 2025 年 6 月 12 日宣布 Meta 投资,Meta 后续披露非投票少数股权,并有 137.9 亿美元计入非流通股权投资。这不是收购整个 Scale,也不是其评测业务的简单售价。Cisco 在 4 月 9 日宣布意向后,于 2026 年 5 月 22 日完成 Galileo 收购。这些事实反映资本与平台兴趣,不反映评测质量。本站将商业关系与所有权视为尽调议题:需要询问客户数据访问、公布规则、模型选择和利益冲突管理,不能仅凭投资或客户关系推断结果有偏。
一手产品与交易来源
- Arena Series A ↗https://arena.ai/blog/series-a ·
- Arena Agent Mode methodology ↗https://help.arena.ai/articles/5432423882-how-to-use-agent-mode ·
- Meta quarterly filing: Scale non-voting minority investment ↗https://www.sec.gov/Archives/edgar/data/1326801/000162828025047240/meta-20250930.htm ·
- Artificial Analysis free and commercial APIs ↗https://artificialanalysis.ai/api-reference ·
- Galileo evaluation and guardrail product ↗https://galileo.ai/ ·
- Arena commercial evaluation launch ↗https://arena.ai/blog/ai-evaluations ·
- Artificial Analysis index methodology ↗https://artificialanalysis.ai/methodology/intelligence-benchmarking ·
- Scale investment and independence statement ↗https://scale.com/blog/scale-ai-announces-next-phase-of-company-evolution ·
- Cisco Galileo acquisition completion ↗https://www.cisco.com/site/us/en/about/corporate-development/acquisitions/galileo/index.html ·
- Databricks current MLflow observability and portability FAQ ↗https://docs.databricks.com/aws/en/mlflow3/genai/tracing/faq ·
- Microsoft Foundry traces to evaluation datasets ↗https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/traces-to-dataset ·
- Amazon Bedrock Evaluations ↗https://aws.amazon.com/bedrock/evaluations/ ·
- Google Cloud Gen AI evaluation overview ↗https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview ·
14 · How AI giants compete through evaluation
Frontier-model competition is partly a competition over what counts as progress. A provider can emphasize mathematical reasoning, coding, agentic task completion, multimodal understanding, latency, price or safety. Those dimensions matter, but selecting the favorable subset can create a misleading overall impression. Read a model launch as a collection of scoped claims rather than a universal ranking.
For every headline comparison, ask whether model versions, prompts, tools, inference budgets and grader settings match. A result with extensive search, retries and a large compute budget does not measure the same product experience as a single low-latency response. If proprietary scaffolding contributes to the score, describe the result as a system result unless the contribution has been isolated.
Providers have real advantages in evaluating their own systems: detailed access to versions, logs and failure modes. They also have incentives to present progress favorably. Independent evaluators can diversify the evidence, but independence is not guaranteed merely by being a separate company. Examine funding, commercial relationships, publication policies, access privileges and the ability to report negative findings.
Enterprise buyers should build a small portfolio of evidence: public capability measurements for breadth, domain-specific held-out tasks for fit, adversarial tests for boundaries, and randomized online experiments for user impact. Require a candidate to improve the dimensions that matter to the business rather than chasing every public leaderboard movement.
The durable competitive advantage may lie in a high-quality feedback loop: representative failures become reviewed tasks, interventions are tested fairly, successful changes are deployed cautiously, and production outcomes verify the hypothesis. This is an engineering interpretation of the market, not a claim that any named company has already solved that loop.
Cloud platforms compete for the evaluation workflow
Microsoft Foundryhttps://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-overview documents evaluation of existing datasets, generated model or agent outputs, and production traces. This is a workflow integration proposition: keeping execution, permissions and evidence near the deployment environment. Buyers should still export a versioned task set and verify that scores remain reproducible outside the portal.
Google Cloud's Gen AI evaluation servicehttps://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview documents evaluating third-party models as well as models in its own ecosystem. This highlights an important separation: the generator being tested need not be the judge. A cross-provider evaluation is not automatically independent, however; check the judge's calibration and version on the target domain.
Amazon Bedrockhttps://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html documents model-evaluation jobs that use a separate LLM to score responses and explain the score. A generated explanation is useful evidence for review, not proof that the score is correct. Keep an adjudicated calibration set, record evaluator configuration and include judge cost in the experiment ledger.
OpenAI and Anthropic also contribute provider-authored audits and engineering guidance, discussed earlier. Across these companies, the competitive axes are model quality, integration, operating cost, access to feedback and trust in measurement. The practical strategy is to keep your evaluation contract portable while using integrations where they reduce work. Do not give the vendor's default rubric authority to redefine your business outcome.
Continue the learning path
Read the companion handbook: evaluation and observability · AI security · AI infrastructure
Learning library · Interview and projects · Editorial standards and corrections
Educational extensions: OpenAI Academyhttps://academy.openai.com/ · Anthropic learning resourceshttps://academy.claude.com/ · Microsoft AI traininghttps://learn.microsoft.com/en-us/ai/.
Primary references are attached to the relevant claims. Conceptual diagrams, synthetic arithmetic and research proposals are original teaching material. Vendor descriptions establish product positioning, not independent proof of performance. Corrections should identify the claim, its source and the proposed replacement.
实践与验证
阶段五:反馈闭环与文末验证
线上反馈延迟且有选择偏差,沉默用户不代表成功。故障样本进入开发或验证集后,不再是独立考试。保留事件→案例→评分规则→运行→上线决定的版本链;替换评分器造成的分数变化不能冒充能力提高。交付包应含任务合同、配对结果、关键切片、评分校准、资源账单和回滚条件。
实践验收集中在这里:先只改变文档ACL,验证越权检索被拒绝;再删除政策,要求拒答。把“30天”改为“300天”,结构评分通过而依据评分失败。复制同一案例十次,不应把独立样本量放大十倍。注入一次越权操作,发布门禁必须阻断;评分器故障必须显示未知。逐项保留预期、实测和解释,最后用未参与修改的新案例验证迁移。
15 · Open research problems worth a serious experiment
External validity. Can an offline score predict a deployment outcome across organizations? Construct several domain-specific task sets, preregister the predicted ordering, and compare it with a later controlled rollout. Report where the ordering reverses. A single positive correlation on one product is not a universal transfer law.
Adaptive contamination. How can a benchmark remain informative when developers repeatedly optimize against it? Compare fixed public tasks, private held-outs and periodically refreshed tasks under a controlled optimization process. Measure performance on a final unseen distribution. The aim is to separate genuine generalization from learning the measurement interface.
Judge robustness. Can a grader resist style changes and adversarial instructions without losing sensitivity to real improvements? Create matched output pairs that preserve correctness but change verbosity, formatting or persuasion. Then create pairs with identical style but different factual quality. The grader should be invariant to the former when irrelevant and responsive to the latter.
Long-horizon reliability. How do errors accumulate across tool interactions? Vary task length while controlling task type, and distinguish exposure from intrinsic deterioration. A longer task creates more opportunities to fail; a higher session-level failure rate does not alone show that per-action reliability worsened.
Evaluation under limited budgets. Which tasks should be labeled or rerun next? Compare uniform sampling with an adaptive policy, but preserve a probability sample for unbiased reporting. An adaptive strategy can find more failures while producing a distorted raw failure rate if selection probabilities are ignored.
Reward hacking in self-improving systems. Does an optimization loop improve the target capability or exploit its grader? Hold a second evaluator and task set outside the loop, impose equal compute budgets, and compare against a non-adaptive baseline. Publish negative results and evidence of evaluator exploitation. A rising training reward is a hypothesis about improvement, not its verification.
16 · A four-week capstone with a defensible release decision
Build a small document-support agent that answers policy questions and can propose—but not execute—an account change. The baseline uses one retrieval method and a fixed prompt. The candidate changes exactly one component. Use synthetic documents and mock tools so no personal data or real transactions are involved.
In week one, define 60–100 tasks across straightforward retrieval, conflicting evidence, missing evidence and unauthorized requests. Write a rubric and deterministic postcondition checks. Reserve an untouched test partition. Annotate a small calibration set with two reviewers and resolve disagreements by revising the rubric rather than averaging incompatible interpretations.
In week two, implement the harness with task IDs, seeds where applicable, budget limits, artifacts and explicit infrastructure-error states. Run baseline and candidate on the same tasks. Repeat a subset to estimate execution variance. Add a local report showing paired outcomes, critical slices, latency and cost per accepted result.
In week three, audit failures. Test whether the judge can be manipulated by candidate text. Perform an ablation that restores the baseline component, and a budget-matched comparison that prevents extra compute from masquerading as algorithmic improvement. Add discovered failures to the development set while keeping the final test untouched.
In week four, make the release decision under a preregistered rule. A strong report includes the task distribution, exact versions, uncertainty, failed examples, resource costs, safety constraints and reasons to defer deployment. The reference success criterion is not “candidate wins.” It is “another engineer can reproduce and critically evaluate the conclusion.”
For a research extension, measure whether the offline decision predicts a simulated deployment distribution with different proportions of difficult tasks. Explain the discrepancy when distribution shift changes the conclusion.
17 · Original interview practice and scoring
These are original practice questions, not reported questions from any company. Use the same 1–5 progression for each: 1 identifies the relevant concept; 2 gives a correct mechanism; 3 applies it to the concrete example; 4 proposes a sound experiment with controls; 5 explains assumptions, failure cases and the limits of the conclusion.
1. Candidate B wins 55% of public preference comparisons. Should you replace the support model?
Hint 1: Identify the population and outcome being measured.
Hint 2: Distinguish preference from policy correctness and authorization.
Reveal answer and analysis
Answer: Not on that evidence alone. Run a paired comparison on representative support tasks with correctness, policy and safety graders, then examine uncertainty and cost. A strong answer also checks the public comparison's confidence interval, category and versions, and proposes a guarded online experiment after offline gates pass.
2. Why can 90% judge agreement still be unacceptable?
Hint 1: Consider an imbalanced dataset.
Hint 2: Ask which disagreement type is costly.
Reveal answer and analysis
Answer: If almost all examples are acceptable, a judge that passes everything can achieve high agreement while missing every dangerous failure. Inspect class-conditional errors, calibrate on consequential cases and audit false passes. A top answer explains why agreement with one imperfect annotator is not ground truth and defines an adjudication process.
3. An agent succeeds at least once in five runs but often fails the first run. Which metric matters?
Hint 1: What attempts can the product afford?
Hint 2: Can it identify the successful attempt reliably?
Reveal answer and analysis
Answer: Pass@5 is relevant only if five attempts and reliable selection are available. For a single-shot workflow, first-attempt success and failure severity matter. For repeated dependable execution, measure all-attempt reliability and task-level variance. Include latency, cost, correlated failures and side effects from unsuccessful attempts.
4. A retrieval change raises answer quality. How do you establish why?
Hint 1: Separate evidence availability from generation.
Hint 2: Match the context and compute budgets.
Reveal answer and analysis
Answer: Measure evidence recall, answer support and correctness on the same tasks, keeping the generator fixed. Compare retrieval-only ablations under equal context budgets. Inspect whether gains concentrate on previously missing evidence and test distraction and stale-document cases. A high-scoring answer avoids claiming causality from aggregate correlation alone.
5. A self-improving agent raises its own benchmark score every day. What would convince you it improved?
Hint 1: Identify what the agent can modify or observe.
Hint 2: Keep independent evidence outside the loop.
Reveal answer and analysis
Answer: Freeze an independent evaluator and held-out tasks, compare under equal resource budgets, and check whether gains transfer to new tasks and robust behavior. Audit grader exploitation and contamination. The strongest answer defines a preregistered stopping rule, preserves failed runs and distinguishes increased capability from increased search effort.
Reproduce the arithmetic: a complete local reference
Imagine evaluating a replacement support agent on twenty matched tasks. Twelve tasks pass under both versions, four pass only under B, two pass only under A, and two fail under both. Model A scores 14/20 and B scores 16/20. We resample task pairs, keeping the relationship between outcomes intact. The resulting interval is wide: this demonstration does not establish that B is better in the population. The individual Wilson interval is a separate description of B's success rate; overlap between two marginal intervals is not the paired comparison test.
Here n is the number of independent tasks and outcomes are binary. The second expression is the exact one-sided upper confidence bound on failure probability after zero observed failures, assuming independent identically distributed Bernoulli trials; alpha = 0.05 gives 95% confidence. Correlated tasks or an unrepresentative sample invalidate that interpretation.
Revisit the opening score-and-uncertainty diagram. Its error bars are marginal Wilson intervals; the paired interval below addresses the difference directly.
The bootstrap is a percentile approximation, not a guarantee of calibrated coverage with just twenty examples. If multiple prompts belong to one customer or document, resample customers or documents. More repeated generations per item measure stochastic variability; they do not manufacture more independent customers.
Try the denominator locally
No input leaves your browser. This is an arithmetic illustration, not a production risk estimate.
Runtime: Python 3.10 or newer; standard library only. The evaluation bootstrap uses seed 17 and 10,000 resamples. Observability arithmetic is deterministic and needs no seed. These programs make no network or model calls.
Download the complete Python reference
python3 evaluation-audit.py"""Synthetic paired evaluation; Python 3.10+, standard library, seed 17.
No API calls. Items, rather than individual judgments, are bootstrap units.
"""
import math
import random
import json
from statistics import mean
def wilson(successes, n, z=1.959963984540054):
if not 0 <= successes <= n or n <= 0:
raise ValueError('Require 0 <= successes <= n and n > 0')
p = successes / n
den = 1 + z*z/n
center = (p + z*z/(2*n)) / den
radius = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / den
return center-radius, center+radius
def paired_bootstrap(a, b, repetitions=10000, seed=17):
if len(a) != len(b) or not a:
raise ValueError('Require nonempty matched items')
if not isinstance(repetitions, int) or repetitions < 100:
raise ValueError('Use at least 100 bootstrap repetitions')
if any(v not in (0, 1) for v in a + b):
raise ValueError('This example requires binary outcomes')
rng = random.Random(seed)
differences = [y-x for x, y in zip(a, b)]
means = sorted(mean(rng.choices(differences, k=len(a)))
for _ in range(repetitions))
return mean(differences), means[int(.025*repetitions)], means[int(.975*repetitions)]
def grade_response(raw, allowed_sources, expected_status):
"""Original deterministic contract grader, NOT a semantic truth judge.
Candidate text is parsed as data; never eval(), execute(), or prompt a judge.
"""
try:
response = json.loads(raw)
except (json.JSONDecodeError, TypeError):
return {"schema": False, "evidence_ids": False, "status": False}
schema = (isinstance(response, dict)
and set(response) == {"answer", "sources", "status"}
and isinstance(response["answer"], str)
and isinstance(response["sources"], list)
and all(isinstance(x, str) for x in response["sources"])
and response["status"] in ("answer", "abstain"))
if not schema:
return {"schema": False, "evidence_ids": False, "status": False}
return {"schema": True,
"evidence_ids": (all(x in allowed_sources for x in response["sources"])
and (response["status"] != "answer" or bool(response["sources"]))),
"status": response["status"] == expected_status}
valid = '{"answer":"The policy says 30 days.","sources":["policy-v3"],"status":"answer"}'
assert all(grade_response(valid, {"policy-v3"}, "answer").values())
assert not grade_response('not JSON', set(), 'abstain')['schema']
assert not grade_response(valid, {"policy-v4"}, "answer")['evidence_ids']
assert not grade_response(valid, {"policy-v3"}, "abstain")['status']
# A plausible-looking lie can pass these structural checks. Add human/semantic
# review against the actual source; do not call this contract pass "correctness".
lie = valid.replace('30 days', '300 days')
assert all(grade_response(lie, {"policy-v3"}, "answer").values())
# Same 20 synthetic tasks: 12 both pass, 4 only B, 2 only A, 2 neither.
a = [1]*12 + [0]*4 + [1]*2 + [0]*2
b = [1]*12 + [1]*4 + [0]*2 + [0]*2
assert sum(a) == 14 and sum(b) == 16
low, high = wilson(sum(b), len(b))
delta, lower, upper = paired_bootstrap(a, b)
assert lower <= 0 <= upper # small sample does not establish superiority
assert paired_bootstrap(a, a) == (0, 0, 0)
assert wilson(0, 20)[0] >= -1e-12
assert math.isclose(1 - .05**(1/100), .029513049607039932)
# Trial independence is an assumption, not a property guaranteed by a model.
assert math.isclose(1-(1-.8)**3, .992)
assert math.isclose(.8**3, .512)
print(f'A: {mean(a):.1%}; B: {mean(b):.1%}')
print(f'B Wilson 95% interval: [{low:.3f}, {high:.3f}]')
print(f'Paired delta: {delta:.1%}; bootstrap 95% interval: [{lower:.3f}, {upper:.3f}]')
print('Zero failures in 100: one-sided 95% upper bound 2.95%')
print('All evaluation checks passed; synthetic data, not model performance.')
Expected output:
A: 70.0%; B: 80.0%
B Wilson 95% interval: [0.584, 0.919]
Paired delta: 10.0%; bootstrap 95% interval: [-0.150, 0.350]
Zero failures in 100: one-sided 95% upper bound 2.95%
All evaluation checks passed; synthetic data, not model performance.Test the failure cases: empty samples are rejected; the evaluation comparison of a model with itself has zero difference; the trace algorithm rejects cycles. Next add clustered tasks or missing spans, and document exactly which assumptions stop holding. Unit checks validate this implementation, not a deployed model.