AI evaluation: from measurements to decisions, research and market power
A field guide to traditional ML, LLM and agent evaluation: statistics, Arena, engineering, research frontiers, companies and acquisition evidence.
Jump to the runnable reference and tests
출처 및 버전 정보
Contents · chapters, research and reference code
이 글의 목차
평가: 시스템 출시 여부 판단
이 사례는 청구서 도우미를 따라갑니다. 테넌트 범위의 문서를 읽고 결제를 제안하지만 실행에는 별도의 정확한 권한이 필요합니다. 시나리오와 실험은 합성 교육 자료이며 실제 배포 사례가 아닙니다.
먼저 결정할 것: 청구서 에이전트를 배포해도 되는가
합성 예시의 도우미는 청구서를 설명하고 테넌트 자료를 읽으며 결제를 제안합니다. 실제 실행에는 정확한 작업에 대한 독립적 승인이 필요합니다. 문장이 좋아지고 문맥이 늘었다고 조합 전체를 배포할 근거가 되지는 않습니다. 대상 업무 모집단과 분석 단위를 정합니다. 재시도 여섯 번도 고객 사례 하나이며 모든 시도의 비용과 위험을 포함합니다.
평가 계약은 모델, 프롬프트, 문서 스냅샷, 도구, 예산, 제한 시간, 채점 규칙, 제외 조건을 고정합니다. 근거 있는 답 또는 적절한 보류, 시간, 권한을 따로 보고하며 유창성으로 권한 위반을 상쇄하지 않습니다. 채점기 중단은 알 수 없음으로 남기고 몰래 제외하지 않습니다. 미완료는 사전 규칙으로 판단합니다. 근거가 부족하면 배포하지 않는 것도 유효한 결정입니다.
1단계: 업무를 재현 가능한 환경으로 구성하기
정상 청구서, 정책 누락, 버전 충돌, 오래된 잔액, 타 테넌트 자료, 수취인 변경, 승인 철회, 중복 제출을 포함합니다. 초기 상태와 허용 종결 상태, 금지 부작용을 명시합니다. 결제 준비는 결제 실행과 다릅니다. 교육용 기준은 합성 문서와 모의 도구만 사용합니다.
고객, 문서 계열 또는 시간으로 개발·검증·최종 보류 집합을 나눕니다. 한 청구서의 표현 열 개는 보통 독립 사례 열 개가 아닙니다. 올바른 근거를 직접 주는 조건으로 생성 문제를 분리하고 근거 누락 조건으로 보류를 확인합니다. 운영의 90%가 쉬운 사례이고 10%가 어려운 사례라면 반반인 진단 집합으로 전체 성능을 바로 추정하지 말고 층별 결과와 가중치를 제시합니다.
2단계: 관측을 방어 가능한 판단으로 바꾸기
스키마, 허용 문서, 결제 인자, 원장 최종 상태, 중복 효과, 승인 일회 소비는 결정적 검사로 판정합니다. 의미적 근거 관계에는 사람이나 보정한 모델 채점기를 씁니다. 호출 기록은 시도의 증거이고 업무 상태가 결과의 증거입니다. 실제 인용도 주장을 뒷받침하지 않을 수 있습니다.
독립 검토 표본으로 잘못 통과시킨 경우와 잘못 탈락시킨 경우를 나눠 측정합니다. 합성 1000개 중 오류 50개라면 항상 통과시키는 채점기는 정확도 95%이면서 오류를 모두 놓칩니다. 순서, 길이, 설득 문구를 바꿔 편향을 확인합니다. 비교 전에 채점기 버전을 고정하고 중단 시 미상 상태, 라벨 범위, 유형별 오류를 보고합니다.
3단계: 변수를 통제하고 불확실성을 해석하기
동일 사례와 자원·행동 예산으로 기준과 후보를 비교합니다. 반복은 확률적 변동을 추정하지만 고객 수를 늘리지 않습니다. 모델 고정 검색 비교, 검색 고정 모델 비교 후 결합을 확인해야 상호작용이 보입니다. 추가 문맥과 재시도를 별도로 기록합니다. API 시드는 버전 간 완전한 결정성을 보장하지 않습니다.
합성 독립 사례 200개에서 둘 다 통과 150, 후보만 20, 기준만 10, 둘 다 실패 20이면 기준 80%, 후보 85%, 차이 5%포인트입니다. 결과가 다른 30쌍이 불확실성 계산에 중요합니다. 교육용 정규 근사의 표준오차는 약 2.72%포인트이고 95% 구간은 0을 약간 포함합니다. 고객 내 상관이 있다면 개별 응답 대신 고객을 재표집합니다.
4단계: 점수가 높아도 거부할 수 있는 배포 기준
결과 전에 최소 유용 개선, 허용 불확실성, 비용·지연 한도와 강제 안전 조건을 정합니다. 평균 점수 상승으로 무단 결제를 보상할 수 없습니다. 적대 사례 100개에서 실패가 없어도 위험 0은 아닙니다. 독립 베르누이 가정의 단측 95% 상한은 약 2.95%이며 비무작위 공격 집합으로 운영 발생률을 추정할 수 없습니다.
부작용 없는 섀도 실행 후 외부 권한 통제를 유지한 제한 카나리로 진행합니다. 동시기 집단을 비교하고 고정 관찰 기간이나 유효한 순차 분석을 사용합니다. 유리한 순간에 멈추지 않습니다. 롤백 시 이미 확정한 업무는 대조·정산해야 하며 재실행하지 않습니다. 차단 조건마다 책임자와 복구 근거가 필요합니다.
How the four disciplines fit together
Security determines which actions are permitted. Infrastructure executes work within capacity and failure constraints. Observability preserves evidence of what happened. Evaluation tests whether a change improves the outcome under a declared contract. None substitutes for the others.
메커니즘, 연구, 시장 배경 — 아래 참고 자료는 영어입니다
The following chapters provide the mathematical examples, experiments, research papers and dated company evidence behind the operational story. Return to the lifecycle when deciding where each mechanism belongs.
01 · Evaluation begins with a decision, not a leaderboard
Imagine that an engineering team is replacing the model behind a customer-support assistant. The new model wins more public head-to-head comparisons and writes more polished answers. Yet its first production week produces more incorrect refunds. Both observations can be true. A public preference ranking measures how a particular population judges particular responses; a refund workflow requires correct policy interpretation, authorization and state transitions. The mistake is not necessarily in either measurement. It is in treating one measurement as a substitute for another.
Definition. An AI evaluation is a specified procedure for gathering evidence about a system's behavior under stated conditions and using that evidence to support a decision. A benchmark is a reusable collection of tasks and measurement rules. A metric summarizes observations. A grader turns an output, action or environment state into a judgment. A harness controls execution and records the conditions. These objects are related, but none is interchangeable with the others.
The decision should be written before collecting scores: “Release candidate B only if it improves successful resolution on our target tasks without increasing unauthorized actions, while meeting our latency and cost budgets.” This statement identifies the target population, the benefit, the constraints and the deployment boundary. It also exposes missing knowledge: what counts as successful resolution, how frequently dangerous cases occur, and how confidently we can measure the difference.
Before this lesson, you should understand a sample mean as an estimate of a population property, a conditional probability as a statement restricted to specified conditions, and a train/test split as a barrier against adapting to the answers. You do not need to have memorized advanced statistics. You do need to recognize that running the same task repeatedly measures execution variability, while collecting new tasks measures a different source of uncertainty.
By the end, you should be able to design a release gate, interpret an Arena-style comparison, calibrate an automated judge, build an auditable evaluation pipeline, and explain why a high score can coexist with an unsuitable product. The examples in this chapter are original teaching examples unless explicitly attributed; they are not measurements of commercial models.
Choose a reading route
First encounter: read the definitions and worked examples, then predict the local experiment result before changing its input. Engineering implementation: focus on the data contract, architecture and Python reference, then write one failure test of your own. Research reading: inspect the cited paper versions, challenge their evaluation assumptions, and choose one open problem with a falsifiable baseline.
02 · Traditional ML, LLMs and agents: what changes?
Traditional supervised evaluation often starts with a fixed input, a reference label and a loss function. A classifier receives an image and predicts a class; a forecasting model predicts a number. Accuracy, precision, recall, calibration and error by subgroup can be useful because the target is comparatively explicit. Even here, labels can be wrong, class distributions can shift, and the cost of a false negative can differ from that of a false positive. Traditional evaluation was never automatically simple or unbiased.
For a language model, several outputs may be acceptable. A concise explanation and a longer explanation can both be correct; exact string matching can reject one for irrelevant reasons. Meanwhile, an apparently correct conclusion may contain fabricated evidence. Evaluation therefore separates dimensions such as factual correctness, instruction following, evidence support, completeness and usefulness. Combining them into one number is a policy choice that requires explicit weights.
An agent adds actions and environmental state. It can finish a task through different trajectories, use extra tools, encounter failures, recover, or damage something unrelated to the task. A final paragraph saying “Done” is not evidence that the task was completed. A database query, repository diff or sandbox state may be the appropriate verifier. The evaluated object is now the model plus its prompt, tool definitions, memory, permissions, retry logic and surrounding application.
Three distinctions prevent many misleading comparisons. First, model evaluation isolates a model as far as possible; system evaluation tests the assembled product. Second, capability evaluation asks what can be accomplished under a budget; reliability evaluation asks how consistently the behavior occurs. Third, offline evaluation estimates performance on controlled tasks, whereas online experimentation estimates effects on users under an actual deployment.
None replaces the others. A model can be capable but unreliable, a reliable product can depend on expensive scaffolding, and a statistically positive online result can conceal a severe safety regression. Keep the axes visible until the decision requires an explicit tradeoff.
03 · Write an evaluation contract
A useful evaluation contract names the task distribution, unit of analysis, execution budget, grading rubric, exclusion policy and release rule. For a support agent, a unit might be one customer case, not one model call. For a coding assistant, it might be one repository issue, not every generated patch. If a case produces ten retries, treating those retries as ten independent customers exaggerates the amount of evidence.
Make the task set versioned and immutable during a comparison. Record the repository snapshot or document corpus, model identifier, prompt hash, tool schema, inference settings, maximum actions, wall-clock timeout and grader version. Record unavailable dependencies and API errors as explicit outcomes. Quietly deleting inconvenient failures changes the question being answered.
Use three task collections with distinct roles. A development collection helps engineers investigate errors. A validation collection supports regular tuning and model selection. A held-out test collection supports a final estimate and should not become a daily feedback channel. When the team has repeatedly inspected the held-out answers, it has effectively converted that set into development data. Refreshing a test set is not a failure; pretending it remains untouched is.
A contract also specifies the aggregation rule. Suppose the task mix is 80% simple retrieval and 20% difficult account changes. Averaging those tasks equally may answer a different question from weighting them by production traffic. Conversely, production-weighting alone may hide a rare but unacceptable unauthorized transfer. Report an overall estimate and decision-critical slices, then apply hard constraints separately where appropriate.
Finally, define inconclusive outcomes. “Insufficient evidence to release” is a legitimate result. A good process does not force every experiment into a winner and a loser. It preserves uncertainty until the expected value of further testing is lower than the cost of waiting.
04 · Uncertainty, paired comparisons and repeated trials
A score is an estimate, not a physical constant. If candidate A succeeds on 80 of 100 tasks, the observed success rate is 80%, but a different sample could produce another value. A confidence interval communicates sampling uncertainty under a specified model. It does not account automatically for mislabeled tasks, a biased sample, contaminated training data or a broken grader.
Compare two systems on the same tasks whenever possible. For each task, calculate the difference between candidate and baseline. Paired comparisons remove some variation due to task difficulty: a hard task is hard for both systems. Bootstrap whole tasks or higher-level clusters such as users or repositories when those are the independent units. Do not independently resample individual tool calls from the same trajectory.
Repeated trials answer a second question: how much does behavior vary when the task is held fixed? Keep the task identifier and trial identifier separate. A system that solves each of ten tasks in one lucky run out of ten has different production behavior from one that reliably solves nine tasks and always fails the tenth. Their pooled success rates can conceal these patterns.
Pass@k describes the chance that at least one of k attempts succeeds under the evaluation's sampling and selection assumptions. It is relevant when a user can actually generate and identify a good candidate. It does not mean the first answer is reliable. A requirement that all k repeated attempts succeed is a different reliability criterion, sometimes written pass^k. Under independent identical success probability p, these teaching models become 1−(1−p)^k and p^k respectively; independence is often unrealistic in deployed systems.
For rare failures, zero observations is not proof of zero risk. Under independent Bernoulli trials, the rough “rule of three” gives a 95% upper bound near 3/n after zero failures in n trials. With 300 clean trials, a failure rate around 1% is still compatible with this approximation. Correlated tasks or an unrepresentative test population weaken that reassurance further.
At least one success is not dependable repetition
With p = 0.8 and k = 3 independent attempts, these values are 0.992 and 0.512. They answer different product questions. For sampled code completions, a common finite-sample pass@k estimator with n generated candidates and c passing candidates is one minus the ratio of combinations shown below. It estimates oracle availability; an imperfect selector can perform worse.
When fewer than k failures exist, the numerator is zero. For n = 10, c = 2 and k = 3, the estimate is 1 − 56/120 = 0.5333. Keep failed, truncated and unparseable attempts in the declared denominator. This estimator does not remove task selection bias.
Source for the sampled-completion estimator: Evaluating Large Language Models Trained on Codehttps://arxiv.org/abs/2107.03374v2, first submitted July 7, 2021; v2 July 14, 2021. Versioned PDFhttps://arxiv.org/pdf/2107.03374v2. The numerical examples here are original and do not reproduce its model scores.
05 · Build graders, then evaluate the graders
Choose the strongest feasible evidence for each requirement. If a generated file must parse, execute a parser. If a balance must remain unchanged, inspect the balance. If an answer must cite an accessible source, verify the source and its relationship to the claim. A language-model judge is useful for semantic judgments that cannot be captured cheaply by deterministic checks, but it should not replace a stronger available verifier.
A rubric should define observable differences between scores. “Excellent” and “poor” are adjectives, not measurement rules. For an evidence-grounded answer, a rubric might distinguish unsupported central claims, partially supported claims with missing qualifications, and a complete answer whose important claims are supported by the supplied documents. Include examples near the decision boundary, not only obvious successes and failures.
Calibrate the judge on an independently annotated set. Measure agreement and inspect the disagreement matrix, especially false passes on consequential errors. An overall agreement score can look strong when almost all examples pass. Include adversarial examples, long fluent wrong answers, concise correct answers, irrelevant citations and attempts to instruct the judge from inside the candidate output.
The foundational MT-Bench and LLM-as-a-judge paperhttps://arxiv.org/abs/2306.05685v4 studies position, verbosity and self-enhancement biases. Its reported human agreement belongs to its tested models and tasks, not every future judge. The paper first appeared on 9 June 2023; the linked v4 is dated 24 December 2023. Read the PDFhttps://arxiv.org/pdf/2306.05685v4.
In your own protocol, randomize answer order, reverse a subset of pairs, blind provider names, constrain the rubric, and audit failures manually. Preserve judge versions and outputs so a score can be reproduced. Treat a judge upgrade as a measurement-system change: regrade a bridge set before interpreting a historical trend as model improvement.
A passing schema can still contain a false answer
The complete Python reference below includes a contract grader for an answer, its source identifiers and an answer/abstain status. It rejects malformed JSON and references outside the allowed source set. A deliberate counterexample changes “30 days” to “300 days” while preserving the schema and source ID: the contract grader still passes. This test is intentional. Structural validity, citation existence and semantic support are different properties; a real application needs a separately calibrated check against source content.
Keep grader failure distinct from candidate failure. An unavailable judge should produce an unknown evaluation state and a retry/review policy, not a fabricated zero or perfect score. Never execute a candidate response as Python merely to parse its structure.
06 · Arena: preference measurement and its boundaries
Arena, formerly LMArena and Chatbot Arena, made pairwise human preference a prominent part of model evaluation. In a blind comparison, a person sees responses to a prompt and expresses a preference before model identities are revealed. The statistical problem is to infer relative strengths from a sparse network of comparisons, not simply to count total wins across unequal opponents.
The Bradley–Terry model represents the probability that model A is preferred to B as a logistic function of their strength difference. The strengths are relative: adding the same constant to every strength leaves pairwise probabilities unchanged. A model with a higher estimated strength is more likely to win under the observed comparison process. This is not an absolute percentage of “intelligence.”
The original Chatbot Arena paperhttps://proceedings.mlr.press/v235/chiang24b.html is an ICML 2024 publication, with its official proceedings PDFhttps://raw.githubusercontent.com/mlresearch/v235/main/assets/chiang24b/chiang24b.pdf. The research establishes a concrete pairwise evaluation framework; it does not establish that public preference is sufficient for every deployment.
Read rankings with their uncertainty, category, date and methodology. Differences can be too small to distinguish confidently. The population of people submitting prompts is not necessarily your customer population. Fluent presentation can affect preference. Different categories can reverse a ranking. Connected comparison coverage matters: a model tested against a narrow or atypical opponent set requires careful estimation.
Arena's 14 July 2026 factuality announcementhttps://arena.ai/blog/factuality-in-arena describes a separate, initially optional Text/Search view combining preference with claim-level factuality audits. Its stated default factuality weight is 25%. This is a useful illustration of an evolving construct: changing the mixture changes what the ranking means. Do not splice the adjusted score into an older preference-only time series without explaining the break.
Use Arena as external evidence and a source of candidate models. Then run a matched evaluation on your own tasks. The engineering question is not “Who is number one?” but “Does this candidate improve our outcome under our constraints?”
Work through the pairwise model
The strengths theta are dimensionless log-odds parameters in this unscaled formulation. A difference of ln(3) gives a 75% predicted preference probability, not a 75% task accuracy. Fix one strength or constrain their sum to remove the arbitrary shared offset. Estimate strengths from the comparison graph, and inspect task and voter dependence before trusting naive independent-vote uncertainty. Ties require an explicit modeling policy rather than silently counting every tie as a decisive win.
07 · Current frontier: measuring validity, contamination and alignment
The research frontier increasingly concerns whether measurements justify their claims. Measuring what Mattershttps://arxiv.org/abs/2511.04703v1, submitted 3 November 2025 and formally published in the NeurIPS 2025 Datasets and Benchmarks trackhttps://proceedings.neurips.cc/paper_files/paper/2025/hash/1967e0fc3aa6cbbace562f5cb8e3954e-Abstract-Datasets_and_Benchmarks_Track.html, reviews 445 benchmarks with 29 expert reviewers. Its central concern is construct validity: whether the task and metric represent the phenomenon claimed. PDFhttps://arxiv.org/pdf/2511.04703v1. A practical implication is to draw the chain from real-world requirement to task, observation and score, then challenge each link.
Contamination is another threat. On 23 February 2026, OpenAI reported problems with SWE-bench Verifiedhttps://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/, including flawed tests and contamination. Its 59.4% flawed-test finding applies to an audited difficult subset of 138 problems, not automatically the entire benchmark. This is a provider-authored audit, and its scope matters. The lesson is to inspect failed cases and grading assumptions, not to assume every low score is a model defect.
Arena's 8 October 2026 Alignment Indexhttps://arena.ai/blog/ai-alignment-index describes three observable signals: unauthorized action, false attribution and deceptive completion. It uses judged evidence and adjusts for conversation length. This is an operational measurement of selected behaviors, not a proof of complete alignment. A narrow, clearly defined indicator can be useful precisely because its limits are visible.
These developments point toward an important discipline: evaluate the evaluation. Test whether a benchmark responds to genuine improvements, whether superficial changes can raise its score, whether labels are reproducible, and whether the score predicts a meaningful deployment outcome. A benchmark that is difficult but unrelated to the product is not automatically valuable.
08 · RAG evaluation: separate retrieval from answering
A retrieval-augmented system can fail before the language model sees the evidence. If the relevant policy paragraph is absent from the retrieved context, improving answer wording will not solve the underlying problem. Conversely, perfect retrieval does not guarantee that the model uses the evidence correctly. Evaluate the pipeline at its interfaces.
For retrieval, define relevance at the unit you actually retrieve: document, chunk or passage. Measure whether the required evidence appears within the allowed context budget. Recall@k is meaningful only with a suitable relevance set and a specified k. A large k can improve recall while increasing latency, cost and distracting context. Test reranking and chunking under the same token budget rather than comparing an inexpensive short context against a much larger one.
For generation, separate answer correctness from support. An answer can be factually correct but unsupported by the retrieved material; it can quote a source accurately while answering the wrong question. Include citation validity, entailment of important claims, completeness, abstention when evidence is insufficient, and handling of conflicting or outdated documents.
Use counterfactual tests. Remove the decisive passage and check whether the system acknowledges uncertainty. Replace a policy with a newer version and inspect whether the answer changes appropriately. Insert an irrelevant but persuasive paragraph and test resistance to distraction. Add a malicious instruction inside a retrieved document and verify that the system treats it as data rather than authority.
A fair ablation changes one component at a time: retrieval method, chunk size, reranker or generator. Keep the corpus snapshot, questions, answer rubric and budget fixed. The result should explain a mechanism—such as improved evidence recall on long policies—rather than merely report a higher aggregate number.
09 · Agent evaluation: outcomes, permissions and recovery
For an agent, evaluate both the final state and the path constraints. A travel assistant that books the right flight using an unauthorized payment method has not succeeded. A coding agent that fixes a test by deleting the test has not solved the intended problem. The verifier needs to inspect meaningful invariants, not only whether the final response sounds successful.
Construct a sandbox with explicit initial state, permitted actions and expected postconditions. Record tool inputs and outputs, environment changes, retries, timeouts and side effects. A task should define whether asking for clarification is acceptable, when human approval is required, and what should happen when a tool fails. Otherwise, the grader may punish sensible behavior or reward recklessness.
Anthropic's 9 January 2026 agent-evaluation guidehttps://www.anthropic.com/research/demystifying-evals-for-ai-agents emphasizes tasks, trials, graders, transcripts and outcomes. The engineering takeaway is to inspect trajectories when scores behave unexpectedly and to combine grader types. The guide is practitioner evidence, not a universal benchmark result.
Security evaluation needs an attacker model. Specify what the attacker controls: a retrieved page, email body, repository file or tool response. State the protected boundary and observable violation. Measure attack success and benign-task completion together; a system that refuses every request may look secure while being useless. Keep attack development separate from the final test set.
Recovery deserves its own tasks. Inject a transient timeout, stale credential, contradictory tool response or partial write, then verify whether the agent retries safely, rolls back, asks for help or stops. Do not assume that a successful rerun demonstrates a reliable repair policy. Compare against a retry-only baseline and account for extra compute.
For recursive self-improvement, freeze the evaluator outside the optimization loop. If the same system proposes changes, rewrites the tests and judges its own success, a rising score may reflect adaptation to the measurement process. Independent holdouts, resource budgets and explicit stopping rules are essential.
10 · Multimodal and physical AI need different evidence
Multimodal evaluation is not text evaluation with an image attached. A visual system may need to locate a small object, read a chart, distinguish a temporal event or combine speech with a scene. A text-only final answer can hide whether the model used the relevant modality or guessed from linguistic priors.
Design modality-specific counterfactuals. Change the number in a chart while leaving the question unchanged. Shift an event to a different video timestamp. Replace spoken words while preserving background noise. Mask the decisive region of an image. A system that gives the same answer after the decisive evidence changes may be relying on the wrong signal.
Report input resolution, frame sampling, audio duration, preprocessing and latency budgets. A model receiving every video frame is not being compared fairly with one receiving a few selected frames unless that resource difference is part of the question. Human ratings of aesthetics or naturalness should be separated from task correctness and accessibility.
Physical AI adds dynamics and safety. A robot policy evaluated only in simulation may exploit simulator artifacts or fail under sensor noise, wear, contact variation and unexpected objects. Report the distribution of environments, reset procedure, intervention rate, task completion, collision or constraint violations, and recovery behavior. A successful demonstration is not a reliability estimate.
The broader principle is to preserve the causal connection between the evidence and the claimed ability. If the claim is spatial reasoning, eliminate shortcuts that can solve the task without spatial evidence. If the claim is robust control, vary the physical conditions rather than replaying one favorable trajectory.
11 · An engineering architecture for continuous evaluation
A production evaluation system has four separable layers: a versioned task store, an execution harness, grading services and a decision/reporting layer. Keeping these boundaries explicit makes it possible to rerun old outputs through a new grader, compare a new model under an unchanged harness, and diagnose whether a regression came from the product or the measurement system.
The task store contains inputs, references, metadata, permissions and provenance. The harness creates isolated environments, enforces budgets and captures artifacts. Graders produce dimension-specific results with evidence. The reporting layer aggregates by task slice, uncertainty and cost, then applies release rules. Every result should be traceable to the exact versions that produced it.
Run inexpensive deterministic checks on every change. Run a representative regression suite in continuous integration. Schedule larger repeated-trial and adversarial evaluations when a model, prompt, retrieval index or tool changes materially. Expensive human review should focus on ambiguous or consequential cases rather than being allocated uniformly without purpose.
Connect evaluation to observability without confusing them. Observability records what happened in production: traces, errors, latency, usage and feedback. Evaluation applies a criterion to selected behavior. A trace is evidence for debugging; it is not automatically a quality label. A useful feedback loop turns appropriately redacted production failures into reviewed regression tasks, while keeping a separate holdout for unbiased assessment.
Protect the measurement pipeline itself. Candidate outputs are untrusted data and can contain instructions aimed at the judge. Sandbox executable submissions, restrict network access, prevent test leakage and log grading exceptions. A failed grader should be reported as unknown or infrastructure failure, not silently converted into a model pass or fail.
산업 구도: 제어 지점, 구매자, 경쟁
산업 근거 확인일: 2026년 11월 8일. 제품 문서는 기능 제공을 나타내며 독립 측정 효과를 입증하지 않습니다. 경쟁 구도와 기회는 Token Lab의 분석입니다. 합성 예제는 기업의 실측 결과가 아닙니다.
평가 산업: 증거를 파는 기업과 결정을 소유하는 기업
2026년 11월 8일 기준입니다. AI 평가는 단일 순위표 사업이 아니라 여러 시장입니다. 모델 연구소는 능력 피드백을, 앱 팀은 회귀 절차를, 위험 부서는 감사 가능한 승인 증거를, 인프라 구매자는 비용·성능 비교를 원합니다. 한 시장에서 강해도 다른 시장을 대체하지 못합니다. 어떤 조건에서 어떤 결정을 뒷받침하며 증거가 틀리면 누가 책임지는지가 핵심입니다. 공개된 제품·거래와 상업적 유인에 대한 분석을 구별합니다. 투자금이나 광고는 평가 정확도의 독립 증거가 아닙니다.
네 가지 결정과 네 가지 증거 제품
공개 벤치마크는 모델 후보를, 비공개 업무 데이터는 출시를, 안전 평가는 위협을, 운영 결과 연구는 실제 업무 개선을 판단합니다. 서로 보완적입니다. 송장 도우미가 문체 선호에서 이겨도 무단 공급업체 정보를 사용할 수 있습니다. AI 심사기를 광고하는지가 아니라 어떤 결정을 개선하는지로 기업을 분류해야 합니다. 구매자는 모집단, 작업, 승인 기준, 실패 비용, 갱신 빈도를 먼저 정한 뒤 사람 패널, 코드 검사, 모델 심사기, 관리 서비스를 비교해야 합니다.
참조 아키텍처: 독립적인 출시 판단
순위 수집이 아닌 출시 제어 아키텍처입니다. 저장소에서 기준과 후보 버전을 고정합니다. 고정 사례와 맞춘 도구 권한으로 격리 실행하고 실패와 비용을 포함한 원본 관측을 저장합니다. 버전별 평가기와 사람 판정은 별도 근거로 집계됩니다. 미리 선언한 품질·위험 기준으로 제한적 카나리를 결정합니다. 최종 홀드아웃은 후보 최적화와 격리하고 평가 절차에 따라 사용합니다.
개발과 최종 테스트 피드백을 구분합니다. 홀드아웃으로 반복 선정하면 개발 데이터가 됩니다. 후보는 과제, 평가 코드, 점수와 승격 규칙을 수정할 수 없습니다. 구성과 원본 결과를 보존해 개선을 모델·검색·프롬프트·도구 변화에 연결합니다. 카나리는 오염된 비교를 복구하지 못합니다. 롤백 권한은 평가 대상이 아니라 출시 시스템에 둡니다.
기업과 제품의 위치
선별한 산업 지도이며 시장 점유율 순위나 인증이 아닙니다. 범위는 제품 버전, 통합, 배포 및 집행 모드에 따라 달라집니다.
Arena / LMArena
- 소유 / 조직
- Arena Intelligence Inc.
- 제어 지점
- 사람 선호·외부 평가
Arena는 사람 피드백 기반 외부 평가를 판매하며 고객 내부 회귀 세트 운영과 다릅니다. 2025년 9월 16일 시작한 상용 서비스는 분석, 대표 표본, 납기를 제공합니다. 커뮤니티 참여가 차별화된 입력이고 판매와 신뢰할 중립성을 함께 유지하는 것이 과제입니다. 선호에서 이겨도 도구 권한이나 업무 정확성을 인증하지 않습니다. 현재 Agent Mode는 기존 쌍대 선호 순위와 다르므로 제품과 방법을 명확히 해야 합니다.
Arena commercial evaluation launch ↗https://arena.ai/blog/ai-evaluations
Artificial Analysis
- 소유 / 조직
- 독립 벤치마크 기업
- 제어 지점
- 모델·인프라 선정
Artificial Analysis는 모델, 에이전트, 추론 공급자, 하드웨어를 측정하고 공개 정보와 무료·상용 API를 제공합니다. 구매자는 출시 게이트보다 선정·조달 정보를 원할 가능성이 큽니다. 품질·성능·경제성 비교가 가치입니다. 종합 지수는 구성된 지표이므로 버전, 설정, 날짜를 남겨야 합니다. 현재 Intelligence Index는 주로 영어 텍스트이며 다른 언어·모달리티는 별도 증거가 필요합니다. 후보 선정 후 실제 업무로 시험해야 합니다.
Scale AI / SEAL / Scale Labs
- 소유 / 조직
- 독립 기업; Meta 의결권 없는 소수 투자
- 제어 지점
- 전문가 데이터·연구 평가 서비스
Scale은 전문가 데이터 운영, 평가 서비스, SEAL·Scale Labs 연구를 결합합니다. 자동 라벨이 부족한 어려운 작업과 전문가 판정을 대규모로 조직하는 것이 잠재적 강점입니다. Meta 소수 투자는 사실이지만 모든 평가의 편향이나 전체 인수의 증거는 아닙니다. 데이터 접근, 계약상 분리, 공개 절차를 실사해야 합니다. 전문가 지식은 타당성을 높일 수 있지만 서비스 비용과 판정 일관성은 상업적 제약입니다.
Galileo / Cisco
- 소유 / 조직
- 2026년 5월 22일 인수 완료
- 제어 지점
- 평가에서 런타임 가드레일로
Galileo는 소형 Luna 평가기를 포함해 평가, 관측, 런타임 가드레일을 연결합니다. Cisco는 2026년 5월 22일 인수를 완료했으므로 독립 기업으로 설명하면 틀립니다. 오프라인 품질 기준을 저렴한 운영 통제와 기업 유통으로 전환하는 기회입니다. 비용·지연 주장은 같은 조건에서 검증하지 않으면 업체 주장입니다. 빠른 평가기도 보정과 불확실성 대응이 필요하며 자원 권한, 정확한 승인, 이미 확정된 부작용 복구를 대체하지 않습니다.
Databricks / MLflow
- 소유 / 조직
- Databricks 플랫폼·오픈소스 MLflow
- 제어 지점
- 데이터 거버넌스 기반 평가
MLflow는 추적, 평가, 심사기, 프롬프트, 사람 피드백을 모델 개발과 연결합니다. Databricks는 관리형 운영과 기업 데이터 거버넌스의 근접성을 더합니다. 데이터 집적과 기존 구매가 강점이고 오픈소스 API가 이동성을 지원한다는 해석입니다. 과거 모든 버전이 같다고 가정하지 말고 버전과 에디션을 확인해야 합니다. 관리된 데이터와 모델 레지스트리는 유용하지만 특정 요청의 외부 결제·삭제 권한을 결정하지는 않습니다.
Microsoft Foundry
- 소유 / 조직
- Microsoft 클라우드 플랫폼
- 제어 지점
- 트레이스 데이터화·출시 통합
Microsoft Foundry는 추적, 평가, 감시, CI/CD 품질 게이트를 기업 개발 환경에 결합합니다. 트레이스를 정제·버전 관리된 평가 데이터로 만드는 문서가 있습니다. Microsoft 중심 조직의 통합·구매 마찰을 줄이는 것이 강점입니다. 전문 기업은 추가 플랫폼 부담보다 큰 의사결정 개선을 입증해야 합니다. 데이터 생성에는 대표성 있는 선정과 민감 데이터 통제가 필요하며 자동 수집이 라벨 정확도나 최소 권한을 증명하지는 않습니다.
AWS Bedrock / Google Cloud
- 소유 / 조직
- 서로 다른 클라우드 제품
- 제어 지점
- 관리형 모델·앱 평가
AWS Bedrock은 모델·RAG 평가, 사람 평가, 모델 심사 작업을 제공합니다. Google Cloud 문서는 외부 모델을 포함한 모델·앱 평가를 다룹니다. 서로 다른 제품이며 클라우드 유통의 경쟁 압력 때문에 묶었습니다. 기존 인프라와 계약은 도입을 쉽게 하지만 중립성과 이동성은 실사해야 합니다. 관리형 실행이 올바른 작업 모집단을 선택하거나 심사기를 자동 검증하지는 않습니다. 결과에는 구성, 예산, 지표 버전을 남겨야 합니다.
Amazon Bedrock Evaluations ↗https://aws.amazon.com/bedrock/evaluations/ · Google Cloud Gen AI evaluation overview ↗https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview
라벨과 재현 가능한 환경의 경제성
평가 비용은 심사 모델 토큰 요금만이 아닙니다. 작업 설계, 전문가 판정, 환경 초기화, 추론, 반복, 제품 변경에 따른 유지가 포함됩니다. 점수를 늘리는 대신 설명 가능한 결정의 비용을 줄여야 가치가 있습니다. 재사용 가능한 전문가 절차, 얻기 어려운 대표 작업, 오염 통제, 재현 가능한 환경은 일반 평가기보다 강한 우위일 수 있습니다. 하지만 고객 라벨을 다른 테넌트에 재사용하는 것이 합법적이거나 적절하지 않을 수 있습니다. 데이터 네트워크 효과라면 동의, 격리, 이전 가능성, 기밀 작업을 노출하지 않는 일반화를 설명해야 합니다.
기술 핵심: 품질을 판단하는 세 가지 계약
벤치마크는 명시한 과제 해결을, 선호 아레나는 제시 및 표본 조건에서의 사용자 선호를, 배포 평가는 품질·안전·지연·비용 제약 안에서 구성된 업무 완료를 측정합니다. 선호하는 답이 근거를 잃거나 정답이 늦게 도착할 수 있으며 뛰어난 모델도 검색이나 도구 통합 때문에 실패할 수 있습니다. 구매 판단에 합치기 전에 각 근거를 구분해야 합니다.
공정한 비교는 같은 사례에 두 후보를 실행하고 짝지은 차이를 살펴봅니다. 선정 전에 데이터, 채점 기준, 평가기 버전과 도구 권한을 고정하고 최종 홀드아웃은 최적화에서 제외합니다. LLM 심판은 판정된 예제로 보정하며 위치·문체 편향과 후보와의 공통 오류를 살펴야 합니다. 불확실성과 과제군별 결과를 보고합니다. 비용이 큰 실패가 악화됐다면 평균 점수 상승만으로 개선을 입증할 수 없습니다. 방어 가능한 제품은 반복 가능한 의사결정 과정입니다.
점수에서 안전한 결정까지 남은 간극
심사기는 일관되면서도 틀릴 수 있고 여러 심사기도 같은 사각지대를 가질 수 있습니다. 공개 순위의 언어·작업·표현 선호는 구매자의 업무와 다를 수 있습니다. 에이전트는 하네스, 도구, 외부 상태, 예산, 초기화에도 의존하므로 모델명만으로 비교할 수 없습니다. 운영 승인은 업무 결과와 테넌트 간 무단 접근 금지, 미승인 부작용 금지, 숨은 권한 확대 금지를 결합해야 합니다. 다음 가치는 불확실성과 책임이 분명한 증거 통합이라는 분석입니다. 전문가 판단, 재현 실행, 업무 결과를 연결하되 자동 대리지표를 정답으로 포장해서는 안 됩니다.
인수, 자본, 전략적 영향
자본과 소유: 독립성을 정확히 설명하기
Arena는 2026년 1월 6일 1억5천만 달러 시리즈 A를 발표했습니다. Scale은 2025년 6월 12일 Meta 투자를 발표했고 Meta는 이후 의결권 없는 소수 지분과 비시장성 지분 투자에 포함된 137억9천만 달러를 공시했습니다. Scale 전체 인수나 평가 사업만의 가격이 아닙니다. Cisco는 4월 9일 의향 발표 후 2026년 5월 22일 Galileo 인수를 완료했습니다. 자본 관심이지 평가 품질의 증거가 아닙니다. 소유·상업 관계는 고객 데이터 접근, 공개 규칙, 모델 선정, 이해충돌 관리를 검토할 이유지만 관계만으로 편향을 단정할 수 없습니다.
제품 및 거래의 1차 출처
- Arena Series A ↗https://arena.ai/blog/series-a ·
- Arena Agent Mode methodology ↗https://help.arena.ai/articles/5432423882-how-to-use-agent-mode ·
- Meta quarterly filing: Scale non-voting minority investment ↗https://www.sec.gov/Archives/edgar/data/1326801/000162828025047240/meta-20250930.htm ·
- Artificial Analysis free and commercial APIs ↗https://artificialanalysis.ai/api-reference ·
- Galileo evaluation and guardrail product ↗https://galileo.ai/ ·
- Arena commercial evaluation launch ↗https://arena.ai/blog/ai-evaluations ·
- Artificial Analysis index methodology ↗https://artificialanalysis.ai/methodology/intelligence-benchmarking ·
- Scale investment and independence statement ↗https://scale.com/blog/scale-ai-announces-next-phase-of-company-evolution ·
- Cisco Galileo acquisition completion ↗https://www.cisco.com/site/us/en/about/corporate-development/acquisitions/galileo/index.html ·
- Databricks current MLflow observability and portability FAQ ↗https://docs.databricks.com/aws/en/mlflow3/genai/tracing/faq ·
- Microsoft Foundry traces to evaluation datasets ↗https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/traces-to-dataset ·
- Amazon Bedrock Evaluations ↗https://aws.amazon.com/bedrock/evaluations/ ·
- Google Cloud Gen AI evaluation overview ↗https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview ·
14 · How AI giants compete through evaluation
Frontier-model competition is partly a competition over what counts as progress. A provider can emphasize mathematical reasoning, coding, agentic task completion, multimodal understanding, latency, price or safety. Those dimensions matter, but selecting the favorable subset can create a misleading overall impression. Read a model launch as a collection of scoped claims rather than a universal ranking.
For every headline comparison, ask whether model versions, prompts, tools, inference budgets and grader settings match. A result with extensive search, retries and a large compute budget does not measure the same product experience as a single low-latency response. If proprietary scaffolding contributes to the score, describe the result as a system result unless the contribution has been isolated.
Providers have real advantages in evaluating their own systems: detailed access to versions, logs and failure modes. They also have incentives to present progress favorably. Independent evaluators can diversify the evidence, but independence is not guaranteed merely by being a separate company. Examine funding, commercial relationships, publication policies, access privileges and the ability to report negative findings.
Enterprise buyers should build a small portfolio of evidence: public capability measurements for breadth, domain-specific held-out tasks for fit, adversarial tests for boundaries, and randomized online experiments for user impact. Require a candidate to improve the dimensions that matter to the business rather than chasing every public leaderboard movement.
The durable competitive advantage may lie in a high-quality feedback loop: representative failures become reviewed tasks, interventions are tested fairly, successful changes are deployed cautiously, and production outcomes verify the hypothesis. This is an engineering interpretation of the market, not a claim that any named company has already solved that loop.
Cloud platforms compete for the evaluation workflow
Microsoft Foundryhttps://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-overview documents evaluation of existing datasets, generated model or agent outputs, and production traces. This is a workflow integration proposition: keeping execution, permissions and evidence near the deployment environment. Buyers should still export a versioned task set and verify that scores remain reproducible outside the portal.
Google Cloud's Gen AI evaluation servicehttps://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview documents evaluating third-party models as well as models in its own ecosystem. This highlights an important separation: the generator being tested need not be the judge. A cross-provider evaluation is not automatically independent, however; check the judge's calibration and version on the target domain.
Amazon Bedrockhttps://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html documents model-evaluation jobs that use a separate LLM to score responses and explain the score. A generated explanation is useful evidence for review, not proof that the score is correct. Keep an adjudicated calibration set, record evaluator configuration and include judge cost in the experiment ledger.
OpenAI and Anthropic also contribute provider-authored audits and engineering guidance, discussed earlier. Across these companies, the competitive axes are model quality, integration, operating cost, access to feedback and trust in measurement. The practical strategy is to keep your evaluation contract portable while using integrations where they reduce work. Do not give the vendor's default rubric authority to redefine your business outcome.
Continue the learning path
Read the companion handbook: evaluation and observability · AI security · AI infrastructure
Learning library · Interview and projects · Editorial standards and corrections
Educational extensions: OpenAI Academyhttps://academy.openai.com/ · Anthropic learning resourceshttps://academy.claude.com/ · Microsoft AI traininghttps://learn.microsoft.com/en-us/ai/.
Primary references are attached to the relevant claims. Conceptual diagrams, synthetic arithmetic and research proposals are original teaching material. Vendor descriptions establish product positioning, not independent proof of performance. Corrections should identify the claim, its source and the proposed replacement.
실습과 검증
5단계: 피드백 순환과 마지막 실습 검증
운영 피드백은 늦고 선택 편향이 있습니다. 침묵은 성공이 아닙니다. 수정에 사용한 장애는 더 이상 독립 시험이 아닙니다. 사건→사례→평가 규칙→실행→배포 결정의 계보를 남기며 채점기 변경을 능력 향상과 혼동하지 않습니다. 최종 증거에는 계약, 쌍별 결과, 중요 구간, 채점 보정, 비용과 롤백 조건을 포함합니다.
마지막 실습에서 ACL만 바꿔 무단 검색 차단을 확인하고 정책을 없애 답변 보류를 요구합니다. 30일을 300일로 바꾸면 구조 검사는 통과하되 근거 검사는 실패해야 합니다. 같은 사례를 열 번 복제해도 독립 표본 수를 부풀리지 않습니다. 무단 작업 한 건은 배포를 막아야 하며 채점기 장애는 미상으로 표시합니다. 기대·실측·설명을 보존하고 수정에 쓰지 않은 새 사례로 전이를 검증합니다.
15 · Open research problems worth a serious experiment
External validity. Can an offline score predict a deployment outcome across organizations? Construct several domain-specific task sets, preregister the predicted ordering, and compare it with a later controlled rollout. Report where the ordering reverses. A single positive correlation on one product is not a universal transfer law.
Adaptive contamination. How can a benchmark remain informative when developers repeatedly optimize against it? Compare fixed public tasks, private held-outs and periodically refreshed tasks under a controlled optimization process. Measure performance on a final unseen distribution. The aim is to separate genuine generalization from learning the measurement interface.
Judge robustness. Can a grader resist style changes and adversarial instructions without losing sensitivity to real improvements? Create matched output pairs that preserve correctness but change verbosity, formatting or persuasion. Then create pairs with identical style but different factual quality. The grader should be invariant to the former when irrelevant and responsive to the latter.
Long-horizon reliability. How do errors accumulate across tool interactions? Vary task length while controlling task type, and distinguish exposure from intrinsic deterioration. A longer task creates more opportunities to fail; a higher session-level failure rate does not alone show that per-action reliability worsened.
Evaluation under limited budgets. Which tasks should be labeled or rerun next? Compare uniform sampling with an adaptive policy, but preserve a probability sample for unbiased reporting. An adaptive strategy can find more failures while producing a distorted raw failure rate if selection probabilities are ignored.
Reward hacking in self-improving systems. Does an optimization loop improve the target capability or exploit its grader? Hold a second evaluator and task set outside the loop, impose equal compute budgets, and compare against a non-adaptive baseline. Publish negative results and evidence of evaluator exploitation. A rising training reward is a hypothesis about improvement, not its verification.
16 · A four-week capstone with a defensible release decision
Build a small document-support agent that answers policy questions and can propose—but not execute—an account change. The baseline uses one retrieval method and a fixed prompt. The candidate changes exactly one component. Use synthetic documents and mock tools so no personal data or real transactions are involved.
In week one, define 60–100 tasks across straightforward retrieval, conflicting evidence, missing evidence and unauthorized requests. Write a rubric and deterministic postcondition checks. Reserve an untouched test partition. Annotate a small calibration set with two reviewers and resolve disagreements by revising the rubric rather than averaging incompatible interpretations.
In week two, implement the harness with task IDs, seeds where applicable, budget limits, artifacts and explicit infrastructure-error states. Run baseline and candidate on the same tasks. Repeat a subset to estimate execution variance. Add a local report showing paired outcomes, critical slices, latency and cost per accepted result.
In week three, audit failures. Test whether the judge can be manipulated by candidate text. Perform an ablation that restores the baseline component, and a budget-matched comparison that prevents extra compute from masquerading as algorithmic improvement. Add discovered failures to the development set while keeping the final test untouched.
In week four, make the release decision under a preregistered rule. A strong report includes the task distribution, exact versions, uncertainty, failed examples, resource costs, safety constraints and reasons to defer deployment. The reference success criterion is not “candidate wins.” It is “another engineer can reproduce and critically evaluate the conclusion.”
For a research extension, measure whether the offline decision predicts a simulated deployment distribution with different proportions of difficult tasks. Explain the discrepancy when distribution shift changes the conclusion.
17 · Original interview practice and scoring
These are original practice questions, not reported questions from any company. Use the same 1–5 progression for each: 1 identifies the relevant concept; 2 gives a correct mechanism; 3 applies it to the concrete example; 4 proposes a sound experiment with controls; 5 explains assumptions, failure cases and the limits of the conclusion.
1. Candidate B wins 55% of public preference comparisons. Should you replace the support model?
Hint 1: Identify the population and outcome being measured.
Hint 2: Distinguish preference from policy correctness and authorization.
Reveal answer and analysis
Answer: Not on that evidence alone. Run a paired comparison on representative support tasks with correctness, policy and safety graders, then examine uncertainty and cost. A strong answer also checks the public comparison's confidence interval, category and versions, and proposes a guarded online experiment after offline gates pass.
2. Why can 90% judge agreement still be unacceptable?
Hint 1: Consider an imbalanced dataset.
Hint 2: Ask which disagreement type is costly.
Reveal answer and analysis
Answer: If almost all examples are acceptable, a judge that passes everything can achieve high agreement while missing every dangerous failure. Inspect class-conditional errors, calibrate on consequential cases and audit false passes. A top answer explains why agreement with one imperfect annotator is not ground truth and defines an adjudication process.
3. An agent succeeds at least once in five runs but often fails the first run. Which metric matters?
Hint 1: What attempts can the product afford?
Hint 2: Can it identify the successful attempt reliably?
Reveal answer and analysis
Answer: Pass@5 is relevant only if five attempts and reliable selection are available. For a single-shot workflow, first-attempt success and failure severity matter. For repeated dependable execution, measure all-attempt reliability and task-level variance. Include latency, cost, correlated failures and side effects from unsuccessful attempts.
4. A retrieval change raises answer quality. How do you establish why?
Hint 1: Separate evidence availability from generation.
Hint 2: Match the context and compute budgets.
Reveal answer and analysis
Answer: Measure evidence recall, answer support and correctness on the same tasks, keeping the generator fixed. Compare retrieval-only ablations under equal context budgets. Inspect whether gains concentrate on previously missing evidence and test distraction and stale-document cases. A high-scoring answer avoids claiming causality from aggregate correlation alone.
5. A self-improving agent raises its own benchmark score every day. What would convince you it improved?
Hint 1: Identify what the agent can modify or observe.
Hint 2: Keep independent evidence outside the loop.
Reveal answer and analysis
Answer: Freeze an independent evaluator and held-out tasks, compare under equal resource budgets, and check whether gains transfer to new tasks and robust behavior. Audit grader exploitation and contamination. The strongest answer defines a preregistered stopping rule, preserves failed runs and distinguishes increased capability from increased search effort.
Reproduce the arithmetic: a complete local reference
Imagine evaluating a replacement support agent on twenty matched tasks. Twelve tasks pass under both versions, four pass only under B, two pass only under A, and two fail under both. Model A scores 14/20 and B scores 16/20. We resample task pairs, keeping the relationship between outcomes intact. The resulting interval is wide: this demonstration does not establish that B is better in the population. The individual Wilson interval is a separate description of B's success rate; overlap between two marginal intervals is not the paired comparison test.
Here n is the number of independent tasks and outcomes are binary. The second expression is the exact one-sided upper confidence bound on failure probability after zero observed failures, assuming independent identically distributed Bernoulli trials; alpha = 0.05 gives 95% confidence. Correlated tasks or an unrepresentative sample invalidate that interpretation.
Revisit the opening score-and-uncertainty diagram. Its error bars are marginal Wilson intervals; the paired interval below addresses the difference directly.
The bootstrap is a percentile approximation, not a guarantee of calibrated coverage with just twenty examples. If multiple prompts belong to one customer or document, resample customers or documents. More repeated generations per item measure stochastic variability; they do not manufacture more independent customers.
Try the denominator locally
No input leaves your browser. This is an arithmetic illustration, not a production risk estimate.
Runtime: Python 3.10 or newer; standard library only. The evaluation bootstrap uses seed 17 and 10,000 resamples. Observability arithmetic is deterministic and needs no seed. These programs make no network or model calls.
Download the complete Python reference
python3 evaluation-audit.py"""Synthetic paired evaluation; Python 3.10+, standard library, seed 17.
No API calls. Items, rather than individual judgments, are bootstrap units.
"""
import math
import random
import json
from statistics import mean
def wilson(successes, n, z=1.959963984540054):
if not 0 <= successes <= n or n <= 0:
raise ValueError('Require 0 <= successes <= n and n > 0')
p = successes / n
den = 1 + z*z/n
center = (p + z*z/(2*n)) / den
radius = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / den
return center-radius, center+radius
def paired_bootstrap(a, b, repetitions=10000, seed=17):
if len(a) != len(b) or not a:
raise ValueError('Require nonempty matched items')
if not isinstance(repetitions, int) or repetitions < 100:
raise ValueError('Use at least 100 bootstrap repetitions')
if any(v not in (0, 1) for v in a + b):
raise ValueError('This example requires binary outcomes')
rng = random.Random(seed)
differences = [y-x for x, y in zip(a, b)]
means = sorted(mean(rng.choices(differences, k=len(a)))
for _ in range(repetitions))
return mean(differences), means[int(.025*repetitions)], means[int(.975*repetitions)]
def grade_response(raw, allowed_sources, expected_status):
"""Original deterministic contract grader, NOT a semantic truth judge.
Candidate text is parsed as data; never eval(), execute(), or prompt a judge.
"""
try:
response = json.loads(raw)
except (json.JSONDecodeError, TypeError):
return {"schema": False, "evidence_ids": False, "status": False}
schema = (isinstance(response, dict)
and set(response) == {"answer", "sources", "status"}
and isinstance(response["answer"], str)
and isinstance(response["sources"], list)
and all(isinstance(x, str) for x in response["sources"])
and response["status"] in ("answer", "abstain"))
if not schema:
return {"schema": False, "evidence_ids": False, "status": False}
return {"schema": True,
"evidence_ids": (all(x in allowed_sources for x in response["sources"])
and (response["status"] != "answer" or bool(response["sources"]))),
"status": response["status"] == expected_status}
valid = '{"answer":"The policy says 30 days.","sources":["policy-v3"],"status":"answer"}'
assert all(grade_response(valid, {"policy-v3"}, "answer").values())
assert not grade_response('not JSON', set(), 'abstain')['schema']
assert not grade_response(valid, {"policy-v4"}, "answer")['evidence_ids']
assert not grade_response(valid, {"policy-v3"}, "abstain")['status']
# A plausible-looking lie can pass these structural checks. Add human/semantic
# review against the actual source; do not call this contract pass "correctness".
lie = valid.replace('30 days', '300 days')
assert all(grade_response(lie, {"policy-v3"}, "answer").values())
# Same 20 synthetic tasks: 12 both pass, 4 only B, 2 only A, 2 neither.
a = [1]*12 + [0]*4 + [1]*2 + [0]*2
b = [1]*12 + [1]*4 + [0]*2 + [0]*2
assert sum(a) == 14 and sum(b) == 16
low, high = wilson(sum(b), len(b))
delta, lower, upper = paired_bootstrap(a, b)
assert lower <= 0 <= upper # small sample does not establish superiority
assert paired_bootstrap(a, a) == (0, 0, 0)
assert wilson(0, 20)[0] >= -1e-12
assert math.isclose(1 - .05**(1/100), .029513049607039932)
# Trial independence is an assumption, not a property guaranteed by a model.
assert math.isclose(1-(1-.8)**3, .992)
assert math.isclose(.8**3, .512)
print(f'A: {mean(a):.1%}; B: {mean(b):.1%}')
print(f'B Wilson 95% interval: [{low:.3f}, {high:.3f}]')
print(f'Paired delta: {delta:.1%}; bootstrap 95% interval: [{lower:.3f}, {upper:.3f}]')
print('Zero failures in 100: one-sided 95% upper bound 2.95%')
print('All evaluation checks passed; synthetic data, not model performance.')
Expected output:
A: 70.0%; B: 80.0%
B Wilson 95% interval: [0.584, 0.919]
Paired delta: 10.0%; bootstrap 95% interval: [-0.150, 0.350]
Zero failures in 100: one-sided 95% upper bound 2.95%
All evaluation checks passed; synthetic data, not model performance.Test the failure cases: empty samples are rejected; the evaluation comparison of a model with itself has zero difference; the trace algorithm rejects cycles. Next add clustered tasks or missing spans, and document exactly which assumptions stop holding. Unit checks validate this implementation, not a deployed model.