Token LabSearch

English article. Synthetic examples and vendor claims are identified separately.

AI EVALUATION / RESEARCH & ENGINEERING HANDBOOK

AI evaluation: from measurements to decisions, research and market power

A field guide to traditional ML, LLM and agent evaluation: statistics, Arena, engineering, research frontiers, companies and acquisition evidence.

Jump to the runnable reference and tests

Sources & version details
Contents · chapters, research and reference code
In this article
PART I / ONE SYSTEM, END TO END

Evaluation: decide whether the same system is ready to ship

This original case study follows an invoice assistant. It reads tenant-scoped documents and proposes a payment; executing a payment requires separate, exact authorization. The scenario and experiments are synthetic teaching material, not a reported deployment.

Evaluation: decide whether the same system is ready to ship: conceptual lifecycle diagram
Original architecture illustration. Lines represent dependencies or evidence, not measured performance.

Start with the decision: may this invoice agent be released?

Our synthetic assistant answers invoice questions, reads tenant documents and proposes payments. It may execute only the exact approved action. A new model writes better explanations, while a new retrieval configuration supplies more context. The team wants to ship both. A public leaderboard cannot answer whether this combined system will resolve the company’s cases correctly without crossing a permission boundary. The evaluation must begin with that deployment decision.

Define an accepted task as a supported answer or correct abstention, completed within the budget, with no unauthorized action. Keep these dimensions separately visible: a composite pass rate alone can hide an unacceptable security regression. Identify the unit of analysis as a customer case, not a model call. A case with six retries is still one case, though every retry contributes cost and risk.

The evaluation contract names the intended task population, model and prompt versions, document snapshot, tool schemas, action budget, timeout, grader versions and exclusion rules. It also names what happens when evidence is inconclusive. “Do not release yet” is a valid output. Without this contract, engineers can unknowingly change the question until the result looks favorable.

Stage 1: build tasks that expose the whole workflow

Construct cases across normal invoices, conflicting versions, missing evidence, stale balances, foreign-tenant documents, changed payees, revoked approvals and duplicate submissions. Use synthetic documents and mock payment tools in the local baseline. A task should specify the initial environment state and the allowed final state, not only an expected sentence. “Prepare a payment” and “execute a payment” are intentionally different tasks.

Partition by meaningful sources of dependence. Ten paraphrases of one invoice are not ten independent business cases. Split by customer, document family or time when those factors create leakage. Keep development tasks for debugging, validation tasks for selection and a hidden final test for a bounded release decision. Restrict access to test answers and record when the team has inspected them.

Retrieval evaluation asks whether the necessary authorized evidence was available. Generation evaluation asks whether the answer was supported by that evidence. An end-to-end failure can originate in either layer. Include an oracle-context condition that supplies the correct evidence and a deliberately missing-evidence condition requiring abstention. These controls reveal whether better retrieval or better use of context is needed.

As a synthetic example, suppose production contains 90% routine tasks and 10% difficult exceptions, while the diagnostic suite deliberately contains equal numbers. Its unweighted average does not estimate the production average. Report results for each stratum and publish the weighting rule; do not conceal a weak critical slice inside a favorable aggregate.

Stage 2: turn observations into defensible judgments

Use deterministic graders for properties that can be checked exactly: JSON schema, allowed document IDs, exact payment arguments, ledger state, duplicate effects and whether an approval was consumed only once. Use human or calibrated model-assisted review for semantic properties such as whether the cited policy supports the explanation. Do not ask a language judge to replace a database postcondition that can be inspected directly.

Anthropic’s January 9, 2026 evaluation guidehttps://www.anthropic.com/research/demystifying-evals-for-ai-agents distinguishes tasks, trials, graders and outcomes and discusses combining grader types. That practical separation is useful here: a tool-call transcript is evidence about the attempt, while the business environment determines whether the intended outcome occurred. Provider guidance is not a universal experimental guarantee.

Calibrate an automated judge on a reviewed sample containing both ordinary and consequential failures. Measure false passes and false failures by category. If 95% of examples are acceptable, an always-pass judge achieves 95% accuracy while missing every bad case. Evaluate sensitivity to response order, verbosity, persuasive instructions and irrelevant formatting. Keep candidate text as untrusted data within the grading process.

Stage 3: run controlled comparisons and account for uncertainty

Run baseline and candidate on the same cases with the same action and resource budgets. Pin model versions where possible and record all unavoidable differences. Repeat selected cases to estimate stochastic variability, while keeping task and trial identifiers separate. Seeds can help reproduce local components, but an external API seed does not necessarily guarantee deterministic execution across infrastructure or model changes.

A fair ablation changes one component at a time. Compare old retrieval with old model, new retrieval with old model, old retrieval with new model and the combined candidate. This reveals interactions that a single before-and-after score hides. If the candidate receives twice the context or retries, report that resource increase explicitly and add a budget-matched condition.

Consider a synthetic paired test of 200 independent cases. Both systems pass 150; only the candidate passes 20; only the baseline passes 10; both fail 20. The baseline scores 80%, the candidate 85%, and the paired improvement is five percentage points. The useful information includes the 30 discordant cases, not just two marginal percentages. A normal teaching approximation gives standard error sqrt((0.15−0.05²)/200), about 2.72 percentage points, so a rough 95% interval extends slightly below zero. Use an appropriate paired interval or test for the actual analysis and inspect clustering.

Stage 4: make a release decision that can reject a higher score

Before seeing results, define the minimum useful improvement, acceptable uncertainty, cost and latency budgets, and hard safety constraints. A candidate that improves average answer quality but introduces unauthorized payments should fail the gate. A candidate that is safe but statistically inconclusive may need more testing or a restricted rollout. A single weighted score should not permit a gain in fluency to compensate for an unacceptable action.

In our synthetic comparison, the five-point improvement is promising but uncertain. Suppose the candidate also passes every permission test in 100 adversarial cases. That is useful evidence, yet zero observed failures does not establish zero risk. Under independent Bernoulli assumptions, the one-sided 95% upper bound after zero failures in 100 trials is about 2.95%. An adversarial test set is usually not a random sample of production, so interpret it as evidence about the tested attack classes rather than a population incidence estimate.

Start with shadow execution where proposed actions cannot commit. A later canary uses a small approved population with the same external permission controls as the baseline. Compare contemporaneous groups and define a stopping rule. Do not repeatedly inspect ordinary fixed-horizon confidence intervals and stop only when a favorable result appears; use a planned horizon or a valid sequential design.

How the four disciplines fit together

Security determines which actions are permitted. Infrastructure executes work within capacity and failure constraints. Observability preserves evidence of what happened. Evaluation tests whether a change improves the outcome under a declared contract. None substitutes for the others.

PART II / REFERENCE CHAPTERS

Mechanisms, research and market context — English reference material

The following chapters provide the mathematical examples, experiments, research papers and dated company evidence behind the operational story. Return to the lifecycle when deciding where each mechanism belongs.

Synthetic paired evaluation scores with confidence intervals.
Original illustration: uncertain estimates require paired analysis, not a leaderboard-style verdict.
Capability, preference and deployment evaluation answer different questions.
Conceptual taxonomy, not experimental evidence.

01 · Evaluation begins with a decision, not a leaderboard

Imagine that an engineering team is replacing the model behind a customer-support assistant. The new model wins more public head-to-head comparisons and writes more polished answers. Yet its first production week produces more incorrect refunds. Both observations can be true. A public preference ranking measures how a particular population judges particular responses; a refund workflow requires correct policy interpretation, authorization and state transitions. The mistake is not necessarily in either measurement. It is in treating one measurement as a substitute for another.

Definition. An AI evaluation is a specified procedure for gathering evidence about a system's behavior under stated conditions and using that evidence to support a decision. A benchmark is a reusable collection of tasks and measurement rules. A metric summarizes observations. A grader turns an output, action or environment state into a judgment. A harness controls execution and records the conditions. These objects are related, but none is interchangeable with the others.

The decision should be written before collecting scores: “Release candidate B only if it improves successful resolution on our target tasks without increasing unauthorized actions, while meeting our latency and cost budgets.” This statement identifies the target population, the benefit, the constraints and the deployment boundary. It also exposes missing knowledge: what counts as successful resolution, how frequently dangerous cases occur, and how confidently we can measure the difference.

Before this lesson, you should understand a sample mean as an estimate of a population property, a conditional probability as a statement restricted to specified conditions, and a train/test split as a barrier against adapting to the answers. You do not need to have memorized advanced statistics. You do need to recognize that running the same task repeatedly measures execution variability, while collecting new tasks measures a different source of uncertainty.

By the end, you should be able to design a release gate, interpret an Arena-style comparison, calibrate an automated judge, build an auditable evaluation pipeline, and explain why a high score can coexist with an unsuitable product. The examples in this chapter are original teaching examples unless explicitly attributed; they are not measurements of commercial models.

Choose a reading route

First encounter: read the definitions and worked examples, then predict the local experiment result before changing its input. Engineering implementation: focus on the data contract, architecture and Python reference, then write one failure test of your own. Research reading: inspect the cited paper versions, challenge their evaluation assumptions, and choose one open problem with a falsifiable baseline.

02 · Traditional ML, LLMs and agents: what changes?

Traditional supervised evaluation often starts with a fixed input, a reference label and a loss function. A classifier receives an image and predicts a class; a forecasting model predicts a number. Accuracy, precision, recall, calibration and error by subgroup can be useful because the target is comparatively explicit. Even here, labels can be wrong, class distributions can shift, and the cost of a false negative can differ from that of a false positive. Traditional evaluation was never automatically simple or unbiased.

For a language model, several outputs may be acceptable. A concise explanation and a longer explanation can both be correct; exact string matching can reject one for irrelevant reasons. Meanwhile, an apparently correct conclusion may contain fabricated evidence. Evaluation therefore separates dimensions such as factual correctness, instruction following, evidence support, completeness and usefulness. Combining them into one number is a policy choice that requires explicit weights.

An agent adds actions and environmental state. It can finish a task through different trajectories, use extra tools, encounter failures, recover, or damage something unrelated to the task. A final paragraph saying “Done” is not evidence that the task was completed. A database query, repository diff or sandbox state may be the appropriate verifier. The evaluated object is now the model plus its prompt, tool definitions, memory, permissions, retry logic and surrounding application.

Three distinctions prevent many misleading comparisons. First, model evaluation isolates a model as far as possible; system evaluation tests the assembled product. Second, capability evaluation asks what can be accomplished under a budget; reliability evaluation asks how consistently the behavior occurs. Third, offline evaluation estimates performance on controlled tasks, whereas online experimentation estimates effects on users under an actual deployment.

None replaces the others. A model can be capable but unreliable, a reliable product can depend on expensive scaffolding, and a statistically positive online result can conceal a severe safety regression. Keep the axes visible until the decision requires an explicit tradeoff.

03 · Write an evaluation contract

A useful evaluation contract names the task distribution, unit of analysis, execution budget, grading rubric, exclusion policy and release rule. For a support agent, a unit might be one customer case, not one model call. For a coding assistant, it might be one repository issue, not every generated patch. If a case produces ten retries, treating those retries as ten independent customers exaggerates the amount of evidence.

Make the task set versioned and immutable during a comparison. Record the repository snapshot or document corpus, model identifier, prompt hash, tool schema, inference settings, maximum actions, wall-clock timeout and grader version. Record unavailable dependencies and API errors as explicit outcomes. Quietly deleting inconvenient failures changes the question being answered.

Use three task collections with distinct roles. A development collection helps engineers investigate errors. A validation collection supports regular tuning and model selection. A held-out test collection supports a final estimate and should not become a daily feedback channel. When the team has repeatedly inspected the held-out answers, it has effectively converted that set into development data. Refreshing a test set is not a failure; pretending it remains untouched is.

A contract also specifies the aggregation rule. Suppose the task mix is 80% simple retrieval and 20% difficult account changes. Averaging those tasks equally may answer a different question from weighting them by production traffic. Conversely, production-weighting alone may hide a rare but unacceptable unauthorized transfer. Report an overall estimate and decision-critical slices, then apply hard constraints separately where appropriate.

Finally, define inconclusive outcomes. “Insufficient evidence to release” is a legitimate result. A good process does not force every experiment into a winner and a loser. It preserves uncertainty until the expected value of further testing is lower than the cost of waiting.

04 · Uncertainty, paired comparisons and repeated trials

A score is an estimate, not a physical constant. If candidate A succeeds on 80 of 100 tasks, the observed success rate is 80%, but a different sample could produce another value. A confidence interval communicates sampling uncertainty under a specified model. It does not account automatically for mislabeled tasks, a biased sample, contaminated training data or a broken grader.

Compare two systems on the same tasks whenever possible. For each task, calculate the difference between candidate and baseline. Paired comparisons remove some variation due to task difficulty: a hard task is hard for both systems. Bootstrap whole tasks or higher-level clusters such as users or repositories when those are the independent units. Do not independently resample individual tool calls from the same trajectory.

Repeated trials answer a second question: how much does behavior vary when the task is held fixed? Keep the task identifier and trial identifier separate. A system that solves each of ten tasks in one lucky run out of ten has different production behavior from one that reliably solves nine tasks and always fails the tenth. Their pooled success rates can conceal these patterns.

Pass@k describes the chance that at least one of k attempts succeeds under the evaluation's sampling and selection assumptions. It is relevant when a user can actually generate and identify a good candidate. It does not mean the first answer is reliable. A requirement that all k repeated attempts succeed is a different reliability criterion, sometimes written pass^k. Under independent identical success probability p, these teaching models become 1−(1−p)^k and p^k respectively; independence is often unrealistic in deployed systems.

For rare failures, zero observations is not proof of zero risk. Under independent Bernoulli trials, the rough “rule of three” gives a 95% upper bound near 3/n after zero failures in n trials. With 300 clean trials, a failure rate around 1% is still compatible with this approximation. Correlated tasks or an unrepresentative test population weaken that reassurance further.

At least one success is not dependable repetition

With p = 0.8 and k = 3 independent attempts, these values are 0.992 and 0.512. They answer different product questions. For sampled code completions, a common finite-sample pass@k estimator with n generated candidates and c passing candidates is one minus the ratio of combinations shown below. It estimates oracle availability; an imperfect selector can perform worse.

When fewer than k failures exist, the numerator is zero. For n = 10, c = 2 and k = 3, the estimate is 1 − 56/120 = 0.5333. Keep failed, truncated and unparseable attempts in the declared denominator. This estimator does not remove task selection bias.

Source for the sampled-completion estimator: Evaluating Large Language Models Trained on Codehttps://arxiv.org/abs/2107.03374v2, first submitted July 7, 2021; v2 July 14, 2021. Versioned PDFhttps://arxiv.org/pdf/2107.03374v2. The numerical examples here are original and do not reproduce its model scores.

05 · Build graders, then evaluate the graders

Choose the strongest feasible evidence for each requirement. If a generated file must parse, execute a parser. If a balance must remain unchanged, inspect the balance. If an answer must cite an accessible source, verify the source and its relationship to the claim. A language-model judge is useful for semantic judgments that cannot be captured cheaply by deterministic checks, but it should not replace a stronger available verifier.

A rubric should define observable differences between scores. “Excellent” and “poor” are adjectives, not measurement rules. For an evidence-grounded answer, a rubric might distinguish unsupported central claims, partially supported claims with missing qualifications, and a complete answer whose important claims are supported by the supplied documents. Include examples near the decision boundary, not only obvious successes and failures.

Calibrate the judge on an independently annotated set. Measure agreement and inspect the disagreement matrix, especially false passes on consequential errors. An overall agreement score can look strong when almost all examples pass. Include adversarial examples, long fluent wrong answers, concise correct answers, irrelevant citations and attempts to instruct the judge from inside the candidate output.

The foundational MT-Bench and LLM-as-a-judge paperhttps://arxiv.org/abs/2306.05685v4 studies position, verbosity and self-enhancement biases. Its reported human agreement belongs to its tested models and tasks, not every future judge. The paper first appeared on 9 June 2023; the linked v4 is dated 24 December 2023. Read the PDFhttps://arxiv.org/pdf/2306.05685v4.

In your own protocol, randomize answer order, reverse a subset of pairs, blind provider names, constrain the rubric, and audit failures manually. Preserve judge versions and outputs so a score can be reproduced. Treat a judge upgrade as a measurement-system change: regrade a bridge set before interpreting a historical trend as model improvement.

A passing schema can still contain a false answer

The complete Python reference below includes a contract grader for an answer, its source identifiers and an answer/abstain status. It rejects malformed JSON and references outside the allowed source set. A deliberate counterexample changes “30 days” to “300 days” while preserving the schema and source ID: the contract grader still passes. This test is intentional. Structural validity, citation existence and semantic support are different properties; a real application needs a separately calibrated check against source content.

Keep grader failure distinct from candidate failure. An unavailable judge should produce an unknown evaluation state and a retry/review policy, not a fabricated zero or perfect score. Never execute a candidate response as Python merely to parse its structure.

06 · Arena: preference measurement and its boundaries

Arena, formerly LMArena and Chatbot Arena, made pairwise human preference a prominent part of model evaluation. In a blind comparison, a person sees responses to a prompt and expresses a preference before model identities are revealed. The statistical problem is to infer relative strengths from a sparse network of comparisons, not simply to count total wins across unequal opponents.

The Bradley–Terry model represents the probability that model A is preferred to B as a logistic function of their strength difference. The strengths are relative: adding the same constant to every strength leaves pairwise probabilities unchanged. A model with a higher estimated strength is more likely to win under the observed comparison process. This is not an absolute percentage of “intelligence.”

The original Chatbot Arena paperhttps://proceedings.mlr.press/v235/chiang24b.html is an ICML 2024 publication, with its official proceedings PDFhttps://raw.githubusercontent.com/mlresearch/v235/main/assets/chiang24b/chiang24b.pdf. The research establishes a concrete pairwise evaluation framework; it does not establish that public preference is sufficient for every deployment.

Read rankings with their uncertainty, category, date and methodology. Differences can be too small to distinguish confidently. The population of people submitting prompts is not necessarily your customer population. Fluent presentation can affect preference. Different categories can reverse a ranking. Connected comparison coverage matters: a model tested against a narrow or atypical opponent set requires careful estimation.

Arena's 14 July 2026 factuality announcementhttps://arena.ai/blog/factuality-in-arena describes a separate, initially optional Text/Search view combining preference with claim-level factuality audits. Its stated default factuality weight is 25%. This is a useful illustration of an evolving construct: changing the mixture changes what the ranking means. Do not splice the adjusted score into an older preference-only time series without explaining the break.

Use Arena as external evidence and a source of candidate models. Then run a matched evaluation on your own tasks. The engineering question is not “Who is number one?” but “Does this candidate improve our outcome under our constraints?”

Work through the pairwise model

The strengths theta are dimensionless log-odds parameters in this unscaled formulation. A difference of ln(3) gives a 75% predicted preference probability, not a 75% task accuracy. Fix one strength or constrain their sum to remove the arbitrary shared offset. Estimate strengths from the comparison graph, and inspect task and voter dependence before trusting naive independent-vote uncertainty. Ties require an explicit modeling policy rather than silently counting every tie as a decisive win.

07 · Current frontier: measuring validity, contamination and alignment

The research frontier increasingly concerns whether measurements justify their claims. Measuring what Mattershttps://arxiv.org/abs/2511.04703v1, submitted 3 November 2025 and formally published in the NeurIPS 2025 Datasets and Benchmarks trackhttps://proceedings.neurips.cc/paper_files/paper/2025/hash/1967e0fc3aa6cbbace562f5cb8e3954e-Abstract-Datasets_and_Benchmarks_Track.html, reviews 445 benchmarks with 29 expert reviewers. Its central concern is construct validity: whether the task and metric represent the phenomenon claimed. PDFhttps://arxiv.org/pdf/2511.04703v1. A practical implication is to draw the chain from real-world requirement to task, observation and score, then challenge each link.

Contamination is another threat. On 23 February 2026, OpenAI reported problems with SWE-bench Verifiedhttps://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/, including flawed tests and contamination. Its 59.4% flawed-test finding applies to an audited difficult subset of 138 problems, not automatically the entire benchmark. This is a provider-authored audit, and its scope matters. The lesson is to inspect failed cases and grading assumptions, not to assume every low score is a model defect.

Arena's 8 October 2026 Alignment Indexhttps://arena.ai/blog/ai-alignment-index describes three observable signals: unauthorized action, false attribution and deceptive completion. It uses judged evidence and adjusts for conversation length. This is an operational measurement of selected behaviors, not a proof of complete alignment. A narrow, clearly defined indicator can be useful precisely because its limits are visible.

These developments point toward an important discipline: evaluate the evaluation. Test whether a benchmark responds to genuine improvements, whether superficial changes can raise its score, whether labels are reproducible, and whether the score predicts a meaningful deployment outcome. A benchmark that is difficult but unrelated to the product is not automatically valuable.

08 · RAG evaluation: separate retrieval from answering

A retrieval-augmented system can fail before the language model sees the evidence. If the relevant policy paragraph is absent from the retrieved context, improving answer wording will not solve the underlying problem. Conversely, perfect retrieval does not guarantee that the model uses the evidence correctly. Evaluate the pipeline at its interfaces.

For retrieval, define relevance at the unit you actually retrieve: document, chunk or passage. Measure whether the required evidence appears within the allowed context budget. Recall@k is meaningful only with a suitable relevance set and a specified k. A large k can improve recall while increasing latency, cost and distracting context. Test reranking and chunking under the same token budget rather than comparing an inexpensive short context against a much larger one.

For generation, separate answer correctness from support. An answer can be factually correct but unsupported by the retrieved material; it can quote a source accurately while answering the wrong question. Include citation validity, entailment of important claims, completeness, abstention when evidence is insufficient, and handling of conflicting or outdated documents.

Use counterfactual tests. Remove the decisive passage and check whether the system acknowledges uncertainty. Replace a policy with a newer version and inspect whether the answer changes appropriately. Insert an irrelevant but persuasive paragraph and test resistance to distraction. Add a malicious instruction inside a retrieved document and verify that the system treats it as data rather than authority.

A fair ablation changes one component at a time: retrieval method, chunk size, reranker or generator. Keep the corpus snapshot, questions, answer rubric and budget fixed. The result should explain a mechanism—such as improved evidence recall on long policies—rather than merely report a higher aggregate number.

09 · Agent evaluation: outcomes, permissions and recovery

For an agent, evaluate both the final state and the path constraints. A travel assistant that books the right flight using an unauthorized payment method has not succeeded. A coding agent that fixes a test by deleting the test has not solved the intended problem. The verifier needs to inspect meaningful invariants, not only whether the final response sounds successful.

Construct a sandbox with explicit initial state, permitted actions and expected postconditions. Record tool inputs and outputs, environment changes, retries, timeouts and side effects. A task should define whether asking for clarification is acceptable, when human approval is required, and what should happen when a tool fails. Otherwise, the grader may punish sensible behavior or reward recklessness.

Anthropic's 9 January 2026 agent-evaluation guidehttps://www.anthropic.com/research/demystifying-evals-for-ai-agents emphasizes tasks, trials, graders, transcripts and outcomes. The engineering takeaway is to inspect trajectories when scores behave unexpectedly and to combine grader types. The guide is practitioner evidence, not a universal benchmark result.

Security evaluation needs an attacker model. Specify what the attacker controls: a retrieved page, email body, repository file or tool response. State the protected boundary and observable violation. Measure attack success and benign-task completion together; a system that refuses every request may look secure while being useless. Keep attack development separate from the final test set.

Recovery deserves its own tasks. Inject a transient timeout, stale credential, contradictory tool response or partial write, then verify whether the agent retries safely, rolls back, asks for help or stops. Do not assume that a successful rerun demonstrates a reliable repair policy. Compare against a retry-only baseline and account for extra compute.

For recursive self-improvement, freeze the evaluator outside the optimization loop. If the same system proposes changes, rewrites the tests and judges its own success, a rising score may reflect adaptation to the measurement process. Independent holdouts, resource budgets and explicit stopping rules are essential.

10 · Multimodal and physical AI need different evidence

Multimodal evaluation is not text evaluation with an image attached. A visual system may need to locate a small object, read a chart, distinguish a temporal event or combine speech with a scene. A text-only final answer can hide whether the model used the relevant modality or guessed from linguistic priors.

Design modality-specific counterfactuals. Change the number in a chart while leaving the question unchanged. Shift an event to a different video timestamp. Replace spoken words while preserving background noise. Mask the decisive region of an image. A system that gives the same answer after the decisive evidence changes may be relying on the wrong signal.

Report input resolution, frame sampling, audio duration, preprocessing and latency budgets. A model receiving every video frame is not being compared fairly with one receiving a few selected frames unless that resource difference is part of the question. Human ratings of aesthetics or naturalness should be separated from task correctness and accessibility.

Physical AI adds dynamics and safety. A robot policy evaluated only in simulation may exploit simulator artifacts or fail under sensor noise, wear, contact variation and unexpected objects. Report the distribution of environments, reset procedure, intervention rate, task completion, collision or constraint violations, and recovery behavior. A successful demonstration is not a reliability estimate.

The broader principle is to preserve the causal connection between the evidence and the claimed ability. If the claim is spatial reasoning, eliminate shortcuts that can solve the task without spatial evidence. If the claim is robust control, vary the physical conditions rather than replaying one favorable trajectory.

11 · An engineering architecture for continuous evaluation

A production evaluation system has four separable layers: a versioned task store, an execution harness, grading services and a decision/reporting layer. Keeping these boundaries explicit makes it possible to rerun old outputs through a new grader, compare a new model under an unchanged harness, and diagnose whether a regression came from the product or the measurement system.

The task store contains inputs, references, metadata, permissions and provenance. The harness creates isolated environments, enforces budgets and captures artifacts. Graders produce dimension-specific results with evidence. The reporting layer aggregates by task slice, uncertainty and cost, then applies release rules. Every result should be traceable to the exact versions that produced it.

Run inexpensive deterministic checks on every change. Run a representative regression suite in continuous integration. Schedule larger repeated-trial and adversarial evaluations when a model, prompt, retrieval index or tool changes materially. Expensive human review should focus on ambiguous or consequential cases rather than being allocated uniformly without purpose.

Connect evaluation to observability without confusing them. Observability records what happened in production: traces, errors, latency, usage and feedback. Evaluation applies a criterion to selected behavior. A trace is evidence for debugging; it is not automatically a quality label. A useful feedback loop turns appropriately redacted production failures into reviewed regression tasks, while keeping a separate holdout for unbiased assessment.

Protect the measurement pipeline itself. Candidate outputs are untrusted data and can contain instructions aimed at the judge. Sandbox executable submissions, restrict network access, prevent test leakage and log grading exceptions. A failed grader should be reported as unknown or infrastructure failure, not silently converted into a model pass or fail.

Industry landscape: control points, buyers and competition

Market evidence checked November 8, 2026. Product documentation establishes an offering, not independently measured efficacy. Competitive theses and unmet opportunities are Token Lab analysis; examples labeled synthetic are not company results.

The evaluation industry: who sells evidence, and who owns the decision?

Checked November 8, 2026. AI evaluation is several markets rather than one leaderboard business. Model labs buy feedback on capabilities; application teams buy regression workflows; enterprise risk teams need auditable acceptance evidence; infrastructure buyers compare cost and performance. A company can be strong in one market without replacing the others. The relevant question is what decision its evidence supports, under which conditions, and who bears responsibility when the evidence is wrong. This analysis separates disclosed products and transactions from our interpretation of business incentives. Funding totals and vendor claims are not independent evidence of evaluation accuracy.

The evaluation industry: who sells evidence, and who owns the decision?
Conceptual map of responsibilities; not a vendor score or measured coverage percentage.

Four decisions, four different evidence products

A public benchmark helps shortlist a model. A private task dataset helps decide whether to release an application change. A safety evaluation probes a defined threat model. A production outcome study asks whether the system improves the actual business process. These are complementary: the invoice assistant can win a writing-preference contest while using an unauthorized supplier record. Our market thesis is that evaluation providers should be classified by the decision they improve, not by whether they advertise an AI judge. Good procurement first specifies the population, task, acceptance criterion, failure cost and update frequency. Only then can a buyer compare human panels, code checks, model judges and managed services fairly.

Reference architecture: an independent release decision

This is a release-control architecture rather than a leaderboard scraper. A registry fixes baseline and candidate versions. Frozen cases and matched tool permissions feed isolated runners; the output is raw observations, including failures and cost. Versioned graders and human adjudication supply separate evidence to paired aggregation. The release gate uses predeclared quality and risk criteria before a limited canary. A final holdout is inaccessible to candidate optimization and is consumed only according to the evaluation protocol.

Keep development feedback separate from final-test feedback. Repeatedly selecting against a holdout makes it development data. The candidate must not write tasks, grader code, scores or promotion rules. Store configurations and raw outputs so an apparent gain can be traced to a model, retrieval, prompt or tool change. Canary monitoring checks deployment behavior but does not repair a contaminated offline comparison; rollback authority remains with the release system, not the evaluated agent.

Reference architecture: an independent release decision
Conceptual map of responsibilities; not a vendor score or measured coverage percentage.

Company and product positions

A selective map, not market share or a vendor certification. Coverage depends on edition, integration, deployment and enforcement mode.

Arena / LMArena

Ownership / organization
Arena Intelligence Inc.
Control point
Human preference and external evaluation

Arena sells external evaluation grounded in human feedback, distinct from operating a customer’s internal regression suite. Its commercial service launched September 16, 2025 with feedback analytics, representative samples and delivery commitments. Community participation is a distinctive input; maintaining credible neutrality while selling services is the commercial challenge. A preference win does not certify tool permissions or business correctness. Specify the Arena product and methodology: current Agent Mode differs from classic pairwise preference ranking, so one statistical description should not cover every product.

Arena commercial evaluation launch ↗https://arena.ai/blog/ai-evaluations

Artificial Analysis

Ownership / organization
Independent benchmarking company
Control point
Model and infrastructure selection

Artificial Analysis measures models, agents, serving providers and hardware, with public information and free/commercial API offerings. Its likely buyer needs selection and procurement intelligence rather than an application release gate. Comparability across quality, performance and economic dimensions is the proposed value. An aggregate index remains a constructed metric: preserve version, settings and date. The current Intelligence Index methodology describes a primarily text-based English-language suite; multilingual and other modalities need separate evidence. Use results to shortlist, then test the customer’s actual workload.

Artificial Analysis index methodology ↗https://artificialanalysis.ai/methodology/intelligence-benchmarking

Scale AI / SEAL / Scale Labs

Ownership / organization
Independent company; Meta non-voting minority investment
Control point
Expert data and research evaluation services

Scale combines expert-data operations, evaluation services and research through SEAL and Scale Labs. Its potential advantage is organizing difficult tasks and expert adjudication at scale, particularly where simple automatic labels are inadequate. Meta’s minority investment is a real ownership fact, not proof that all Scale evaluations are biased or that Meta bought the company. Enterprise diligence should examine data access, contractual separation and publication procedures. Human expertise can improve validity, but service delivery cost and consistent adjudication remain important commercial constraints.

Scale investment and independence statement ↗https://scale.com/blog/scale-ai-announces-next-phase-of-company-evolution

Galileo / Cisco

Ownership / organization
Acquisition completed May 22, 2026
Control point
Evaluation-to-runtime guardrails

Galileo connects evaluation, observability and runtime guardrails, including compact Luna evaluators. Cisco completed the acquisition on May 22, 2026: describing Galileo as still independent would be wrong. The strategic opportunity is taking an offline quality criterion into affordable production enforcement and enterprise distribution. Cost and latency claims remain vendor assertions unless tested under matched workloads. Even a fast scorer needs calibration and a defined action on uncertainty; it does not replace resource authorization, exact approval or recovery from a committed side effect.

Cisco Galileo acquisition completion ↗https://www.cisco.com/site/us/en/about/corporate-development/acquisitions/galileo/index.html

Databricks / MLflow

Ownership / organization
Databricks platform and open-source MLflow
Control point
Data-governed evaluation workflow

MLflow connects tracing, evaluation, judges, prompts and human feedback to a familiar model-development workflow. Databricks adds managed operation and proximity to enterprise data governance. Our interpretation is that its advantage comes from data gravity and existing procurement, while open-source APIs support portability. Check the exact version and edition rather than assuming every historical release had identical capabilities. A governed dataset and model registry are valuable, but neither defines whether an external payment or deletion was authorized for this particular request.

Databricks current MLflow observability and portability FAQ ↗https://docs.databricks.com/aws/en/mlflow3/genai/tracing/faq

Microsoft Foundry

Ownership / organization
Microsoft cloud platform
Control point
Trace-to-dataset and release integration

Microsoft Foundry brings tracing, evaluation, monitoring and CI/CD quality gates into an enterprise development environment. Its documentation describes converting agent traces into curated, versioned evaluation datasets. The business advantage is reducing integration and procurement friction for Microsoft-centered organizations. A specialist must justify why another platform improves decisions enough to offset that friction. Dataset creation still needs representative selection and sensitive-data controls. Automatic collection of traces does not establish that labels are correct or that deployed agents have least privilege.

Microsoft Foundry traces to evaluation datasets ↗https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/traces-to-dataset

AWS Bedrock / Google Cloud

Ownership / organization
Separate cloud-platform offerings
Control point
Managed model and application evaluation

AWS Bedrock offers model/RAG evaluation, human evaluation options and model-as-judge jobs. Google Cloud’s Gen AI evaluation documentation covers models and applications, including third-party models. These are separate products, grouped here only because cloud distribution creates similar competitive pressure. Existing infrastructure and contracts can make adoption convenient; cross-provider neutrality and portability remain diligence questions. For either service, managed execution does not choose the right task population or validate a judge automatically. Public results must preserve configuration, budget and metric version.

Amazon Bedrock Evaluations ↗https://aws.amazon.com/bedrock/evaluations/ · Google Cloud Gen AI evaluation overview ↗https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview

Economics: labels and reproducible environments are expensive

The cost of an evaluation is not merely the judge-model token bill. It includes task design, expert adjudication, environment reset, inference, repeated trials and maintenance as the product changes. A provider creates value if it lowers the cost of a defensible decision, not if it produces more scores. Our interpretation is that reusable expert workflows, hard-to-obtain representative tasks, contamination controls and reproducible agent environments can be stronger advantages than another generic evaluator. However, customer-specific labels may not be legally or operationally reusable across tenants. A supposed data network effect must therefore explain consent, isolation, transferability and how learning generalizes without exposing confidential tasks.

Technical detail: three different contracts for quality

A benchmark asks whether a model solves a declared task set. A preference arena asks which response sampled users prefer under its presentation and sampling process. A deployment evaluation asks whether the configured system completes the buyer’s workflow within quality, safety, latency and cost limits. These are different targets. A preferred answer may be unsupported; a correct answer may arrive too late; a capable model may fail after a retrieval or tool integration error. Keep the evidence streams separate before combining them in a procurement decision.

For a fair comparison, run both candidates on the same cases and inspect paired differences. Freeze the dataset, rubric, grader version and tool permissions before selection; keep the final holdout outside optimization. An LLM judge requires calibration against adjudicated examples and checks for position bias, style bias and shared errors with the candidate. Report uncertainty and task-family breakdowns. A higher aggregate score does not establish improvement if one costly failure family worsens. The defensible product is a repeatable decision process, not a lone leaderboard number.

Technical detail: three different contracts for quality
Conceptual map of responsibilities; not a vendor score or measured coverage percentage.

The unresolved gap: from a score to a safe decision

A judge can be consistent yet wrong; several judges can share the same blind spot. A public rank can reflect language, task and presentation preferences that differ from the buyer’s workload. Agent tests also depend on harness, tools, external state, budget and reset policy, so model name alone is an incomplete unit of comparison. Production acceptance must combine task outcomes with security invariants: no cross-tenant access, no unauthorized side effects, no hidden escalation. Our view is that the next valuable layer is evidence integration with clear uncertainty and ownership. It must connect expert judgment, reproducible executions and business outcomes without disguising an automated proxy as ground truth.

Acquisitions, capital and strategic consequences

Capital and ownership: independence requires precision

Arena announced a $150 million Series A on January 6, 2026. Scale announced Meta investment on June 12, 2025; Meta subsequently reported a non-voting minority stake and $13.79 billion included in nonmarketable equity investments. That is not an acquisition of all Scale, nor a simple price for its evaluation business. Cisco completed its Galileo acquisition on May 22, 2026, following the April 9 intent announcement. These facts show capital and platform interest, not evaluation quality. Our analysis treats commercial relationships and ownership as diligence topics: ask about access to customer data, publication rules, model-selection procedures and conflict management. Do not infer biased results merely from an investor or customer relationship.

Primary product and transaction sources

14 · How AI giants compete through evaluation

Frontier-model competition is partly a competition over what counts as progress. A provider can emphasize mathematical reasoning, coding, agentic task completion, multimodal understanding, latency, price or safety. Those dimensions matter, but selecting the favorable subset can create a misleading overall impression. Read a model launch as a collection of scoped claims rather than a universal ranking.

For every headline comparison, ask whether model versions, prompts, tools, inference budgets and grader settings match. A result with extensive search, retries and a large compute budget does not measure the same product experience as a single low-latency response. If proprietary scaffolding contributes to the score, describe the result as a system result unless the contribution has been isolated.

Providers have real advantages in evaluating their own systems: detailed access to versions, logs and failure modes. They also have incentives to present progress favorably. Independent evaluators can diversify the evidence, but independence is not guaranteed merely by being a separate company. Examine funding, commercial relationships, publication policies, access privileges and the ability to report negative findings.

Enterprise buyers should build a small portfolio of evidence: public capability measurements for breadth, domain-specific held-out tasks for fit, adversarial tests for boundaries, and randomized online experiments for user impact. Require a candidate to improve the dimensions that matter to the business rather than chasing every public leaderboard movement.

The durable competitive advantage may lie in a high-quality feedback loop: representative failures become reviewed tasks, interventions are tested fairly, successful changes are deployed cautiously, and production outcomes verify the hypothesis. This is an engineering interpretation of the market, not a claim that any named company has already solved that loop.

Cloud platforms compete for the evaluation workflow

Microsoft Foundryhttps://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-overview documents evaluation of existing datasets, generated model or agent outputs, and production traces. This is a workflow integration proposition: keeping execution, permissions and evidence near the deployment environment. Buyers should still export a versioned task set and verify that scores remain reproducible outside the portal.

Google Cloud's Gen AI evaluation servicehttps://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview documents evaluating third-party models as well as models in its own ecosystem. This highlights an important separation: the generator being tested need not be the judge. A cross-provider evaluation is not automatically independent, however; check the judge's calibration and version on the target domain.

Amazon Bedrockhttps://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html documents model-evaluation jobs that use a separate LLM to score responses and explain the score. A generated explanation is useful evidence for review, not proof that the score is correct. Keep an adjudicated calibration set, record evaluator configuration and include judge cost in the experiment ledger.

OpenAI and Anthropic also contribute provider-authored audits and engineering guidance, discussed earlier. Across these companies, the competitive axes are model quality, integration, operating cost, access to feedback and trust in measurement. The practical strategy is to keep your evaluation contract portable while using integrations where they reduce work. Do not give the vendor's default rubric authority to redefine your business outcome.

Continue the learning path

Read the companion handbook: evaluation and observability · AI security · AI infrastructure

Learning library · Interview and projects · Editorial standards and corrections

Educational extensions: OpenAI Academyhttps://academy.openai.com/ · Anthropic learning resourceshttps://academy.claude.com/ · Microsoft AI traininghttps://learn.microsoft.com/en-us/ai/.

Primary references are attached to the relevant claims. Conceptual diagrams, synthetic arithmetic and research proposals are original teaching material. Vendor descriptions establish product positioning, not independent proof of performance. Corrections should identify the claim, its source and the proposed replacement.

Practice and verification

Stage 5: monitor drift without training on the final exam

After deployment, observability provides operational outcomes, delayed feedback and incident candidates. Evaluation decides which observations satisfy a rubric. Join the two with versioned identifiers, but do not treat every complaint as ground truth or every silent user as a success. Feedback is selected and delayed. Report label coverage, disagreement and review policy alongside quality estimates.

When a production failure becomes a regression task, assign it to development or validation. Fixing that example and rerunning it demonstrates regression prevention, not independent generalization. Maintain a separate holdout or create a new evaluation slice from later data with documented separation. A self-improving agent must not control the test answers, grader, resource accounting and release threshold used to certify its own progress.

Look for several kinds of drift: changed user requests, changed documents, changed tool behavior, model-version updates and grader changes. A score shift after replacing the judge is not automatically a capability shift. Where possible, rerun stored outputs under both graders to estimate the measurement change, and rerun the old system on current tasks to separate traffic shift from code regression.

The final deliverable is an evidence package: task contract, version manifest, paired results, critical slices, grader calibration, failure examples, resource ledger, release decision and rollback conditions. Public Arena results and company benchmarks remain valuable context for candidate selection, but the system earns deployment through evidence matched to its actual job. An expert can explain both what the evaluation establishes and what it leaves unresolved; the goal is not to produce an impressive number, but to make a decision that survives scrutiny.

Practice and verification, after the explanation

Acceptance test: another engineer receives the contract and can determine whether a missing citation, a refused unauthorized request and a timed-out grader count as success, failure or unknown. If they cannot, improve the specification before running more models. The research question is external validity: which offline measurements predict the actual business outcome under later traffic, and where does that relationship break?

Acceptance test: change only a document’s tenant ACL while preserving its text. The assistant must stop retrieving it for the unauthorized user. Then remove the relevant policy and verify abstention rather than a memorized answer. The open research problem is constructing difficult, representative private tasks without making evaluation opaque; publish the task-generation and review process even when the confidential task contents cannot be released.

Acceptance test: a fluent answer cites a real document ID but changes “30 days” to “300 days.” The schema grader should pass, while the support check should fail. Also make the grader unavailable and confirm an unknown grading state rather than a model failure or automatic pass. An open question is whether a cheaper judge preserves detection of rare consequential failures; compare class-conditional errors and review cost, not aggregate agreement alone.

Acceptance test: duplicate every case ten times and ensure the analysis does not claim ten times as many independent customers. Preserve infrastructure errors under the declared policy and report their rate separately. The research question is how to allocate repeated trials versus new cases under a fixed budget; more repeated generations can reduce uncertainty about one task while doing little to establish generalization.

Acceptance test: inject one unauthorized-action attempt into an otherwise high-scoring candidate and verify that the release gate blocks it. Test rollback after a document schema change and confirm that already committed actions are reconciled rather than replayed. The open question is how offline evidence transfers to a changing operating environment; evaluate that prediction against controlled rollout outcomes and retain negative results.

15 · Open research problems worth a serious experiment

External validity. Can an offline score predict a deployment outcome across organizations? Construct several domain-specific task sets, preregister the predicted ordering, and compare it with a later controlled rollout. Report where the ordering reverses. A single positive correlation on one product is not a universal transfer law.

Adaptive contamination. How can a benchmark remain informative when developers repeatedly optimize against it? Compare fixed public tasks, private held-outs and periodically refreshed tasks under a controlled optimization process. Measure performance on a final unseen distribution. The aim is to separate genuine generalization from learning the measurement interface.

Judge robustness. Can a grader resist style changes and adversarial instructions without losing sensitivity to real improvements? Create matched output pairs that preserve correctness but change verbosity, formatting or persuasion. Then create pairs with identical style but different factual quality. The grader should be invariant to the former when irrelevant and responsive to the latter.

Long-horizon reliability. How do errors accumulate across tool interactions? Vary task length while controlling task type, and distinguish exposure from intrinsic deterioration. A longer task creates more opportunities to fail; a higher session-level failure rate does not alone show that per-action reliability worsened.

Evaluation under limited budgets. Which tasks should be labeled or rerun next? Compare uniform sampling with an adaptive policy, but preserve a probability sample for unbiased reporting. An adaptive strategy can find more failures while producing a distorted raw failure rate if selection probabilities are ignored.

Reward hacking in self-improving systems. Does an optimization loop improve the target capability or exploit its grader? Hold a second evaluator and task set outside the loop, impose equal compute budgets, and compare against a non-adaptive baseline. Publish negative results and evidence of evaluator exploitation. A rising training reward is a hypothesis about improvement, not its verification.

16 · A four-week capstone with a defensible release decision

Build a small document-support agent that answers policy questions and can propose—but not execute—an account change. The baseline uses one retrieval method and a fixed prompt. The candidate changes exactly one component. Use synthetic documents and mock tools so no personal data or real transactions are involved.

In week one, define 60–100 tasks across straightforward retrieval, conflicting evidence, missing evidence and unauthorized requests. Write a rubric and deterministic postcondition checks. Reserve an untouched test partition. Annotate a small calibration set with two reviewers and resolve disagreements by revising the rubric rather than averaging incompatible interpretations.

In week two, implement the harness with task IDs, seeds where applicable, budget limits, artifacts and explicit infrastructure-error states. Run baseline and candidate on the same tasks. Repeat a subset to estimate execution variance. Add a local report showing paired outcomes, critical slices, latency and cost per accepted result.

In week three, audit failures. Test whether the judge can be manipulated by candidate text. Perform an ablation that restores the baseline component, and a budget-matched comparison that prevents extra compute from masquerading as algorithmic improvement. Add discovered failures to the development set while keeping the final test untouched.

In week four, make the release decision under a preregistered rule. A strong report includes the task distribution, exact versions, uncertainty, failed examples, resource costs, safety constraints and reasons to defer deployment. The reference success criterion is not “candidate wins.” It is “another engineer can reproduce and critically evaluate the conclusion.”

For a research extension, measure whether the offline decision predicts a simulated deployment distribution with different proportions of difficult tasks. Explain the discrepancy when distribution shift changes the conclusion.

17 · Original interview practice and scoring

These are original practice questions, not reported questions from any company. Use the same 1–5 progression for each: 1 identifies the relevant concept; 2 gives a correct mechanism; 3 applies it to the concrete example; 4 proposes a sound experiment with controls; 5 explains assumptions, failure cases and the limits of the conclusion.

1. Candidate B wins 55% of public preference comparisons. Should you replace the support model?

Hint 1: Identify the population and outcome being measured.

Hint 2: Distinguish preference from policy correctness and authorization.

Reveal answer and analysis

Answer: Not on that evidence alone. Run a paired comparison on representative support tasks with correctness, policy and safety graders, then examine uncertainty and cost. A strong answer also checks the public comparison's confidence interval, category and versions, and proposes a guarded online experiment after offline gates pass.

2. Why can 90% judge agreement still be unacceptable?

Hint 1: Consider an imbalanced dataset.

Hint 2: Ask which disagreement type is costly.

Reveal answer and analysis

Answer: If almost all examples are acceptable, a judge that passes everything can achieve high agreement while missing every dangerous failure. Inspect class-conditional errors, calibrate on consequential cases and audit false passes. A top answer explains why agreement with one imperfect annotator is not ground truth and defines an adjudication process.

3. An agent succeeds at least once in five runs but often fails the first run. Which metric matters?

Hint 1: What attempts can the product afford?

Hint 2: Can it identify the successful attempt reliably?

Reveal answer and analysis

Answer: Pass@5 is relevant only if five attempts and reliable selection are available. For a single-shot workflow, first-attempt success and failure severity matter. For repeated dependable execution, measure all-attempt reliability and task-level variance. Include latency, cost, correlated failures and side effects from unsuccessful attempts.

4. A retrieval change raises answer quality. How do you establish why?

Hint 1: Separate evidence availability from generation.

Hint 2: Match the context and compute budgets.

Reveal answer and analysis

Answer: Measure evidence recall, answer support and correctness on the same tasks, keeping the generator fixed. Compare retrieval-only ablations under equal context budgets. Inspect whether gains concentrate on previously missing evidence and test distraction and stale-document cases. A high-scoring answer avoids claiming causality from aggregate correlation alone.

5. A self-improving agent raises its own benchmark score every day. What would convince you it improved?

Hint 1: Identify what the agent can modify or observe.

Hint 2: Keep independent evidence outside the loop.

Reveal answer and analysis

Answer: Freeze an independent evaluator and held-out tasks, compare under equal resource budgets, and check whether gains transfer to new tasks and robust behavior. Audit grader exploitation and contamination. The strongest answer defines a preregistered stopping rule, preserves failed runs and distinguishes increased capability from increased search effort.

Reproduce the arithmetic: a complete local reference

Imagine evaluating a replacement support agent on twenty matched tasks. Twelve tasks pass under both versions, four pass only under B, two pass only under A, and two fail under both. Model A scores 14/20 and B scores 16/20. We resample task pairs, keeping the relationship between outcomes intact. The resulting interval is wide: this demonstration does not establish that B is better in the population. The individual Wilson interval is a separate description of B's success rate; overlap between two marginal intervals is not the paired comparison test.

Here n is the number of independent tasks and outcomes are binary. The second expression is the exact one-sided upper confidence bound on failure probability after zero observed failures, assuming independent identically distributed Bernoulli trials; alpha = 0.05 gives 95% confidence. Correlated tasks or an unrepresentative sample invalidate that interpretation.

Revisit the opening score-and-uncertainty diagram. Its error bars are marginal Wilson intervals; the paired interval below addresses the difference directly.

The bootstrap is a percentile approximation, not a guarantee of calibrated coverage with just twenty examples. If multiple prompts belong to one customer or document, resample customers or documents. More repeated generations per item measure stochastic variability; they do not manufacture more independent customers.

Try the denominator locally

No input leaves your browser. This is an arithmetic illustration, not a production risk estimate.

Runtime: Python 3.10 or newer; standard library only. The evaluation bootstrap uses seed 17 and 10,000 resamples. Observability arithmetic is deterministic and needs no seed. These programs make no network or model calls.

Download the complete Python reference

python3 evaluation-audit.py
"""Synthetic paired evaluation; Python 3.10+, standard library, seed 17.
No API calls. Items, rather than individual judgments, are bootstrap units.
"""
import math
import random
import json
from statistics import mean


def wilson(successes, n, z=1.959963984540054):
    if not 0 <= successes <= n or n <= 0:
        raise ValueError('Require 0 <= successes <= n and n > 0')
    p = successes / n
    den = 1 + z*z/n
    center = (p + z*z/(2*n)) / den
    radius = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / den
    return center-radius, center+radius


def paired_bootstrap(a, b, repetitions=10000, seed=17):
    if len(a) != len(b) or not a:
        raise ValueError('Require nonempty matched items')
    if not isinstance(repetitions, int) or repetitions < 100:
        raise ValueError('Use at least 100 bootstrap repetitions')
    if any(v not in (0, 1) for v in a + b):
        raise ValueError('This example requires binary outcomes')
    rng = random.Random(seed)
    differences = [y-x for x, y in zip(a, b)]
    means = sorted(mean(rng.choices(differences, k=len(a)))
                   for _ in range(repetitions))
    return mean(differences), means[int(.025*repetitions)], means[int(.975*repetitions)]



def grade_response(raw, allowed_sources, expected_status):
    """Original deterministic contract grader, NOT a semantic truth judge.
    Candidate text is parsed as data; never eval(), execute(), or prompt a judge.
    """
    try:
        response = json.loads(raw)
    except (json.JSONDecodeError, TypeError):
        return {"schema": False, "evidence_ids": False, "status": False}
    schema = (isinstance(response, dict)
              and set(response) == {"answer", "sources", "status"}
              and isinstance(response["answer"], str)
              and isinstance(response["sources"], list)
              and all(isinstance(x, str) for x in response["sources"])
              and response["status"] in ("answer", "abstain"))
    if not schema:
        return {"schema": False, "evidence_ids": False, "status": False}
    return {"schema": True,
            "evidence_ids": (all(x in allowed_sources for x in response["sources"])
                             and (response["status"] != "answer" or bool(response["sources"]))),
            "status": response["status"] == expected_status}


valid = '{"answer":"The policy says 30 days.","sources":["policy-v3"],"status":"answer"}'
assert all(grade_response(valid, {"policy-v3"}, "answer").values())
assert not grade_response('not JSON', set(), 'abstain')['schema']
assert not grade_response(valid, {"policy-v4"}, "answer")['evidence_ids']
assert not grade_response(valid, {"policy-v3"}, "abstain")['status']
# A plausible-looking lie can pass these structural checks. Add human/semantic
# review against the actual source; do not call this contract pass "correctness".
lie = valid.replace('30 days', '300 days')
assert all(grade_response(lie, {"policy-v3"}, "answer").values())

# Same 20 synthetic tasks: 12 both pass, 4 only B, 2 only A, 2 neither.
a = [1]*12 + [0]*4 + [1]*2 + [0]*2
b = [1]*12 + [1]*4 + [0]*2 + [0]*2
assert sum(a) == 14 and sum(b) == 16
low, high = wilson(sum(b), len(b))
delta, lower, upper = paired_bootstrap(a, b)
assert lower <= 0 <= upper  # small sample does not establish superiority
assert paired_bootstrap(a, a) == (0, 0, 0)
assert wilson(0, 20)[0] >= -1e-12
assert math.isclose(1 - .05**(1/100), .029513049607039932)
# Trial independence is an assumption, not a property guaranteed by a model.
assert math.isclose(1-(1-.8)**3, .992)
assert math.isclose(.8**3, .512)
print(f'A: {mean(a):.1%}; B: {mean(b):.1%}')
print(f'B Wilson 95% interval: [{low:.3f}, {high:.3f}]')
print(f'Paired delta: {delta:.1%}; bootstrap 95% interval: [{lower:.3f}, {upper:.3f}]')
print('Zero failures in 100: one-sided 95% upper bound 2.95%')
print('All evaluation checks passed; synthetic data, not model performance.')

Expected output:

A: 70.0%; B: 80.0%
B Wilson 95% interval: [0.584, 0.919]
Paired delta: 10.0%; bootstrap 95% interval: [-0.150, 0.350]
Zero failures in 100: one-sided 95% upper bound 2.95%
All evaluation checks passed; synthetic data, not model performance.

Test the failure cases: empty samples are rejected; the evaluation comparison of a model with itself has zero difference; the trace algorithm rejects cycles. Next add clustered tasks or missing spans, and document exactly which assumptions stop holding. Unit checks validate this implementation, not a deployed model.

Foundations explained

Review the shared reasoning behind this method, then use the worked checks to identify gaps in your understanding.

An experiment estimates a specific quantity under stated conditions. Define the unit of comparison before averaging: a request, problem, training seed or robot episode are not interchangeable. A paired comparison runs methods on the same cases so difficulty is controlled. An ablation changes one mechanism while keeping relevant data, budgets and evaluation fixed; removing a component and also giving the baseline fewer resources confounds the conclusion.

Work it out before reading the solution

On three identical cases, method A scores [0,1,1] and B scores [1,0,1]. What does the mean improvement tell you, and what does it hide?

Reasoning, step by step

Paired differences are [1,−1,0], so the average change is zero. One case improves and another regresses. An unchanged average is not equivalence, and three cases cannot establish a reliable population effect. Inspect subgroups and uncertainty; repeated correlated samples are not independent evidence.

Why this matters in practice

A frontier result needs a chain from claim to measurement: task, dataset split, model version, budget, metric and failure cases. Separate a paper’s reported result from a local teaching example. Reproducing arithmetic does not reproduce training, and adapting to a benchmark can invalidate it as a held-out test.

Evidence of understanding

Write the claim you would falsify, lock the evaluation before tuning and save configuration plus raw outcomes. Compare a credible baseline at equal budget and report negative results. For self-improvement, keep external held-out tasks and measure regressions as well as gains.

Run the complete Python example

Python 3.10+ · standard library only · deterministic, no random seed needed · no network. Save the file and run the command below. These are teaching checks, not model benchmarks.

Download Python · evidence.py
python3 evidence.py
# Python 3.10+; standard library; deterministic; no network.
a, b = [0, 1, 1], [1, 0, 1]
assert len(a) == len(b)
differences = [new - old for old, new in zip(a, b)]
mean_change = sum(differences) / len(differences)
assert mean_change == 0
assert sum(x > 0 for x in differences) == 1
assert sum(x < 0 for x in differences) == 1
print("mean change=0; improved=1; regressed=1; unchanged=1")

Expected output

mean change=0; improved=1; regressed=1; unchanged=1

Review the prerequisite →

Foundation revised:

Your next learning step

Freeze the holdout; compare paired runs, uncertainty, failures and compute budgets.

Build / verify → · Continue your route →

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope