Token Lab検索

Translation status: this page is not fully translated. Read the English edition.

システムの主線は翻訳済みです。参考章、図中の文字、一部の実践説明は英語のままです。

AI EVALUATION / RESEARCH & ENGINEERING HANDBOOK

AI evaluation: from measurements to decisions, research and market power

A field guide to traditional ML, LLM and agent evaluation: statistics, Arena, engineering, research frontiers, companies and acquisition evidence.

Jump to the runnable reference and tests

出典とバージョン
Contents · chapters, research and reference code
この記事の目次
第1部 / エンドツーエンドのシステム

評価:リリース可能かを判断する

この独自事例では請求書アシスタントを追います。テナント内の資料を読み支払いを提案しますが、実行には別途正確な認可が必要です。場面と実験は合成教材であり、実際の導入事例ではありません。

評価:リリース可能かを判断する: conceptual lifecycle diagram
独自の概念図です。線は依存関係や根拠を示し、実測性能ではありません。

最初に決めること:請求書エージェントを公開できるか

合成例の助手は請求書を説明し、テナント資料を読み、支払を提案します。実行には正確な操作に対する独立した認可が必要です。文章の改善や検索量の増加だけでは公開を正当化できません。対象の業務母集団と分析単位を先に定めます。6回再試行しても顧客案件は1件であり、全試行の費用とリスクを計上します。

評価契約はモデル、プロンプト、資料、ツール、予算、期限、採点規則、除外条件を固定します。根拠ある回答または適切な棄権、時間、権限を別々に報告し、流暢さで越権を相殺しません。採点器の停止は不明とし、黙って除外しません。未完了案件は事前の規則で判断します。証拠不足なら公開しない判断も有効です。

段階1:業務を再現可能な環境にする

通常の請求、規約欠落、版の矛盾、古い残高、他テナント資料、受取人変更、承認撤回、重複送信を含めます。各案件で初期状態、許される終状態、禁止する副作用を定義します。支払準備と支払実行は別です。教材では合成文書と模擬ツールを使用します。

顧客、文書系列、時間によって開発・検証・最終保留集を分けます。同じ請求書の言い換え10件は通常10個の独立観測ではありません。正しい根拠を直接渡す条件で生成を切り分け、根拠欠落条件で棄権を調べます。本番が簡単な案件90%、難しい案件10%なら、半々の診断集合で全体性能を推定せず、層別結果と重みを示します。

段階2:観測を根拠ある判断へ変える

構造、許可文書、支払引数、台帳終状態、重複効果、承認の一度限りの消費は決定的な検査で判断します。意味的な裏付けに人または校正したモデル採点器を使います。呼出記録は試行の証拠であり、業務状態が結果の証拠です。実在する引用でも主張を支持しない場合があります。

独立にレビューした標本で誤通過と誤棄却を分けて測ります。合成1000件中50件が誤りなら、常に合格とする採点器は正解率95%でも全誤りを見逃します。順序、長さ、誘導文を変えて偏りを確認します。比較前に採点器を固定し、停止時の不明率、ラベル被覆率、カテゴリ別誤りを報告します。

段階3:変数を制御して不確実性を読む

同じ案件と予算で基準系と候補を実行します。反復はランダム性を測りますが顧客数を増やしません。モデル固定で検索を比較し、検索固定でモデルを比較してから組合せを調べます。追加文脈と再試行を明記します。APIの種子は版をまたぐ決定性を保証しません。

合成200独立案件で両者合格150、候補のみ20、基準のみ10、両者失敗20なら、基準80%、候補85%、差5ポイントです。不一致30組が差の不確実性に関わります。教育用正規近似の標準誤差は約2.72ポイントで、95%区間はわずかにゼロをまたぎます。顧客内依存があるときは個々の回答ではなく顧客単位で再標本化します。

段階4:高得点でも公開を止められる基準

結果を見る前に実用的な最小改善、不確実性の許容範囲、費用・遅延上限、絶対的制約を定義します。平均改善で無許可支払は許されません。攻撃的100例で失敗ゼロでもリスクゼロではありません。独立ベルヌーイ仮定で片側95%上限は約2.95%で、非無作為の攻撃集合から本番発生率は推定できません。

副作用を起こさないシャドー実行から始め、外部権限制御を維持した限定カナリアへ進みます。同時期の群を比較し、固定期間または適切な逐次分析を使います。好結果が出た時点で停止しません。ロールバックでは確定済み操作を照合し、再実行しません。停止条件ごとに責任者と復旧根拠が必要です。

How the four disciplines fit together

Security determines which actions are permitted. Infrastructure executes work within capacity and failure constraints. Observability preserves evidence of what happened. Evaluation tests whether a change improves the outcome under a declared contract. None substitutes for the others.

第2部 / 参考章

仕組み・研究・市場背景 — 以下の参考資料は英語です

The following chapters provide the mathematical examples, experiments, research papers and dated company evidence behind the operational story. Return to the lifecycle when deciding where each mechanism belongs.

Synthetic paired evaluation scores with confidence intervals.
Original illustration: uncertain estimates require paired analysis, not a leaderboard-style verdict.
Capability, preference and deployment evaluation answer different questions.
Conceptual taxonomy, not experimental evidence.

01 · Evaluation begins with a decision, not a leaderboard

Imagine that an engineering team is replacing the model behind a customer-support assistant. The new model wins more public head-to-head comparisons and writes more polished answers. Yet its first production week produces more incorrect refunds. Both observations can be true. A public preference ranking measures how a particular population judges particular responses; a refund workflow requires correct policy interpretation, authorization and state transitions. The mistake is not necessarily in either measurement. It is in treating one measurement as a substitute for another.

Definition. An AI evaluation is a specified procedure for gathering evidence about a system's behavior under stated conditions and using that evidence to support a decision. A benchmark is a reusable collection of tasks and measurement rules. A metric summarizes observations. A grader turns an output, action or environment state into a judgment. A harness controls execution and records the conditions. These objects are related, but none is interchangeable with the others.

The decision should be written before collecting scores: “Release candidate B only if it improves successful resolution on our target tasks without increasing unauthorized actions, while meeting our latency and cost budgets.” This statement identifies the target population, the benefit, the constraints and the deployment boundary. It also exposes missing knowledge: what counts as successful resolution, how frequently dangerous cases occur, and how confidently we can measure the difference.

Before this lesson, you should understand a sample mean as an estimate of a population property, a conditional probability as a statement restricted to specified conditions, and a train/test split as a barrier against adapting to the answers. You do not need to have memorized advanced statistics. You do need to recognize that running the same task repeatedly measures execution variability, while collecting new tasks measures a different source of uncertainty.

By the end, you should be able to design a release gate, interpret an Arena-style comparison, calibrate an automated judge, build an auditable evaluation pipeline, and explain why a high score can coexist with an unsuitable product. The examples in this chapter are original teaching examples unless explicitly attributed; they are not measurements of commercial models.

Choose a reading route

First encounter: read the definitions and worked examples, then predict the local experiment result before changing its input. Engineering implementation: focus on the data contract, architecture and Python reference, then write one failure test of your own. Research reading: inspect the cited paper versions, challenge their evaluation assumptions, and choose one open problem with a falsifiable baseline.

02 · Traditional ML, LLMs and agents: what changes?

Traditional supervised evaluation often starts with a fixed input, a reference label and a loss function. A classifier receives an image and predicts a class; a forecasting model predicts a number. Accuracy, precision, recall, calibration and error by subgroup can be useful because the target is comparatively explicit. Even here, labels can be wrong, class distributions can shift, and the cost of a false negative can differ from that of a false positive. Traditional evaluation was never automatically simple or unbiased.

For a language model, several outputs may be acceptable. A concise explanation and a longer explanation can both be correct; exact string matching can reject one for irrelevant reasons. Meanwhile, an apparently correct conclusion may contain fabricated evidence. Evaluation therefore separates dimensions such as factual correctness, instruction following, evidence support, completeness and usefulness. Combining them into one number is a policy choice that requires explicit weights.

An agent adds actions and environmental state. It can finish a task through different trajectories, use extra tools, encounter failures, recover, or damage something unrelated to the task. A final paragraph saying “Done” is not evidence that the task was completed. A database query, repository diff or sandbox state may be the appropriate verifier. The evaluated object is now the model plus its prompt, tool definitions, memory, permissions, retry logic and surrounding application.

Three distinctions prevent many misleading comparisons. First, model evaluation isolates a model as far as possible; system evaluation tests the assembled product. Second, capability evaluation asks what can be accomplished under a budget; reliability evaluation asks how consistently the behavior occurs. Third, offline evaluation estimates performance on controlled tasks, whereas online experimentation estimates effects on users under an actual deployment.

None replaces the others. A model can be capable but unreliable, a reliable product can depend on expensive scaffolding, and a statistically positive online result can conceal a severe safety regression. Keep the axes visible until the decision requires an explicit tradeoff.

03 · Write an evaluation contract

A useful evaluation contract names the task distribution, unit of analysis, execution budget, grading rubric, exclusion policy and release rule. For a support agent, a unit might be one customer case, not one model call. For a coding assistant, it might be one repository issue, not every generated patch. If a case produces ten retries, treating those retries as ten independent customers exaggerates the amount of evidence.

Make the task set versioned and immutable during a comparison. Record the repository snapshot or document corpus, model identifier, prompt hash, tool schema, inference settings, maximum actions, wall-clock timeout and grader version. Record unavailable dependencies and API errors as explicit outcomes. Quietly deleting inconvenient failures changes the question being answered.

Use three task collections with distinct roles. A development collection helps engineers investigate errors. A validation collection supports regular tuning and model selection. A held-out test collection supports a final estimate and should not become a daily feedback channel. When the team has repeatedly inspected the held-out answers, it has effectively converted that set into development data. Refreshing a test set is not a failure; pretending it remains untouched is.

A contract also specifies the aggregation rule. Suppose the task mix is 80% simple retrieval and 20% difficult account changes. Averaging those tasks equally may answer a different question from weighting them by production traffic. Conversely, production-weighting alone may hide a rare but unacceptable unauthorized transfer. Report an overall estimate and decision-critical slices, then apply hard constraints separately where appropriate.

Finally, define inconclusive outcomes. “Insufficient evidence to release” is a legitimate result. A good process does not force every experiment into a winner and a loser. It preserves uncertainty until the expected value of further testing is lower than the cost of waiting.

04 · Uncertainty, paired comparisons and repeated trials

A score is an estimate, not a physical constant. If candidate A succeeds on 80 of 100 tasks, the observed success rate is 80%, but a different sample could produce another value. A confidence interval communicates sampling uncertainty under a specified model. It does not account automatically for mislabeled tasks, a biased sample, contaminated training data or a broken grader.

Compare two systems on the same tasks whenever possible. For each task, calculate the difference between candidate and baseline. Paired comparisons remove some variation due to task difficulty: a hard task is hard for both systems. Bootstrap whole tasks or higher-level clusters such as users or repositories when those are the independent units. Do not independently resample individual tool calls from the same trajectory.

Repeated trials answer a second question: how much does behavior vary when the task is held fixed? Keep the task identifier and trial identifier separate. A system that solves each of ten tasks in one lucky run out of ten has different production behavior from one that reliably solves nine tasks and always fails the tenth. Their pooled success rates can conceal these patterns.

Pass@k describes the chance that at least one of k attempts succeeds under the evaluation's sampling and selection assumptions. It is relevant when a user can actually generate and identify a good candidate. It does not mean the first answer is reliable. A requirement that all k repeated attempts succeed is a different reliability criterion, sometimes written pass^k. Under independent identical success probability p, these teaching models become 1−(1−p)^k and p^k respectively; independence is often unrealistic in deployed systems.

For rare failures, zero observations is not proof of zero risk. Under independent Bernoulli trials, the rough “rule of three” gives a 95% upper bound near 3/n after zero failures in n trials. With 300 clean trials, a failure rate around 1% is still compatible with this approximation. Correlated tasks or an unrepresentative test population weaken that reassurance further.

At least one success is not dependable repetition

With p = 0.8 and k = 3 independent attempts, these values are 0.992 and 0.512. They answer different product questions. For sampled code completions, a common finite-sample pass@k estimator with n generated candidates and c passing candidates is one minus the ratio of combinations shown below. It estimates oracle availability; an imperfect selector can perform worse.

When fewer than k failures exist, the numerator is zero. For n = 10, c = 2 and k = 3, the estimate is 1 − 56/120 = 0.5333. Keep failed, truncated and unparseable attempts in the declared denominator. This estimator does not remove task selection bias.

Source for the sampled-completion estimator: Evaluating Large Language Models Trained on Codehttps://arxiv.org/abs/2107.03374v2, first submitted July 7, 2021; v2 July 14, 2021. Versioned PDFhttps://arxiv.org/pdf/2107.03374v2. The numerical examples here are original and do not reproduce its model scores.

05 · Build graders, then evaluate the graders

Choose the strongest feasible evidence for each requirement. If a generated file must parse, execute a parser. If a balance must remain unchanged, inspect the balance. If an answer must cite an accessible source, verify the source and its relationship to the claim. A language-model judge is useful for semantic judgments that cannot be captured cheaply by deterministic checks, but it should not replace a stronger available verifier.

A rubric should define observable differences between scores. “Excellent” and “poor” are adjectives, not measurement rules. For an evidence-grounded answer, a rubric might distinguish unsupported central claims, partially supported claims with missing qualifications, and a complete answer whose important claims are supported by the supplied documents. Include examples near the decision boundary, not only obvious successes and failures.

Calibrate the judge on an independently annotated set. Measure agreement and inspect the disagreement matrix, especially false passes on consequential errors. An overall agreement score can look strong when almost all examples pass. Include adversarial examples, long fluent wrong answers, concise correct answers, irrelevant citations and attempts to instruct the judge from inside the candidate output.

The foundational MT-Bench and LLM-as-a-judge paperhttps://arxiv.org/abs/2306.05685v4 studies position, verbosity and self-enhancement biases. Its reported human agreement belongs to its tested models and tasks, not every future judge. The paper first appeared on 9 June 2023; the linked v4 is dated 24 December 2023. Read the PDFhttps://arxiv.org/pdf/2306.05685v4.

In your own protocol, randomize answer order, reverse a subset of pairs, blind provider names, constrain the rubric, and audit failures manually. Preserve judge versions and outputs so a score can be reproduced. Treat a judge upgrade as a measurement-system change: regrade a bridge set before interpreting a historical trend as model improvement.

A passing schema can still contain a false answer

The complete Python reference below includes a contract grader for an answer, its source identifiers and an answer/abstain status. It rejects malformed JSON and references outside the allowed source set. A deliberate counterexample changes “30 days” to “300 days” while preserving the schema and source ID: the contract grader still passes. This test is intentional. Structural validity, citation existence and semantic support are different properties; a real application needs a separately calibrated check against source content.

Keep grader failure distinct from candidate failure. An unavailable judge should produce an unknown evaluation state and a retry/review policy, not a fabricated zero or perfect score. Never execute a candidate response as Python merely to parse its structure.

06 · Arena: preference measurement and its boundaries

Arena, formerly LMArena and Chatbot Arena, made pairwise human preference a prominent part of model evaluation. In a blind comparison, a person sees responses to a prompt and expresses a preference before model identities are revealed. The statistical problem is to infer relative strengths from a sparse network of comparisons, not simply to count total wins across unequal opponents.

The Bradley–Terry model represents the probability that model A is preferred to B as a logistic function of their strength difference. The strengths are relative: adding the same constant to every strength leaves pairwise probabilities unchanged. A model with a higher estimated strength is more likely to win under the observed comparison process. This is not an absolute percentage of “intelligence.”

The original Chatbot Arena paperhttps://proceedings.mlr.press/v235/chiang24b.html is an ICML 2024 publication, with its official proceedings PDFhttps://raw.githubusercontent.com/mlresearch/v235/main/assets/chiang24b/chiang24b.pdf. The research establishes a concrete pairwise evaluation framework; it does not establish that public preference is sufficient for every deployment.

Read rankings with their uncertainty, category, date and methodology. Differences can be too small to distinguish confidently. The population of people submitting prompts is not necessarily your customer population. Fluent presentation can affect preference. Different categories can reverse a ranking. Connected comparison coverage matters: a model tested against a narrow or atypical opponent set requires careful estimation.

Arena's 14 July 2026 factuality announcementhttps://arena.ai/blog/factuality-in-arena describes a separate, initially optional Text/Search view combining preference with claim-level factuality audits. Its stated default factuality weight is 25%. This is a useful illustration of an evolving construct: changing the mixture changes what the ranking means. Do not splice the adjusted score into an older preference-only time series without explaining the break.

Use Arena as external evidence and a source of candidate models. Then run a matched evaluation on your own tasks. The engineering question is not “Who is number one?” but “Does this candidate improve our outcome under our constraints?”

Work through the pairwise model

The strengths theta are dimensionless log-odds parameters in this unscaled formulation. A difference of ln(3) gives a 75% predicted preference probability, not a 75% task accuracy. Fix one strength or constrain their sum to remove the arbitrary shared offset. Estimate strengths from the comparison graph, and inspect task and voter dependence before trusting naive independent-vote uncertainty. Ties require an explicit modeling policy rather than silently counting every tie as a decisive win.

07 · Current frontier: measuring validity, contamination and alignment

The research frontier increasingly concerns whether measurements justify their claims. Measuring what Mattershttps://arxiv.org/abs/2511.04703v1, submitted 3 November 2025 and formally published in the NeurIPS 2025 Datasets and Benchmarks trackhttps://proceedings.neurips.cc/paper_files/paper/2025/hash/1967e0fc3aa6cbbace562f5cb8e3954e-Abstract-Datasets_and_Benchmarks_Track.html, reviews 445 benchmarks with 29 expert reviewers. Its central concern is construct validity: whether the task and metric represent the phenomenon claimed. PDFhttps://arxiv.org/pdf/2511.04703v1. A practical implication is to draw the chain from real-world requirement to task, observation and score, then challenge each link.

Contamination is another threat. On 23 February 2026, OpenAI reported problems with SWE-bench Verifiedhttps://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/, including flawed tests and contamination. Its 59.4% flawed-test finding applies to an audited difficult subset of 138 problems, not automatically the entire benchmark. This is a provider-authored audit, and its scope matters. The lesson is to inspect failed cases and grading assumptions, not to assume every low score is a model defect.

Arena's 8 October 2026 Alignment Indexhttps://arena.ai/blog/ai-alignment-index describes three observable signals: unauthorized action, false attribution and deceptive completion. It uses judged evidence and adjusts for conversation length. This is an operational measurement of selected behaviors, not a proof of complete alignment. A narrow, clearly defined indicator can be useful precisely because its limits are visible.

These developments point toward an important discipline: evaluate the evaluation. Test whether a benchmark responds to genuine improvements, whether superficial changes can raise its score, whether labels are reproducible, and whether the score predicts a meaningful deployment outcome. A benchmark that is difficult but unrelated to the product is not automatically valuable.

08 · RAG evaluation: separate retrieval from answering

A retrieval-augmented system can fail before the language model sees the evidence. If the relevant policy paragraph is absent from the retrieved context, improving answer wording will not solve the underlying problem. Conversely, perfect retrieval does not guarantee that the model uses the evidence correctly. Evaluate the pipeline at its interfaces.

For retrieval, define relevance at the unit you actually retrieve: document, chunk or passage. Measure whether the required evidence appears within the allowed context budget. Recall@k is meaningful only with a suitable relevance set and a specified k. A large k can improve recall while increasing latency, cost and distracting context. Test reranking and chunking under the same token budget rather than comparing an inexpensive short context against a much larger one.

For generation, separate answer correctness from support. An answer can be factually correct but unsupported by the retrieved material; it can quote a source accurately while answering the wrong question. Include citation validity, entailment of important claims, completeness, abstention when evidence is insufficient, and handling of conflicting or outdated documents.

Use counterfactual tests. Remove the decisive passage and check whether the system acknowledges uncertainty. Replace a policy with a newer version and inspect whether the answer changes appropriately. Insert an irrelevant but persuasive paragraph and test resistance to distraction. Add a malicious instruction inside a retrieved document and verify that the system treats it as data rather than authority.

A fair ablation changes one component at a time: retrieval method, chunk size, reranker or generator. Keep the corpus snapshot, questions, answer rubric and budget fixed. The result should explain a mechanism—such as improved evidence recall on long policies—rather than merely report a higher aggregate number.

09 · Agent evaluation: outcomes, permissions and recovery

For an agent, evaluate both the final state and the path constraints. A travel assistant that books the right flight using an unauthorized payment method has not succeeded. A coding agent that fixes a test by deleting the test has not solved the intended problem. The verifier needs to inspect meaningful invariants, not only whether the final response sounds successful.

Construct a sandbox with explicit initial state, permitted actions and expected postconditions. Record tool inputs and outputs, environment changes, retries, timeouts and side effects. A task should define whether asking for clarification is acceptable, when human approval is required, and what should happen when a tool fails. Otherwise, the grader may punish sensible behavior or reward recklessness.

Anthropic's 9 January 2026 agent-evaluation guidehttps://www.anthropic.com/research/demystifying-evals-for-ai-agents emphasizes tasks, trials, graders, transcripts and outcomes. The engineering takeaway is to inspect trajectories when scores behave unexpectedly and to combine grader types. The guide is practitioner evidence, not a universal benchmark result.

Security evaluation needs an attacker model. Specify what the attacker controls: a retrieved page, email body, repository file or tool response. State the protected boundary and observable violation. Measure attack success and benign-task completion together; a system that refuses every request may look secure while being useless. Keep attack development separate from the final test set.

Recovery deserves its own tasks. Inject a transient timeout, stale credential, contradictory tool response or partial write, then verify whether the agent retries safely, rolls back, asks for help or stops. Do not assume that a successful rerun demonstrates a reliable repair policy. Compare against a retry-only baseline and account for extra compute.

For recursive self-improvement, freeze the evaluator outside the optimization loop. If the same system proposes changes, rewrites the tests and judges its own success, a rising score may reflect adaptation to the measurement process. Independent holdouts, resource budgets and explicit stopping rules are essential.

10 · Multimodal and physical AI need different evidence

Multimodal evaluation is not text evaluation with an image attached. A visual system may need to locate a small object, read a chart, distinguish a temporal event or combine speech with a scene. A text-only final answer can hide whether the model used the relevant modality or guessed from linguistic priors.

Design modality-specific counterfactuals. Change the number in a chart while leaving the question unchanged. Shift an event to a different video timestamp. Replace spoken words while preserving background noise. Mask the decisive region of an image. A system that gives the same answer after the decisive evidence changes may be relying on the wrong signal.

Report input resolution, frame sampling, audio duration, preprocessing and latency budgets. A model receiving every video frame is not being compared fairly with one receiving a few selected frames unless that resource difference is part of the question. Human ratings of aesthetics or naturalness should be separated from task correctness and accessibility.

Physical AI adds dynamics and safety. A robot policy evaluated only in simulation may exploit simulator artifacts or fail under sensor noise, wear, contact variation and unexpected objects. Report the distribution of environments, reset procedure, intervention rate, task completion, collision or constraint violations, and recovery behavior. A successful demonstration is not a reliability estimate.

The broader principle is to preserve the causal connection between the evidence and the claimed ability. If the claim is spatial reasoning, eliminate shortcuts that can solve the task without spatial evidence. If the claim is robust control, vary the physical conditions rather than replaying one favorable trajectory.

11 · An engineering architecture for continuous evaluation

A production evaluation system has four separable layers: a versioned task store, an execution harness, grading services and a decision/reporting layer. Keeping these boundaries explicit makes it possible to rerun old outputs through a new grader, compare a new model under an unchanged harness, and diagnose whether a regression came from the product or the measurement system.

The task store contains inputs, references, metadata, permissions and provenance. The harness creates isolated environments, enforces budgets and captures artifacts. Graders produce dimension-specific results with evidence. The reporting layer aggregates by task slice, uncertainty and cost, then applies release rules. Every result should be traceable to the exact versions that produced it.

Run inexpensive deterministic checks on every change. Run a representative regression suite in continuous integration. Schedule larger repeated-trial and adversarial evaluations when a model, prompt, retrieval index or tool changes materially. Expensive human review should focus on ambiguous or consequential cases rather than being allocated uniformly without purpose.

Connect evaluation to observability without confusing them. Observability records what happened in production: traces, errors, latency, usage and feedback. Evaluation applies a criterion to selected behavior. A trace is evidence for debugging; it is not automatically a quality label. A useful feedback loop turns appropriately redacted production failures into reviewed regression tasks, while keeping a separate holdout for unbiased assessment.

Protect the measurement pipeline itself. Candidate outputs are untrusted data and can contain instructions aimed at the judge. Sandbox executable submissions, restrict network access, prevent test leakage and log grading exceptions. A failed grader should be reported as unknown or infrastructure failure, not silently converted into a model pass or fail.

業界構造:制御点・購買者・競争

業界資料の確認日:2026年11月8日。製品文書は機能の提供を示すもので、独立した効果測定ではありません。競争上の見解と未解決機会は Token Lab の分析です。合成例は企業の実測値ではありません。

評価産業:証拠を売る企業と判断を握る企業

確認日は 2026 年 11 月 8 日です。AI 評価は単一のランキング事業ではありません。研究所は能力へのフィードバック、アプリ開発者は回帰の仕組み、企業リスク部門は監査可能な受入証拠、基盤購入者は費用と性能の比較を求めます。一つに強くても他を代替できません。問うべきは、証拠がどの条件で何の判断を支え、誤ったとき誰が責任を負うかです。公開製品・取引と商業的動機の分析を区別します。資金調達や広告は評価精度の独立した証拠ではありません。

評価産業:証拠を売る企業と判断を握る企業
責任分担の概念図です。企業評価や実測カバー率ではありません。

四つの判断と四種類の証拠製品

公開ベンチマークは候補選定、非公開業務データは変更のリリース判断、安全評価は脅威モデルの検証、本番結果の研究は業務改善の判断を支えます。相互補完的で、文章の好みで勝つ請求書エージェントでも無許可の仕入先情報を使い得ます。当サイトは AI 評価器の有無でなく、改善する意思決定によって企業を分類します。調達では母集団、課題、受入基準、失敗費用、更新頻度を先に定め、その後で人、コード、モデル評価器、管理サービスを公平に比較すべきです。

参照構成:独立したリリース判断

ランキング収集ではなくリリース制御の構成です。登録庫で基準版と候補版を固定します。固定事例と等しい権限で隔離実行し、失敗や費用も含む生の観測を保存します。版付き判定器と人の裁定を別の証拠として集計します。事前の品質・リスク基準で限定カナリアへの進行を決めます。最終留保集合は候補最適化から隔離し、評価手順に従ってのみ使います。

開発と最終試験のフィードバックを分けます。留保集合で繰り返し選ぶと開発データになります。候補は課題、判定コード、点数、昇格規則を変更できません。構成と生結果を保存し、改善をモデル・検索・プロンプト・ツールの変更に帰属させます。カナリアは汚染した比較を修復しません。復旧権限は評価対象ではなくリリースシステムに置きます。

参照構成:独立したリリース判断
責任分担の概念図です。企業評価や実測カバー率ではありません。

企業と製品の位置付け

選択的な業界地図であり、市場シェア順位や認証ではありません。対象範囲は版、統合、配置、強制モードによって異なります。

Arena / LMArena

所有者・組織
Arena Intelligence Inc.
制御点
人の好みと外部評価

Arena は人のフィードバックに基づく外部評価を販売し、顧客内部の回帰集運営とは異なります。2025 年 9 月 16 日開始の商用サービスは分析、代表サンプル、納期を提供します。コミュニティが特徴的な入力で、販売と信頼できる中立性の両立が課題です。好みで勝っても権限や業務の正しさは保証しません。現在の Agent Mode は従来の一対比較ランキングと異なるため、製品と方法を指定すべきです。

Arena commercial evaluation launch ↗https://arena.ai/blog/ai-evaluations

Artificial Analysis

所有者・組織
独立ベンチマーク企業
制御点
モデル・基盤の選定

Artificial Analysis はモデル、エージェント、推論事業者、ハードウェアを測定し、公開情報と無料・商用 API を提供します。購入者はリリース判定より選定・調達情報を求めると考えられます。品質、性能、経済性の比較が価値提案です。総合指数は構成された指標なので版、設定、日付を残してください。現在の Intelligence Index は主に英語テキストで、他言語・モダリティには別の証拠が必要です。候補選定後に実業務で試すべきです。

Artificial Analysis index methodology ↗https://artificialanalysis.ai/methodology/intelligence-benchmarking

Scale AI / SEAL / Scale Labs

所有者・組織
独立企業、Meta が議決権のない少数持分
制御点
専門家データと研究評価サービス

Scale は専門家データ運用、評価サービス、SEAL と Scale Labs の研究を組み合わせます。自動ラベルでは足りない難題と専門判断を大規模に組織する点が強みになり得ます。Meta の少数持分は事実ですが、全評価の偏りや会社全体の買収の証拠ではありません。顧客はデータアクセス、契約上の分離、公表手順を調べるべきです。専門家は妥当性を高め得ますが、提供費用と判断の一貫性は事業上の制約です。

Scale investment and independence statement ↗https://scale.com/blog/scale-ai-announces-next-phase-of-company-evolution

Galileo / Cisco

所有者・組織
2026 年 5 月 22 日に買収完了
制御点
評価から実行時ガードレール

Galileo は小型 Luna 評価器を含め、評価、監視、実行時ガードレールを接続します。Cisco は 2026 年 5 月 22 日に買収を完了しており、独立企業と記すのは誤りです。離線の品質基準を低費用の本番制御と企業販売へ変える機会があります。費用・遅延の宣伝値は同条件で検証しない限り主張です。高速な採点にも校正と不確実時の対応が必要で、資源認可、厳密な承認、確定済み操作の復旧を置き換えません。

Cisco Galileo acquisition completion ↗https://www.cisco.com/site/us/en/about/corporate-development/acquisitions/galileo/index.html

Databricks / MLflow

所有者・組織
Databricks 基盤と OSS MLflow
制御点
データ統制下の評価

MLflow は追跡、評価、評価器、プロンプト、人のフィードバックをモデル開発へ接続します。Databricks は管理型運用と企業データ統制との近さを加えます。当サイトはデータ集積と既存調達を優位、OSS API を可搬性の支えと見ます。歴代版が同等と仮定せず、版とエディションを確認してください。統制されたデータやモデル登録簿は有用でも、特定要求での外部支払・削除の認可を決定しません。

Databricks current MLflow observability and portability FAQ ↗https://docs.databricks.com/aws/en/mlflow3/genai/tracing/faq

Microsoft Foundry

所有者・組織
Microsoft クラウド基盤
制御点
トレースのデータ化とリリース連携

Microsoft Foundry はトレース、評価、監視、CI/CD 品質ゲートを企業開発環境へまとめます。文書にはトレースから整理・版管理した評価データを作る方法があります。Microsoft 中心の組織で接続・調達の摩擦を減らす点が利点です。専門企業は追加基盤の負担を上回る判断改善を示す必要があります。データ化にも代表的選定と機密制御が必要で、自動収集は正しいラベルや最小権限を証明しません。

Microsoft Foundry traces to evaluation datasets ↗https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/traces-to-dataset

AWS Bedrock / Google Cloud

所有者・組織
別個のクラウド製品
制御点
管理型モデル・アプリ評価

AWS Bedrock はモデル・RAG 評価、人の評価、モデル評価器のジョブを提供します。Google Cloud の文書は第三者を含むモデルとアプリの評価を扱います。別製品であり、クラウド販売による競争圧力が似るためだけにまとめています。既存基盤・契約は採用を容易にしますが、中立性と可搬性は確認が必要です。管理型実行でも正しい母集団や評価器の妥当性は自動で決まりません。結果には構成、予算、指標版を残すべきです。

Amazon Bedrock Evaluations ↗https://aws.amazon.com/bedrock/evaluations/ · Google Cloud Gen AI evaluation overview ↗https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview

ラベルと再現可能な環境の経済性

評価費用は評価モデルのトークン料金だけではありません。課題設計、専門家判断、環境初期化、推論、反復、製品変更に伴う保守を含みます。価値は数字の量ではなく、説明可能な判断の費用を減らすことです。再利用できる専門家の作業、入手困難な代表的課題、汚染対策、再現可能な環境は一般的な評価器より強い優位になり得ます。ただし顧客ラベルを法的・運用的に他テナントへ再利用できるとは限りません。データのネットワーク効果には同意、隔離、移転性、機密課題を漏らさない汎化の説明が必要です。

技術の要点:品質を判断する三つの契約

ベンチマークは定義した課題の解決、選好アリーナは提示・抽出条件下での好み、実運用評価は品質・安全・遅延・費用制約内での業務完了を測ります。好まれる回答が根拠を欠く場合も、正答が遅すぎる場合も、能力あるモデルが検索やツール統合で失敗する場合もあります。調達判断で統合する前に、それぞれが示す証拠を分けます。

公平な比較では同じ事例を両候補に与え、対応する差を調べます。選定前にデータ、採点基準、判定器の版、ツール権限を固定し、最終留保集合を最適化から外します。LLM 判定器は裁定済み事例で較正し、位置・文体の偏りや候補との共通誤りを調べます。不確実性と課題群別の結果を示します。高損失の失敗が悪化すれば総合点の上昇だけでは改善を示せません。価値は再現可能な意思決定過程にあります。

技術の要点:品質を判断する三つの契約
責任分担の概念図です。企業評価や実測カバー率ではありません。

点数から安全な判断までの未解決部分

評価器は一貫していても誤り、複数が同じ盲点を持つこともあります。公開順位の言語、課題、文体の好みは購入者の業務と異なり得ます。エージェントはハーネス、ツール、外部状態、予算、初期化にも依存し、モデル名だけでは比較単位として不十分です。本番受入は業務結果に、テナント越境禁止、無許可の副作用禁止、隠れた権限拡大禁止を合わせる必要があります。次の価値は、不確実性と責任者を明確にした証拠統合にあると考えます。専門家、再現可能な実行、業務結果を結び、自動代理指標を正解と偽らないことが必要です。

買収・資本・戦略的影響

資本と所有:独立性を正確に論じる

Arena は 2026 年 1 月 6 日に 1 億 5,000 万ドルのシリーズ A を発表しました。Scale は 2025 年 6 月 12 日に Meta の投資を発表し、Meta は後に議決権のない少数持分と、非市場性株式投資に含まれる 137.9 億ドルを開示しました。Scale 全体の買収でも評価事業単独の価格でもありません。Cisco は 4 月 9 日の意向発表後、2026 年 5 月 22 日に Galileo 買収を完了しました。これらは資本の関心であり品質の証明ではありません。所有や商取引について顧客データへのアクセス、公表方針、モデル選定、利益相反管理を調べるべきですが、関係だけから偏りを断定できません。

製品・取引の一次資料

14 · How AI giants compete through evaluation

Frontier-model competition is partly a competition over what counts as progress. A provider can emphasize mathematical reasoning, coding, agentic task completion, multimodal understanding, latency, price or safety. Those dimensions matter, but selecting the favorable subset can create a misleading overall impression. Read a model launch as a collection of scoped claims rather than a universal ranking.

For every headline comparison, ask whether model versions, prompts, tools, inference budgets and grader settings match. A result with extensive search, retries and a large compute budget does not measure the same product experience as a single low-latency response. If proprietary scaffolding contributes to the score, describe the result as a system result unless the contribution has been isolated.

Providers have real advantages in evaluating their own systems: detailed access to versions, logs and failure modes. They also have incentives to present progress favorably. Independent evaluators can diversify the evidence, but independence is not guaranteed merely by being a separate company. Examine funding, commercial relationships, publication policies, access privileges and the ability to report negative findings.

Enterprise buyers should build a small portfolio of evidence: public capability measurements for breadth, domain-specific held-out tasks for fit, adversarial tests for boundaries, and randomized online experiments for user impact. Require a candidate to improve the dimensions that matter to the business rather than chasing every public leaderboard movement.

The durable competitive advantage may lie in a high-quality feedback loop: representative failures become reviewed tasks, interventions are tested fairly, successful changes are deployed cautiously, and production outcomes verify the hypothesis. This is an engineering interpretation of the market, not a claim that any named company has already solved that loop.

Cloud platforms compete for the evaluation workflow

Microsoft Foundryhttps://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-overview documents evaluation of existing datasets, generated model or agent outputs, and production traces. This is a workflow integration proposition: keeping execution, permissions and evidence near the deployment environment. Buyers should still export a versioned task set and verify that scores remain reproducible outside the portal.

Google Cloud's Gen AI evaluation servicehttps://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview documents evaluating third-party models as well as models in its own ecosystem. This highlights an important separation: the generator being tested need not be the judge. A cross-provider evaluation is not automatically independent, however; check the judge's calibration and version on the target domain.

Amazon Bedrockhttps://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html documents model-evaluation jobs that use a separate LLM to score responses and explain the score. A generated explanation is useful evidence for review, not proof that the score is correct. Keep an adjudicated calibration set, record evaluator configuration and include judge cost in the experiment ledger.

OpenAI and Anthropic also contribute provider-authored audits and engineering guidance, discussed earlier. Across these companies, the competitive axes are model quality, integration, operating cost, access to feedback and trust in measurement. The practical strategy is to keep your evaluation contract portable while using integrations where they reduce work. Do not give the vendor's default rubric authority to redefine your business outcome.

Continue the learning path

Read the companion handbook: evaluation and observability · AI security · AI infrastructure

Learning library · Interview and projects · Editorial standards and corrections

Educational extensions: OpenAI Academyhttps://academy.openai.com/ · Anthropic learning resourceshttps://academy.claude.com/ · Microsoft AI traininghttps://learn.microsoft.com/en-us/ai/.

Primary references are attached to the relevant claims. Conceptual diagrams, synthetic arithmetic and research proposals are original teaching material. Vendor descriptions establish product positioning, not independent proof of performance. Corrections should identify the claim, its source and the proposed replacement.

実践と検証

段階5:フィードバックと最後の実践検証

本番フィードバックは遅れ、偏ります。無反応は成功ではありません。修正に使用した障害例は独立試験ではなくなります。障害、案件、ルーブリック、実行、公開判断の系譜を残し、採点器変更を能力向上と混同しません。最終証拠には契約、配対結果、重要層、採点校正、費用、復旧条件を含めます。

最後の演習ではACLだけを変えて検索拒否を確認し、規約を削除して棄権を要求します。30日を300日に改変すると構造検査は通過し、根拠検査は失敗すべきです。同一例の10倍複製で独立標本数を増やしません。越権操作で公開を阻止し、採点器停止を不明と表示します。期待、実測、説明を残し、最後に修正に使わなかった新案件で転移を確認します。

15 · Open research problems worth a serious experiment

External validity. Can an offline score predict a deployment outcome across organizations? Construct several domain-specific task sets, preregister the predicted ordering, and compare it with a later controlled rollout. Report where the ordering reverses. A single positive correlation on one product is not a universal transfer law.

Adaptive contamination. How can a benchmark remain informative when developers repeatedly optimize against it? Compare fixed public tasks, private held-outs and periodically refreshed tasks under a controlled optimization process. Measure performance on a final unseen distribution. The aim is to separate genuine generalization from learning the measurement interface.

Judge robustness. Can a grader resist style changes and adversarial instructions without losing sensitivity to real improvements? Create matched output pairs that preserve correctness but change verbosity, formatting or persuasion. Then create pairs with identical style but different factual quality. The grader should be invariant to the former when irrelevant and responsive to the latter.

Long-horizon reliability. How do errors accumulate across tool interactions? Vary task length while controlling task type, and distinguish exposure from intrinsic deterioration. A longer task creates more opportunities to fail; a higher session-level failure rate does not alone show that per-action reliability worsened.

Evaluation under limited budgets. Which tasks should be labeled or rerun next? Compare uniform sampling with an adaptive policy, but preserve a probability sample for unbiased reporting. An adaptive strategy can find more failures while producing a distorted raw failure rate if selection probabilities are ignored.

Reward hacking in self-improving systems. Does an optimization loop improve the target capability or exploit its grader? Hold a second evaluator and task set outside the loop, impose equal compute budgets, and compare against a non-adaptive baseline. Publish negative results and evidence of evaluator exploitation. A rising training reward is a hypothesis about improvement, not its verification.

16 · A four-week capstone with a defensible release decision

Build a small document-support agent that answers policy questions and can propose—but not execute—an account change. The baseline uses one retrieval method and a fixed prompt. The candidate changes exactly one component. Use synthetic documents and mock tools so no personal data or real transactions are involved.

In week one, define 60–100 tasks across straightforward retrieval, conflicting evidence, missing evidence and unauthorized requests. Write a rubric and deterministic postcondition checks. Reserve an untouched test partition. Annotate a small calibration set with two reviewers and resolve disagreements by revising the rubric rather than averaging incompatible interpretations.

In week two, implement the harness with task IDs, seeds where applicable, budget limits, artifacts and explicit infrastructure-error states. Run baseline and candidate on the same tasks. Repeat a subset to estimate execution variance. Add a local report showing paired outcomes, critical slices, latency and cost per accepted result.

In week three, audit failures. Test whether the judge can be manipulated by candidate text. Perform an ablation that restores the baseline component, and a budget-matched comparison that prevents extra compute from masquerading as algorithmic improvement. Add discovered failures to the development set while keeping the final test untouched.

In week four, make the release decision under a preregistered rule. A strong report includes the task distribution, exact versions, uncertainty, failed examples, resource costs, safety constraints and reasons to defer deployment. The reference success criterion is not “candidate wins.” It is “another engineer can reproduce and critically evaluate the conclusion.”

For a research extension, measure whether the offline decision predicts a simulated deployment distribution with different proportions of difficult tasks. Explain the discrepancy when distribution shift changes the conclusion.

17 · Original interview practice and scoring

These are original practice questions, not reported questions from any company. Use the same 1–5 progression for each: 1 identifies the relevant concept; 2 gives a correct mechanism; 3 applies it to the concrete example; 4 proposes a sound experiment with controls; 5 explains assumptions, failure cases and the limits of the conclusion.

1. Candidate B wins 55% of public preference comparisons. Should you replace the support model?

Hint 1: Identify the population and outcome being measured.

Hint 2: Distinguish preference from policy correctness and authorization.

Reveal answer and analysis

Answer: Not on that evidence alone. Run a paired comparison on representative support tasks with correctness, policy and safety graders, then examine uncertainty and cost. A strong answer also checks the public comparison's confidence interval, category and versions, and proposes a guarded online experiment after offline gates pass.

2. Why can 90% judge agreement still be unacceptable?

Hint 1: Consider an imbalanced dataset.

Hint 2: Ask which disagreement type is costly.

Reveal answer and analysis

Answer: If almost all examples are acceptable, a judge that passes everything can achieve high agreement while missing every dangerous failure. Inspect class-conditional errors, calibrate on consequential cases and audit false passes. A top answer explains why agreement with one imperfect annotator is not ground truth and defines an adjudication process.

3. An agent succeeds at least once in five runs but often fails the first run. Which metric matters?

Hint 1: What attempts can the product afford?

Hint 2: Can it identify the successful attempt reliably?

Reveal answer and analysis

Answer: Pass@5 is relevant only if five attempts and reliable selection are available. For a single-shot workflow, first-attempt success and failure severity matter. For repeated dependable execution, measure all-attempt reliability and task-level variance. Include latency, cost, correlated failures and side effects from unsuccessful attempts.

4. A retrieval change raises answer quality. How do you establish why?

Hint 1: Separate evidence availability from generation.

Hint 2: Match the context and compute budgets.

Reveal answer and analysis

Answer: Measure evidence recall, answer support and correctness on the same tasks, keeping the generator fixed. Compare retrieval-only ablations under equal context budgets. Inspect whether gains concentrate on previously missing evidence and test distraction and stale-document cases. A high-scoring answer avoids claiming causality from aggregate correlation alone.

5. A self-improving agent raises its own benchmark score every day. What would convince you it improved?

Hint 1: Identify what the agent can modify or observe.

Hint 2: Keep independent evidence outside the loop.

Reveal answer and analysis

Answer: Freeze an independent evaluator and held-out tasks, compare under equal resource budgets, and check whether gains transfer to new tasks and robust behavior. Audit grader exploitation and contamination. The strongest answer defines a preregistered stopping rule, preserves failed runs and distinguishes increased capability from increased search effort.

Reproduce the arithmetic: a complete local reference

Imagine evaluating a replacement support agent on twenty matched tasks. Twelve tasks pass under both versions, four pass only under B, two pass only under A, and two fail under both. Model A scores 14/20 and B scores 16/20. We resample task pairs, keeping the relationship between outcomes intact. The resulting interval is wide: this demonstration does not establish that B is better in the population. The individual Wilson interval is a separate description of B's success rate; overlap between two marginal intervals is not the paired comparison test.

Here n is the number of independent tasks and outcomes are binary. The second expression is the exact one-sided upper confidence bound on failure probability after zero observed failures, assuming independent identically distributed Bernoulli trials; alpha = 0.05 gives 95% confidence. Correlated tasks or an unrepresentative sample invalidate that interpretation.

Revisit the opening score-and-uncertainty diagram. Its error bars are marginal Wilson intervals; the paired interval below addresses the difference directly.

The bootstrap is a percentile approximation, not a guarantee of calibrated coverage with just twenty examples. If multiple prompts belong to one customer or document, resample customers or documents. More repeated generations per item measure stochastic variability; they do not manufacture more independent customers.

Try the denominator locally

No input leaves your browser. This is an arithmetic illustration, not a production risk estimate.

Runtime: Python 3.10 or newer; standard library only. The evaluation bootstrap uses seed 17 and 10,000 resamples. Observability arithmetic is deterministic and needs no seed. These programs make no network or model calls.

Download the complete Python reference

python3 evaluation-audit.py
"""Synthetic paired evaluation; Python 3.10+, standard library, seed 17.
No API calls. Items, rather than individual judgments, are bootstrap units.
"""
import math
import random
import json
from statistics import mean


def wilson(successes, n, z=1.959963984540054):
    if not 0 <= successes <= n or n <= 0:
        raise ValueError('Require 0 <= successes <= n and n > 0')
    p = successes / n
    den = 1 + z*z/n
    center = (p + z*z/(2*n)) / den
    radius = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / den
    return center-radius, center+radius


def paired_bootstrap(a, b, repetitions=10000, seed=17):
    if len(a) != len(b) or not a:
        raise ValueError('Require nonempty matched items')
    if not isinstance(repetitions, int) or repetitions < 100:
        raise ValueError('Use at least 100 bootstrap repetitions')
    if any(v not in (0, 1) for v in a + b):
        raise ValueError('This example requires binary outcomes')
    rng = random.Random(seed)
    differences = [y-x for x, y in zip(a, b)]
    means = sorted(mean(rng.choices(differences, k=len(a)))
                   for _ in range(repetitions))
    return mean(differences), means[int(.025*repetitions)], means[int(.975*repetitions)]



def grade_response(raw, allowed_sources, expected_status):
    """Original deterministic contract grader, NOT a semantic truth judge.
    Candidate text is parsed as data; never eval(), execute(), or prompt a judge.
    """
    try:
        response = json.loads(raw)
    except (json.JSONDecodeError, TypeError):
        return {"schema": False, "evidence_ids": False, "status": False}
    schema = (isinstance(response, dict)
              and set(response) == {"answer", "sources", "status"}
              and isinstance(response["answer"], str)
              and isinstance(response["sources"], list)
              and all(isinstance(x, str) for x in response["sources"])
              and response["status"] in ("answer", "abstain"))
    if not schema:
        return {"schema": False, "evidence_ids": False, "status": False}
    return {"schema": True,
            "evidence_ids": (all(x in allowed_sources for x in response["sources"])
                             and (response["status"] != "answer" or bool(response["sources"]))),
            "status": response["status"] == expected_status}


valid = '{"answer":"The policy says 30 days.","sources":["policy-v3"],"status":"answer"}'
assert all(grade_response(valid, {"policy-v3"}, "answer").values())
assert not grade_response('not JSON', set(), 'abstain')['schema']
assert not grade_response(valid, {"policy-v4"}, "answer")['evidence_ids']
assert not grade_response(valid, {"policy-v3"}, "abstain")['status']
# A plausible-looking lie can pass these structural checks. Add human/semantic
# review against the actual source; do not call this contract pass "correctness".
lie = valid.replace('30 days', '300 days')
assert all(grade_response(lie, {"policy-v3"}, "answer").values())

# Same 20 synthetic tasks: 12 both pass, 4 only B, 2 only A, 2 neither.
a = [1]*12 + [0]*4 + [1]*2 + [0]*2
b = [1]*12 + [1]*4 + [0]*2 + [0]*2
assert sum(a) == 14 and sum(b) == 16
low, high = wilson(sum(b), len(b))
delta, lower, upper = paired_bootstrap(a, b)
assert lower <= 0 <= upper  # small sample does not establish superiority
assert paired_bootstrap(a, a) == (0, 0, 0)
assert wilson(0, 20)[0] >= -1e-12
assert math.isclose(1 - .05**(1/100), .029513049607039932)
# Trial independence is an assumption, not a property guaranteed by a model.
assert math.isclose(1-(1-.8)**3, .992)
assert math.isclose(.8**3, .512)
print(f'A: {mean(a):.1%}; B: {mean(b):.1%}')
print(f'B Wilson 95% interval: [{low:.3f}, {high:.3f}]')
print(f'Paired delta: {delta:.1%}; bootstrap 95% interval: [{lower:.3f}, {upper:.3f}]')
print('Zero failures in 100: one-sided 95% upper bound 2.95%')
print('All evaluation checks passed; synthetic data, not model performance.')

Expected output:

A: 70.0%; B: 80.0%
B Wilson 95% interval: [0.584, 0.919]
Paired delta: 10.0%; bootstrap 95% interval: [-0.150, 0.350]
Zero failures in 100: one-sided 95% upper bound 2.95%
All evaluation checks passed; synthetic data, not model performance.

Test the failure cases: empty samples are rejected; the evaluation comparison of a model with itself has zero difference; the trace algorithm rejects cycles. Next add clustered tasks or missing spans, and document exactly which assumptions stop holding. Unit checks validate this implementation, not a deployed model.

前提知識を理解する

手法を支える共通の基礎を復習し、算例と確認問題で理解の不足を見つけます。

実験は明示した条件で特定の量を推定します。要求、問題、シード、ロボットのエピソードは違う比較単位です。対応比較は同じ例を使います。アブレーションでは機構を一つ変え、データ、予算、評価を固定します。基準側の資源まで減らすと交絡します。

解答を見る前に考える

同じ三例で ,1,1]、,0,1]。平均は何を示し、何を隠しますか。

段階的な解説

差は [1,−1,0]、平均ゼロです。一件改善、一件悪化しています。同じ平均は同等性の証明ではなく、三例では信頼できる一般化も困難です。群と不確実性を調べ、相関した反復を独立証拠と数えません。

実践とのつながり

主張を課題、分割、版、予算、指標、失敗に結びます。論文の結果と教材例を分け、算術の確認を訓練再現と呼びません。ベンチマークへの適応は評価を汚染し得ます。

理解を示すには

反証可能な主張を書き、調整前に評価を固定して設定と生の結果を保存します。同予算の適切な基準と比較し、自己改善では外部留保課題で退行も報告します。

完全な Python 例を実行

Python 3.10+・標準ライブラリのみ・決定的計算で乱数シード不要・通信なし。保存して下のコマンドを実行します。教育用の確認で、モデルの性能評価ではありません。

Python をダウンロード · evidence.py
python3 evidence.py
# Python 3.10+; standard library; deterministic; no network.
a, b = [0, 1, 1], [1, 0, 1]
assert len(a) == len(b)
differences = [new - old for old, new in zip(a, b)]
mean_change = sum(differences) / len(differences)
assert mean_change == 0
assert sum(x > 0 for x in differences) == 1
assert sum(x < 0 for x in differences) == 1
print("mean change=0; improved=1; regressed=1; unchanged=1")

期待する出力

mean change=0; improved=1; regressed=1; unchanged=1

前提を復習 →

基礎解説更新:

次の学習

ホールドアウトを固定し、対応する実行、不確実性、失敗と予算を比較する。

実装と検証 → · 学習ルートへ →

予測・検証・根拠の記録

解答を開く前に結果を予測し、演習後に差異と応用テストを記録します。端末内に保存し、言語版間で共有します。

記録を開く/書き出す/消去する →

数学的確認の範囲は授業ごとに記録されています。論文の実証再現の証明ではありません。疑問点はメモに記録してください。 確認範囲