Token LabBuscar

Translation status: this page is not fully translated. Read the English edition.

El recorrido está traducido. Las referencias, los rótulos de figuras y parte de la práctica siguen en inglés.

AI EVALUATION / RESEARCH & ENGINEERING HANDBOOK

AI evaluation: from measurements to decisions, research and market power

A field guide to traditional ML, LLM and agent evaluation: statistics, Arena, engineering, research frontiers, companies and acquisition evidence.

Jump to the runnable reference and tests

Fuentes y versiones
Contents · chapters, research and reference code
En este artículo
PARTE I / SISTEMA DE PRINCIPIO A FIN

Evaluación: decidir si el sistema puede publicarse

Este caso original sigue a un asistente de facturas. Lee documentos del inquilino y propone un pago; ejecutarlo requiere autorización separada y exacta. El escenario y los experimentos son didácticos y sintéticos, no un despliegue real.

Evaluación: decidir si el sistema puede publicarse: conceptual lifecycle diagram
Arquitectura conceptual original. Las líneas representan dependencias o evidencia, no rendimiento medido.

Primero la decisión: ¿puede desplegarse el agente de facturas?

El asistente sintético responde preguntas, consulta documentos del inquilino y propone pagos; ejecutarlos requiere autorización independiente para la acción exacta. Una redacción mejor y más contexto no justifican desplegar la combinación. Define la población objetivo y la unidad: un caso de cliente sigue siendo un caso aunque tenga seis reintentos, todos incluidos en coste y riesgo.

El contrato fija modelo, prompt, corpus, herramientas, presupuestos, plazos, rúbricas y exclusiones. Informa por separado respuesta fundamentada o abstención correcta, tiempo y permisos. La fluidez no compensa una acción indebida. Un evaluador caído produce una etiqueta desconocida, no una exclusión silenciosa; una tarea incompleta se juzga según la regla predefinida. No desplegar es una decisión válida ante evidencia insuficiente.

Etapa 1: convertir el trabajo en un entorno reproducible

Incluye facturas normales, políticas ausentes, versiones contradictorias, saldos antiguos, documentos ajenos, beneficiarios cambiados, aprobaciones revocadas y duplicados. Cada caso especifica estado inicial, final permitido y efectos prohibidos: preparar un pago no significa ejecutarlo. La referencia docente usa documentos sintéticos y herramientas simuladas.

Separa desarrollo, validación y prueba final por cliente, familia documental o tiempo. Diez paráfrasis de una factura no suelen ser diez observaciones independientes. Una condición con evidencia correcta suministrada directamente aísla la generación; otra sin evidencia comprueba la abstención. Si producción contiene 90% de casos fáciles y 10% difíciles, una muestra diagnóstica equilibrada no estima directamente el promedio: publica resultados por estrato y pesos explícitos.

Etapa 2: convertir observaciones en juicios defendibles

Utiliza comprobaciones deterministas para esquemas, documentos permitidos, argumentos de pago, estado del libro mayor, duplicados y consumo único de aprobaciones. Reserva revisión humana o un juez calibrado para apoyo semántico. Una llamada demuestra un intento; el estado de negocio demuestra el resultado. Una cita existente puede no respaldar la afirmación.

Calibra con una muestra revisada e informa falsos aprobados y falsos rechazos. Entre 1000 ejemplos sintéticos con 50 errores, aprobar siempre logra 95% de exactitud y omite todos los errores. Cambia orden, longitud e instrucciones persuasivas para detectar sesgos. Congela la versión del juez antes de comparar sistemas; su indisponibilidad genera desconocidos, cuya cobertura se publica junto con errores por categoría.

Etapa 3: controlar variables y cuantificar incertidumbre

Ejecuta ambos sistemas sobre los mismos casos y presupuestos. Las repeticiones estiman variabilidad, pero no crean nuevos clientes. Compara primero recuperación manteniendo el modelo, luego modelo manteniendo recuperación, y finalmente la combinación. Contabiliza contexto y reintentos adicionales. Una semilla de API no garantiza determinismo entre versiones.

Ejemplo sintético: de 200 casos independientes, ambos pasan 150, solo el candidato 20, solo la base 10 y ambos fallan 20. La base logra 80% y el candidato 85%; la diferencia es cinco puntos. Los 30 pares discordantes importan: una aproximación normal docente da error estándar de 2,72 puntos y un intervalo del 95% que cruza ligeramente cero. Si hay dependencia por cliente, remuestrea clientes, no respuestas individuales.

Etapa 4: rechazar un sistema aunque tenga mayor puntuación

Define antes del resultado la mejora mínima útil, incertidumbre admisible, presupuesto y restricciones duras. Una mejora media no compensa pagos indebidos. Cero fallos en 100 pruebas adversarias no implica riesgo cero: con ensayos Bernoulli independientes, el límite superior unilateral del 95% ronda 2,95%. Además, una muestra adversaria no aleatoria no estima incidencia real.

Empieza con ejecución en sombra sin efectos y después un canario acotado que conserve controles externos. Compara grupos contemporáneos y fija horizonte o análisis secuencial válido; no pares al primer resultado favorable. Revertir exige conciliar acciones ya comprometidas, no repetirlas. Cada bloqueo necesita responsable y condiciones documentadas de recuperación.

How the four disciplines fit together

Security determines which actions are permitted. Infrastructure executes work within capacity and failure constraints. Observability preserves evidence of what happened. Evaluation tests whether a change improves the outcome under a declared contract. None substitutes for the others.

PARTE II / REFERENCIAS

Mecanismos, investigación y mercado — referencias en inglés

The following chapters provide the mathematical examples, experiments, research papers and dated company evidence behind the operational story. Return to the lifecycle when deciding where each mechanism belongs.

Synthetic paired evaluation scores with confidence intervals.
Original illustration: uncertain estimates require paired analysis, not a leaderboard-style verdict.
Capability, preference and deployment evaluation answer different questions.
Conceptual taxonomy, not experimental evidence.

01 · Evaluation begins with a decision, not a leaderboard

Imagine that an engineering team is replacing the model behind a customer-support assistant. The new model wins more public head-to-head comparisons and writes more polished answers. Yet its first production week produces more incorrect refunds. Both observations can be true. A public preference ranking measures how a particular population judges particular responses; a refund workflow requires correct policy interpretation, authorization and state transitions. The mistake is not necessarily in either measurement. It is in treating one measurement as a substitute for another.

Definition. An AI evaluation is a specified procedure for gathering evidence about a system's behavior under stated conditions and using that evidence to support a decision. A benchmark is a reusable collection of tasks and measurement rules. A metric summarizes observations. A grader turns an output, action or environment state into a judgment. A harness controls execution and records the conditions. These objects are related, but none is interchangeable with the others.

The decision should be written before collecting scores: “Release candidate B only if it improves successful resolution on our target tasks without increasing unauthorized actions, while meeting our latency and cost budgets.” This statement identifies the target population, the benefit, the constraints and the deployment boundary. It also exposes missing knowledge: what counts as successful resolution, how frequently dangerous cases occur, and how confidently we can measure the difference.

Before this lesson, you should understand a sample mean as an estimate of a population property, a conditional probability as a statement restricted to specified conditions, and a train/test split as a barrier against adapting to the answers. You do not need to have memorized advanced statistics. You do need to recognize that running the same task repeatedly measures execution variability, while collecting new tasks measures a different source of uncertainty.

By the end, you should be able to design a release gate, interpret an Arena-style comparison, calibrate an automated judge, build an auditable evaluation pipeline, and explain why a high score can coexist with an unsuitable product. The examples in this chapter are original teaching examples unless explicitly attributed; they are not measurements of commercial models.

Choose a reading route

First encounter: read the definitions and worked examples, then predict the local experiment result before changing its input. Engineering implementation: focus on the data contract, architecture and Python reference, then write one failure test of your own. Research reading: inspect the cited paper versions, challenge their evaluation assumptions, and choose one open problem with a falsifiable baseline.

02 · Traditional ML, LLMs and agents: what changes?

Traditional supervised evaluation often starts with a fixed input, a reference label and a loss function. A classifier receives an image and predicts a class; a forecasting model predicts a number. Accuracy, precision, recall, calibration and error by subgroup can be useful because the target is comparatively explicit. Even here, labels can be wrong, class distributions can shift, and the cost of a false negative can differ from that of a false positive. Traditional evaluation was never automatically simple or unbiased.

For a language model, several outputs may be acceptable. A concise explanation and a longer explanation can both be correct; exact string matching can reject one for irrelevant reasons. Meanwhile, an apparently correct conclusion may contain fabricated evidence. Evaluation therefore separates dimensions such as factual correctness, instruction following, evidence support, completeness and usefulness. Combining them into one number is a policy choice that requires explicit weights.

An agent adds actions and environmental state. It can finish a task through different trajectories, use extra tools, encounter failures, recover, or damage something unrelated to the task. A final paragraph saying “Done” is not evidence that the task was completed. A database query, repository diff or sandbox state may be the appropriate verifier. The evaluated object is now the model plus its prompt, tool definitions, memory, permissions, retry logic and surrounding application.

Three distinctions prevent many misleading comparisons. First, model evaluation isolates a model as far as possible; system evaluation tests the assembled product. Second, capability evaluation asks what can be accomplished under a budget; reliability evaluation asks how consistently the behavior occurs. Third, offline evaluation estimates performance on controlled tasks, whereas online experimentation estimates effects on users under an actual deployment.

None replaces the others. A model can be capable but unreliable, a reliable product can depend on expensive scaffolding, and a statistically positive online result can conceal a severe safety regression. Keep the axes visible until the decision requires an explicit tradeoff.

03 · Write an evaluation contract

A useful evaluation contract names the task distribution, unit of analysis, execution budget, grading rubric, exclusion policy and release rule. For a support agent, a unit might be one customer case, not one model call. For a coding assistant, it might be one repository issue, not every generated patch. If a case produces ten retries, treating those retries as ten independent customers exaggerates the amount of evidence.

Make the task set versioned and immutable during a comparison. Record the repository snapshot or document corpus, model identifier, prompt hash, tool schema, inference settings, maximum actions, wall-clock timeout and grader version. Record unavailable dependencies and API errors as explicit outcomes. Quietly deleting inconvenient failures changes the question being answered.

Use three task collections with distinct roles. A development collection helps engineers investigate errors. A validation collection supports regular tuning and model selection. A held-out test collection supports a final estimate and should not become a daily feedback channel. When the team has repeatedly inspected the held-out answers, it has effectively converted that set into development data. Refreshing a test set is not a failure; pretending it remains untouched is.

A contract also specifies the aggregation rule. Suppose the task mix is 80% simple retrieval and 20% difficult account changes. Averaging those tasks equally may answer a different question from weighting them by production traffic. Conversely, production-weighting alone may hide a rare but unacceptable unauthorized transfer. Report an overall estimate and decision-critical slices, then apply hard constraints separately where appropriate.

Finally, define inconclusive outcomes. “Insufficient evidence to release” is a legitimate result. A good process does not force every experiment into a winner and a loser. It preserves uncertainty until the expected value of further testing is lower than the cost of waiting.

04 · Uncertainty, paired comparisons and repeated trials

A score is an estimate, not a physical constant. If candidate A succeeds on 80 of 100 tasks, the observed success rate is 80%, but a different sample could produce another value. A confidence interval communicates sampling uncertainty under a specified model. It does not account automatically for mislabeled tasks, a biased sample, contaminated training data or a broken grader.

Compare two systems on the same tasks whenever possible. For each task, calculate the difference between candidate and baseline. Paired comparisons remove some variation due to task difficulty: a hard task is hard for both systems. Bootstrap whole tasks or higher-level clusters such as users or repositories when those are the independent units. Do not independently resample individual tool calls from the same trajectory.

Repeated trials answer a second question: how much does behavior vary when the task is held fixed? Keep the task identifier and trial identifier separate. A system that solves each of ten tasks in one lucky run out of ten has different production behavior from one that reliably solves nine tasks and always fails the tenth. Their pooled success rates can conceal these patterns.

Pass@k describes the chance that at least one of k attempts succeeds under the evaluation's sampling and selection assumptions. It is relevant when a user can actually generate and identify a good candidate. It does not mean the first answer is reliable. A requirement that all k repeated attempts succeed is a different reliability criterion, sometimes written pass^k. Under independent identical success probability p, these teaching models become 1−(1−p)^k and p^k respectively; independence is often unrealistic in deployed systems.

For rare failures, zero observations is not proof of zero risk. Under independent Bernoulli trials, the rough “rule of three” gives a 95% upper bound near 3/n after zero failures in n trials. With 300 clean trials, a failure rate around 1% is still compatible with this approximation. Correlated tasks or an unrepresentative test population weaken that reassurance further.

At least one success is not dependable repetition

With p = 0.8 and k = 3 independent attempts, these values are 0.992 and 0.512. They answer different product questions. For sampled code completions, a common finite-sample pass@k estimator with n generated candidates and c passing candidates is one minus the ratio of combinations shown below. It estimates oracle availability; an imperfect selector can perform worse.

When fewer than k failures exist, the numerator is zero. For n = 10, c = 2 and k = 3, the estimate is 1 − 56/120 = 0.5333. Keep failed, truncated and unparseable attempts in the declared denominator. This estimator does not remove task selection bias.

Source for the sampled-completion estimator: Evaluating Large Language Models Trained on Codehttps://arxiv.org/abs/2107.03374v2, first submitted July 7, 2021; v2 July 14, 2021. Versioned PDFhttps://arxiv.org/pdf/2107.03374v2. The numerical examples here are original and do not reproduce its model scores.

05 · Build graders, then evaluate the graders

Choose the strongest feasible evidence for each requirement. If a generated file must parse, execute a parser. If a balance must remain unchanged, inspect the balance. If an answer must cite an accessible source, verify the source and its relationship to the claim. A language-model judge is useful for semantic judgments that cannot be captured cheaply by deterministic checks, but it should not replace a stronger available verifier.

A rubric should define observable differences between scores. “Excellent” and “poor” are adjectives, not measurement rules. For an evidence-grounded answer, a rubric might distinguish unsupported central claims, partially supported claims with missing qualifications, and a complete answer whose important claims are supported by the supplied documents. Include examples near the decision boundary, not only obvious successes and failures.

Calibrate the judge on an independently annotated set. Measure agreement and inspect the disagreement matrix, especially false passes on consequential errors. An overall agreement score can look strong when almost all examples pass. Include adversarial examples, long fluent wrong answers, concise correct answers, irrelevant citations and attempts to instruct the judge from inside the candidate output.

The foundational MT-Bench and LLM-as-a-judge paperhttps://arxiv.org/abs/2306.05685v4 studies position, verbosity and self-enhancement biases. Its reported human agreement belongs to its tested models and tasks, not every future judge. The paper first appeared on 9 June 2023; the linked v4 is dated 24 December 2023. Read the PDFhttps://arxiv.org/pdf/2306.05685v4.

In your own protocol, randomize answer order, reverse a subset of pairs, blind provider names, constrain the rubric, and audit failures manually. Preserve judge versions and outputs so a score can be reproduced. Treat a judge upgrade as a measurement-system change: regrade a bridge set before interpreting a historical trend as model improvement.

A passing schema can still contain a false answer

The complete Python reference below includes a contract grader for an answer, its source identifiers and an answer/abstain status. It rejects malformed JSON and references outside the allowed source set. A deliberate counterexample changes “30 days” to “300 days” while preserving the schema and source ID: the contract grader still passes. This test is intentional. Structural validity, citation existence and semantic support are different properties; a real application needs a separately calibrated check against source content.

Keep grader failure distinct from candidate failure. An unavailable judge should produce an unknown evaluation state and a retry/review policy, not a fabricated zero or perfect score. Never execute a candidate response as Python merely to parse its structure.

06 · Arena: preference measurement and its boundaries

Arena, formerly LMArena and Chatbot Arena, made pairwise human preference a prominent part of model evaluation. In a blind comparison, a person sees responses to a prompt and expresses a preference before model identities are revealed. The statistical problem is to infer relative strengths from a sparse network of comparisons, not simply to count total wins across unequal opponents.

The Bradley–Terry model represents the probability that model A is preferred to B as a logistic function of their strength difference. The strengths are relative: adding the same constant to every strength leaves pairwise probabilities unchanged. A model with a higher estimated strength is more likely to win under the observed comparison process. This is not an absolute percentage of “intelligence.”

The original Chatbot Arena paperhttps://proceedings.mlr.press/v235/chiang24b.html is an ICML 2024 publication, with its official proceedings PDFhttps://raw.githubusercontent.com/mlresearch/v235/main/assets/chiang24b/chiang24b.pdf. The research establishes a concrete pairwise evaluation framework; it does not establish that public preference is sufficient for every deployment.

Read rankings with their uncertainty, category, date and methodology. Differences can be too small to distinguish confidently. The population of people submitting prompts is not necessarily your customer population. Fluent presentation can affect preference. Different categories can reverse a ranking. Connected comparison coverage matters: a model tested against a narrow or atypical opponent set requires careful estimation.

Arena's 14 July 2026 factuality announcementhttps://arena.ai/blog/factuality-in-arena describes a separate, initially optional Text/Search view combining preference with claim-level factuality audits. Its stated default factuality weight is 25%. This is a useful illustration of an evolving construct: changing the mixture changes what the ranking means. Do not splice the adjusted score into an older preference-only time series without explaining the break.

Use Arena as external evidence and a source of candidate models. Then run a matched evaluation on your own tasks. The engineering question is not “Who is number one?” but “Does this candidate improve our outcome under our constraints?”

Work through the pairwise model

The strengths theta are dimensionless log-odds parameters in this unscaled formulation. A difference of ln(3) gives a 75% predicted preference probability, not a 75% task accuracy. Fix one strength or constrain their sum to remove the arbitrary shared offset. Estimate strengths from the comparison graph, and inspect task and voter dependence before trusting naive independent-vote uncertainty. Ties require an explicit modeling policy rather than silently counting every tie as a decisive win.

07 · Current frontier: measuring validity, contamination and alignment

The research frontier increasingly concerns whether measurements justify their claims. Measuring what Mattershttps://arxiv.org/abs/2511.04703v1, submitted 3 November 2025 and formally published in the NeurIPS 2025 Datasets and Benchmarks trackhttps://proceedings.neurips.cc/paper_files/paper/2025/hash/1967e0fc3aa6cbbace562f5cb8e3954e-Abstract-Datasets_and_Benchmarks_Track.html, reviews 445 benchmarks with 29 expert reviewers. Its central concern is construct validity: whether the task and metric represent the phenomenon claimed. PDFhttps://arxiv.org/pdf/2511.04703v1. A practical implication is to draw the chain from real-world requirement to task, observation and score, then challenge each link.

Contamination is another threat. On 23 February 2026, OpenAI reported problems with SWE-bench Verifiedhttps://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/, including flawed tests and contamination. Its 59.4% flawed-test finding applies to an audited difficult subset of 138 problems, not automatically the entire benchmark. This is a provider-authored audit, and its scope matters. The lesson is to inspect failed cases and grading assumptions, not to assume every low score is a model defect.

Arena's 8 October 2026 Alignment Indexhttps://arena.ai/blog/ai-alignment-index describes three observable signals: unauthorized action, false attribution and deceptive completion. It uses judged evidence and adjusts for conversation length. This is an operational measurement of selected behaviors, not a proof of complete alignment. A narrow, clearly defined indicator can be useful precisely because its limits are visible.

These developments point toward an important discipline: evaluate the evaluation. Test whether a benchmark responds to genuine improvements, whether superficial changes can raise its score, whether labels are reproducible, and whether the score predicts a meaningful deployment outcome. A benchmark that is difficult but unrelated to the product is not automatically valuable.

08 · RAG evaluation: separate retrieval from answering

A retrieval-augmented system can fail before the language model sees the evidence. If the relevant policy paragraph is absent from the retrieved context, improving answer wording will not solve the underlying problem. Conversely, perfect retrieval does not guarantee that the model uses the evidence correctly. Evaluate the pipeline at its interfaces.

For retrieval, define relevance at the unit you actually retrieve: document, chunk or passage. Measure whether the required evidence appears within the allowed context budget. Recall@k is meaningful only with a suitable relevance set and a specified k. A large k can improve recall while increasing latency, cost and distracting context. Test reranking and chunking under the same token budget rather than comparing an inexpensive short context against a much larger one.

For generation, separate answer correctness from support. An answer can be factually correct but unsupported by the retrieved material; it can quote a source accurately while answering the wrong question. Include citation validity, entailment of important claims, completeness, abstention when evidence is insufficient, and handling of conflicting or outdated documents.

Use counterfactual tests. Remove the decisive passage and check whether the system acknowledges uncertainty. Replace a policy with a newer version and inspect whether the answer changes appropriately. Insert an irrelevant but persuasive paragraph and test resistance to distraction. Add a malicious instruction inside a retrieved document and verify that the system treats it as data rather than authority.

A fair ablation changes one component at a time: retrieval method, chunk size, reranker or generator. Keep the corpus snapshot, questions, answer rubric and budget fixed. The result should explain a mechanism—such as improved evidence recall on long policies—rather than merely report a higher aggregate number.

09 · Agent evaluation: outcomes, permissions and recovery

For an agent, evaluate both the final state and the path constraints. A travel assistant that books the right flight using an unauthorized payment method has not succeeded. A coding agent that fixes a test by deleting the test has not solved the intended problem. The verifier needs to inspect meaningful invariants, not only whether the final response sounds successful.

Construct a sandbox with explicit initial state, permitted actions and expected postconditions. Record tool inputs and outputs, environment changes, retries, timeouts and side effects. A task should define whether asking for clarification is acceptable, when human approval is required, and what should happen when a tool fails. Otherwise, the grader may punish sensible behavior or reward recklessness.

Anthropic's 9 January 2026 agent-evaluation guidehttps://www.anthropic.com/research/demystifying-evals-for-ai-agents emphasizes tasks, trials, graders, transcripts and outcomes. The engineering takeaway is to inspect trajectories when scores behave unexpectedly and to combine grader types. The guide is practitioner evidence, not a universal benchmark result.

Security evaluation needs an attacker model. Specify what the attacker controls: a retrieved page, email body, repository file or tool response. State the protected boundary and observable violation. Measure attack success and benign-task completion together; a system that refuses every request may look secure while being useless. Keep attack development separate from the final test set.

Recovery deserves its own tasks. Inject a transient timeout, stale credential, contradictory tool response or partial write, then verify whether the agent retries safely, rolls back, asks for help or stops. Do not assume that a successful rerun demonstrates a reliable repair policy. Compare against a retry-only baseline and account for extra compute.

For recursive self-improvement, freeze the evaluator outside the optimization loop. If the same system proposes changes, rewrites the tests and judges its own success, a rising score may reflect adaptation to the measurement process. Independent holdouts, resource budgets and explicit stopping rules are essential.

10 · Multimodal and physical AI need different evidence

Multimodal evaluation is not text evaluation with an image attached. A visual system may need to locate a small object, read a chart, distinguish a temporal event or combine speech with a scene. A text-only final answer can hide whether the model used the relevant modality or guessed from linguistic priors.

Design modality-specific counterfactuals. Change the number in a chart while leaving the question unchanged. Shift an event to a different video timestamp. Replace spoken words while preserving background noise. Mask the decisive region of an image. A system that gives the same answer after the decisive evidence changes may be relying on the wrong signal.

Report input resolution, frame sampling, audio duration, preprocessing and latency budgets. A model receiving every video frame is not being compared fairly with one receiving a few selected frames unless that resource difference is part of the question. Human ratings of aesthetics or naturalness should be separated from task correctness and accessibility.

Physical AI adds dynamics and safety. A robot policy evaluated only in simulation may exploit simulator artifacts or fail under sensor noise, wear, contact variation and unexpected objects. Report the distribution of environments, reset procedure, intervention rate, task completion, collision or constraint violations, and recovery behavior. A successful demonstration is not a reliability estimate.

The broader principle is to preserve the causal connection between the evidence and the claimed ability. If the claim is spatial reasoning, eliminate shortcuts that can solve the task without spatial evidence. If the claim is robust control, vary the physical conditions rather than replaying one favorable trajectory.

11 · An engineering architecture for continuous evaluation

A production evaluation system has four separable layers: a versioned task store, an execution harness, grading services and a decision/reporting layer. Keeping these boundaries explicit makes it possible to rerun old outputs through a new grader, compare a new model under an unchanged harness, and diagnose whether a regression came from the product or the measurement system.

The task store contains inputs, references, metadata, permissions and provenance. The harness creates isolated environments, enforces budgets and captures artifacts. Graders produce dimension-specific results with evidence. The reporting layer aggregates by task slice, uncertainty and cost, then applies release rules. Every result should be traceable to the exact versions that produced it.

Run inexpensive deterministic checks on every change. Run a representative regression suite in continuous integration. Schedule larger repeated-trial and adversarial evaluations when a model, prompt, retrieval index or tool changes materially. Expensive human review should focus on ambiguous or consequential cases rather than being allocated uniformly without purpose.

Connect evaluation to observability without confusing them. Observability records what happened in production: traces, errors, latency, usage and feedback. Evaluation applies a criterion to selected behavior. A trace is evidence for debugging; it is not automatically a quality label. A useful feedback loop turns appropriately redacted production failures into reviewed regression tasks, while keeping a separate holdout for unbiased assessment.

Protect the measurement pipeline itself. Candidate outputs are untrusted data and can contain instructions aimed at the judge. Sandbox executable submissions, restrict network access, prevent test leakage and log grading exceptions. A failed grader should be reported as unknown or infrastructure failure, not silently converted into a model pass or fail.

Panorama: puntos de control, compradores y competencia

Evidencia revisada el 8 de noviembre de 2026. La documentación acredita una oferta, no eficacia medida independientemente. Las tesis competitivas y oportunidades son análisis de Token Lab; los ejemplos sintéticos no son resultados empresariales.

La industria de evaluación: quién vende evidencia y quién decide

Verificado el 8 de noviembre de 2026. La evaluación de IA reúne varios mercados: laboratorios que compran feedback de capacidades, equipos que necesitan regresiones, riesgo empresarial que exige evidencia auditable y compradores de infraestructura que comparan coste y rendimiento. Ser fuerte en uno no sustituye a los demás. Importa qué decisión respalda la evidencia, bajo qué condiciones y quién responde cuando falla. Separamos productos y transacciones divulgados de nuestra interpretación de incentivos. La financiación y las afirmaciones comerciales no son pruebas independientes de exactitud.

La industria de evaluación: quién vende evidencia y quién decide
Mapa conceptual de responsabilidades, no puntuación ni porcentaje medido.

Cuatro decisiones y cuatro productos de evidencia

Un benchmark público ayuda a preseleccionar modelos; un dataset privado decide un lanzamiento; una evaluación de seguridad explora amenazas; un estudio de producción determina si mejora el proceso empresarial. Son complementarios: un asistente puede ganar en estilo y usar datos de proveedor no autorizados. Nuestra tesis clasifica proveedores por la decisión que mejoran, no por anunciar un juez de IA. Compras debe definir población, tarea, aceptación, coste del fallo y frecuencia de actualización antes de comparar paneles humanos, verificaciones de código, jueces y servicios gestionados.

Arquitectura: decisión de lanzamiento independiente

Es una arquitectura de control de lanzamiento, no un recolector de rankings. El registro fija base y candidato. Casos congelados y permisos equiparados alimentan ejecutores aislados; se guardan observaciones, fallos y costes. Evaluadores versionados y adjudicación aportan evidencias a la agregación pareada. Criterios previos de calidad y riesgo gobiernan el canario. El holdout final queda fuera de la optimización y se usa según protocolo.

Separe feedback de desarrollo y prueba final. Seleccionar repetidamente contra un holdout lo convierte en datos de desarrollo. El candidato no puede editar tareas, evaluadores, puntuaciones ni reglas. Guarde configuración y resultados para atribuir mejoras a cambios concretos. El canario observa el despliegue pero no repara comparaciones contaminadas; la reversión pertenece al sistema de lanzamiento.

Arquitectura: decisión de lanzamiento independiente
Mapa conceptual de responsabilidades, no puntuación ni porcentaje medido.

Posiciones de empresas y productos

Mapa selectivo, no cuota de mercado ni certificación. La cobertura depende de edición, integración, despliegue y modo de aplicación.

Arena / LMArena

Propiedad / organización
Arena Intelligence Inc.
Punto de control
Preferencias humanas y evaluación externa

Arena vende evaluación externa basada en personas, distinta de operar regresiones internas. Su servicio comercial comenzó el 16 de septiembre de 2025 con analítica, muestras y compromisos de entrega. La participación comunitaria es un insumo distintivo; mantener neutralidad creíble mientras vende es el reto. Ganar preferencias no certifica permisos ni corrección empresarial. Especifique producto y metodología: Agent Mode difiere del ranking clásico por pares y no debe recibir la misma descripción estadística.

Arena commercial evaluation launch ↗https://arena.ai/blog/ai-evaluations

Artificial Analysis

Propiedad / organización
Empresa independiente de benchmarking
Punto de control
Selección de modelos e infraestructura

Artificial Analysis mide modelos, agentes, proveedores y hardware, con información pública y API gratuita/comercial. Su comprador busca inteligencia de selección y compras, no un gate de lanzamiento. Propone comparabilidad en calidad, rendimiento y economía. El índice agregado es una construcción: conserve versión, ajustes y fecha. La metodología actual del Intelligence Index es principalmente textual y en inglés; otros idiomas y modalidades requieren evidencia aparte. Úselo para preseleccionar y después pruebe el trabajo real.

Artificial Analysis index methodology ↗https://artificialanalysis.ai/methodology/intelligence-benchmarking

Scale AI / SEAL / Scale Labs

Propiedad / organización
Empresa independiente; inversión minoritaria sin voto de Meta
Punto de control
Datos expertos y evaluación de investigación

Scale combina operaciones de expertos, servicios y investigación mediante SEAL y Scale Labs. Su ventaja potencial es organizar tareas difíciles y adjudicación especializada cuando las etiquetas automáticas no bastan. La inversión minoritaria de Meta es un hecho, no prueba de sesgo ni compra de toda la empresa. La diligencia debe revisar acceso a datos, separación contractual y publicación. La experiencia humana puede mejorar validez, pero el coste del servicio y la consistencia de adjudicación limitan el negocio.

Scale investment and independence statement ↗https://scale.com/blog/scale-ai-announces-next-phase-of-company-evolution

Galileo / Cisco

Propiedad / organización
Adquisición completada el 22 de mayo de 2026
Punto de control
De evaluación a guardrails en ejecución

Galileo conecta evaluación, observabilidad y guardrails, incluidos evaluadores Luna compactos. Cisco completó la compra el 22 de mayo de 2026: ya no debe presentarse como independiente. La oportunidad es llevar criterios offline a control productivo asequible y distribución empresarial. Las cifras de coste y latencia son afirmaciones comerciales sin pruebas comparables. Un scorer rápido también exige calibración y respuesta ante incertidumbre; no reemplaza autorización, aprobación exacta ni recuperación de efectos ya confirmados.

Cisco Galileo acquisition completion ↗https://www.cisco.com/site/us/en/about/corporate-development/acquisitions/galileo/index.html

Databricks / MLflow

Propiedad / organización
Plataforma Databricks y MLflow abierto
Punto de control
Evaluación con gobernanza de datos

MLflow conecta trazas, evaluación, jueces, prompts y feedback humano con desarrollo de modelos. Databricks añade operación gestionada y cercanía a la gobernanza de datos. La ventaja analítica es gravedad de datos y compras existentes; las API abiertas apoyan portabilidad. Verifique versión y edición, sin asumir paridad histórica. Un dataset gobernado y un registro de modelos son útiles, pero no determinan si un pago o borrado externo estaba autorizado para una petición concreta.

Databricks current MLflow observability and portability FAQ ↗https://docs.databricks.com/aws/en/mlflow3/genai/tracing/faq

Microsoft Foundry

Propiedad / organización
Plataforma cloud de Microsoft
Punto de control
De trazas a datasets y lanzamiento

Microsoft Foundry reúne trazas, evaluación, monitorización y gates CI/CD en el entorno empresarial. Documenta convertir trazas en datasets curados y versionados. Su ventaja comercial reduce integración y compras en organizaciones centradas en Microsoft. Un especialista debe justificar mejoras suficientes para compensar otra plataforma. Crear datasets exige selección representativa y control de datos sensibles. Capturar trazas automáticamente no demuestra etiquetas correctas ni mínimo privilegio de los agentes.

Microsoft Foundry traces to evaluation datasets ↗https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/traces-to-dataset

AWS Bedrock / Google Cloud

Propiedad / organización
Ofertas cloud distintas
Punto de control
Evaluación gestionada de modelos y aplicaciones

AWS Bedrock ofrece evaluación de modelos/RAG, opciones humanas y trabajos con jueces. Google Cloud documenta evaluación de modelos y aplicaciones, incluidos terceros. Son productos distintos, agrupados solo por la presión competitiva de distribución cloud. Infraestructura y contratos existentes facilitan adopción; neutralidad y portabilidad requieren diligencia. Gestionar la ejecución no selecciona automáticamente la población correcta ni valida al juez. Los resultados deben conservar configuración, presupuesto y versión de métrica.

Amazon Bedrock Evaluations ↗https://aws.amazon.com/bedrock/evaluations/ · Google Cloud Gen AI evaluation overview ↗https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview

Economía: etiquetas y entornos reproducibles cuestan

El coste no es solo la factura de tokens del juez. Incluye diseño de tareas, expertos, reinicio de entornos, inferencia, repeticiones y mantenimiento. El valor consiste en abaratar una decisión defendible, no en producir más números. Los procesos expertos reutilizables, tareas representativas difíciles de conseguir, controles de contaminación y entornos reproducibles pueden defender mejor un negocio que otro evaluador genérico. Pero las etiquetas de un cliente no siempre pueden reutilizarse legal u operativamente entre tenants. Un supuesto efecto de red de datos debe explicar consentimiento, aislamiento, transferencia y generalización sin revelar tareas confidenciales.

Detalle técnico: tres contratos de calidad

Un benchmark mide tareas declaradas; una arena mide preferencias de usuarios bajo un proceso de presentación y muestreo; la evaluación de despliegue mide el flujo configurado con límites de calidad, seguridad, latencia y coste. Una respuesta preferida puede carecer de soporte, una correcta llegar tarde y un modelo capaz fallar por recuperación o herramientas. Separe las evidencias antes de combinarlas para comprar.

Compare candidatos sobre los mismos casos y examine diferencias pareadas. Fije datos, rúbrica, versión del evaluador y permisos antes de seleccionar; excluya el holdout final de la optimización. Calibre jueces LLM contra ejemplos adjudicados y examine sesgos de posición, estilo y errores compartidos. Informe incertidumbre y familias de tareas. Un promedio mayor no prueba mejora si empeora un fallo costoso. El producto defendible es un proceso repetible de decisión.

Detalle técnico: tres contratos de calidad
Mapa conceptual de responsabilidades, no puntuación ni porcentaje medido.

La brecha pendiente: de la puntuación a la decisión segura

Un juez puede ser consistente y erróneo; varios pueden compartir un punto ciego. Una clasificación pública refleja idiomas, tareas y estilo que pueden diferir del trabajo comprado. Los agentes dependen también del harness, herramientas, estado externo, presupuesto y reinicio: el nombre del modelo no basta. La aceptación debe combinar resultados con invariantes: no acceder a otro tenant, no provocar efectos no autorizados ni escalar ocultamente. La siguiente capa valiosa, en nuestra opinión, integra evidencia con incertidumbre y responsables claros. Debe conectar expertos, ejecuciones reproducibles y resultados sin presentar un proxy automático como verdad.

Adquisiciones, capital e implicaciones estratégicas

Capital y propiedad: precisión sobre independencia

Arena anunció 150 millones de dólares de serie A el 6 de enero de 2026. Scale anunció inversión de Meta el 12 de junio de 2025; Meta declaró después una participación minoritaria sin voto y 13.790 millones incluidos en inversiones no negociables. No es la compra de todo Scale ni el precio de su negocio de evaluación. Cisco completó Galileo el 22 de mayo de 2026 tras anunciar intención el 9 de abril. Son hechos de capital, no calidad. Analizamos propiedad y relaciones como asuntos de diligencia: acceso a datos, publicación, selección de modelos y conflictos. Una relación financiera no demuestra por sí sola resultados sesgados.

Fuentes primarias de productos y transacciones

14 · How AI giants compete through evaluation

Frontier-model competition is partly a competition over what counts as progress. A provider can emphasize mathematical reasoning, coding, agentic task completion, multimodal understanding, latency, price or safety. Those dimensions matter, but selecting the favorable subset can create a misleading overall impression. Read a model launch as a collection of scoped claims rather than a universal ranking.

For every headline comparison, ask whether model versions, prompts, tools, inference budgets and grader settings match. A result with extensive search, retries and a large compute budget does not measure the same product experience as a single low-latency response. If proprietary scaffolding contributes to the score, describe the result as a system result unless the contribution has been isolated.

Providers have real advantages in evaluating their own systems: detailed access to versions, logs and failure modes. They also have incentives to present progress favorably. Independent evaluators can diversify the evidence, but independence is not guaranteed merely by being a separate company. Examine funding, commercial relationships, publication policies, access privileges and the ability to report negative findings.

Enterprise buyers should build a small portfolio of evidence: public capability measurements for breadth, domain-specific held-out tasks for fit, adversarial tests for boundaries, and randomized online experiments for user impact. Require a candidate to improve the dimensions that matter to the business rather than chasing every public leaderboard movement.

The durable competitive advantage may lie in a high-quality feedback loop: representative failures become reviewed tasks, interventions are tested fairly, successful changes are deployed cautiously, and production outcomes verify the hypothesis. This is an engineering interpretation of the market, not a claim that any named company has already solved that loop.

Cloud platforms compete for the evaluation workflow

Microsoft Foundryhttps://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-overview documents evaluation of existing datasets, generated model or agent outputs, and production traces. This is a workflow integration proposition: keeping execution, permissions and evidence near the deployment environment. Buyers should still export a versioned task set and verify that scores remain reproducible outside the portal.

Google Cloud's Gen AI evaluation servicehttps://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview documents evaluating third-party models as well as models in its own ecosystem. This highlights an important separation: the generator being tested need not be the judge. A cross-provider evaluation is not automatically independent, however; check the judge's calibration and version on the target domain.

Amazon Bedrockhttps://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html documents model-evaluation jobs that use a separate LLM to score responses and explain the score. A generated explanation is useful evidence for review, not proof that the score is correct. Keep an adjudicated calibration set, record evaluator configuration and include judge cost in the experiment ledger.

OpenAI and Anthropic also contribute provider-authored audits and engineering guidance, discussed earlier. Across these companies, the competitive axes are model quality, integration, operating cost, access to feedback and trust in measurement. The practical strategy is to keep your evaluation contract portable while using integrations where they reduce work. Do not give the vendor's default rubric authority to redefine your business outcome.

Continue the learning path

Read the companion handbook: evaluation and observability · AI security · AI infrastructure

Learning library · Interview and projects · Editorial standards and corrections

Educational extensions: OpenAI Academyhttps://academy.openai.com/ · Anthropic learning resourceshttps://academy.claude.com/ · Microsoft AI traininghttps://learn.microsoft.com/en-us/ai/.

Primary references are attached to the relevant claims. Conceptual diagrams, synthetic arithmetic and research proposals are original teaching material. Vendor descriptions establish product positioning, not independent proof of performance. Corrections should identify the claim, its source and the proposed replacement.

Práctica y verificación

Etapa 5: cerrar el ciclo y validar al final

La retroalimentación es tardía y selectiva; el silencio no prueba éxito. Un incidente usado para corregir deja de ser una prueba independiente. Mantén linaje entre incidente, caso, rúbrica, ejecución y decisión. Cambiar de juez no demuestra mejor capacidad. El paquete final contiene contrato, pares, segmentos críticos, calibración, costes y condiciones de reversión.

Práctica final: cambia solo la ACL y exige que se bloquee la recuperación; elimina la política y exige abstención. Sustituye 30 días por 300: el esquema pasa, pero el soporte falla. Duplica diez veces un caso sin inflar independencia. Inyecta una acción indebida y exige bloqueo de publicación; interrumpe el juez y exige estado desconocido. Guarda expectativa, resultado y explicación; termina con casos nuevos que no guiaron la corrección.

15 · Open research problems worth a serious experiment

External validity. Can an offline score predict a deployment outcome across organizations? Construct several domain-specific task sets, preregister the predicted ordering, and compare it with a later controlled rollout. Report where the ordering reverses. A single positive correlation on one product is not a universal transfer law.

Adaptive contamination. How can a benchmark remain informative when developers repeatedly optimize against it? Compare fixed public tasks, private held-outs and periodically refreshed tasks under a controlled optimization process. Measure performance on a final unseen distribution. The aim is to separate genuine generalization from learning the measurement interface.

Judge robustness. Can a grader resist style changes and adversarial instructions without losing sensitivity to real improvements? Create matched output pairs that preserve correctness but change verbosity, formatting or persuasion. Then create pairs with identical style but different factual quality. The grader should be invariant to the former when irrelevant and responsive to the latter.

Long-horizon reliability. How do errors accumulate across tool interactions? Vary task length while controlling task type, and distinguish exposure from intrinsic deterioration. A longer task creates more opportunities to fail; a higher session-level failure rate does not alone show that per-action reliability worsened.

Evaluation under limited budgets. Which tasks should be labeled or rerun next? Compare uniform sampling with an adaptive policy, but preserve a probability sample for unbiased reporting. An adaptive strategy can find more failures while producing a distorted raw failure rate if selection probabilities are ignored.

Reward hacking in self-improving systems. Does an optimization loop improve the target capability or exploit its grader? Hold a second evaluator and task set outside the loop, impose equal compute budgets, and compare against a non-adaptive baseline. Publish negative results and evidence of evaluator exploitation. A rising training reward is a hypothesis about improvement, not its verification.

16 · A four-week capstone with a defensible release decision

Build a small document-support agent that answers policy questions and can propose—but not execute—an account change. The baseline uses one retrieval method and a fixed prompt. The candidate changes exactly one component. Use synthetic documents and mock tools so no personal data or real transactions are involved.

In week one, define 60–100 tasks across straightforward retrieval, conflicting evidence, missing evidence and unauthorized requests. Write a rubric and deterministic postcondition checks. Reserve an untouched test partition. Annotate a small calibration set with two reviewers and resolve disagreements by revising the rubric rather than averaging incompatible interpretations.

In week two, implement the harness with task IDs, seeds where applicable, budget limits, artifacts and explicit infrastructure-error states. Run baseline and candidate on the same tasks. Repeat a subset to estimate execution variance. Add a local report showing paired outcomes, critical slices, latency and cost per accepted result.

In week three, audit failures. Test whether the judge can be manipulated by candidate text. Perform an ablation that restores the baseline component, and a budget-matched comparison that prevents extra compute from masquerading as algorithmic improvement. Add discovered failures to the development set while keeping the final test untouched.

In week four, make the release decision under a preregistered rule. A strong report includes the task distribution, exact versions, uncertainty, failed examples, resource costs, safety constraints and reasons to defer deployment. The reference success criterion is not “candidate wins.” It is “another engineer can reproduce and critically evaluate the conclusion.”

For a research extension, measure whether the offline decision predicts a simulated deployment distribution with different proportions of difficult tasks. Explain the discrepancy when distribution shift changes the conclusion.

17 · Original interview practice and scoring

These are original practice questions, not reported questions from any company. Use the same 1–5 progression for each: 1 identifies the relevant concept; 2 gives a correct mechanism; 3 applies it to the concrete example; 4 proposes a sound experiment with controls; 5 explains assumptions, failure cases and the limits of the conclusion.

1. Candidate B wins 55% of public preference comparisons. Should you replace the support model?

Hint 1: Identify the population and outcome being measured.

Hint 2: Distinguish preference from policy correctness and authorization.

Reveal answer and analysis

Answer: Not on that evidence alone. Run a paired comparison on representative support tasks with correctness, policy and safety graders, then examine uncertainty and cost. A strong answer also checks the public comparison's confidence interval, category and versions, and proposes a guarded online experiment after offline gates pass.

2. Why can 90% judge agreement still be unacceptable?

Hint 1: Consider an imbalanced dataset.

Hint 2: Ask which disagreement type is costly.

Reveal answer and analysis

Answer: If almost all examples are acceptable, a judge that passes everything can achieve high agreement while missing every dangerous failure. Inspect class-conditional errors, calibrate on consequential cases and audit false passes. A top answer explains why agreement with one imperfect annotator is not ground truth and defines an adjudication process.

3. An agent succeeds at least once in five runs but often fails the first run. Which metric matters?

Hint 1: What attempts can the product afford?

Hint 2: Can it identify the successful attempt reliably?

Reveal answer and analysis

Answer: Pass@5 is relevant only if five attempts and reliable selection are available. For a single-shot workflow, first-attempt success and failure severity matter. For repeated dependable execution, measure all-attempt reliability and task-level variance. Include latency, cost, correlated failures and side effects from unsuccessful attempts.

4. A retrieval change raises answer quality. How do you establish why?

Hint 1: Separate evidence availability from generation.

Hint 2: Match the context and compute budgets.

Reveal answer and analysis

Answer: Measure evidence recall, answer support and correctness on the same tasks, keeping the generator fixed. Compare retrieval-only ablations under equal context budgets. Inspect whether gains concentrate on previously missing evidence and test distraction and stale-document cases. A high-scoring answer avoids claiming causality from aggregate correlation alone.

5. A self-improving agent raises its own benchmark score every day. What would convince you it improved?

Hint 1: Identify what the agent can modify or observe.

Hint 2: Keep independent evidence outside the loop.

Reveal answer and analysis

Answer: Freeze an independent evaluator and held-out tasks, compare under equal resource budgets, and check whether gains transfer to new tasks and robust behavior. Audit grader exploitation and contamination. The strongest answer defines a preregistered stopping rule, preserves failed runs and distinguishes increased capability from increased search effort.

Reproduce the arithmetic: a complete local reference

Imagine evaluating a replacement support agent on twenty matched tasks. Twelve tasks pass under both versions, four pass only under B, two pass only under A, and two fail under both. Model A scores 14/20 and B scores 16/20. We resample task pairs, keeping the relationship between outcomes intact. The resulting interval is wide: this demonstration does not establish that B is better in the population. The individual Wilson interval is a separate description of B's success rate; overlap between two marginal intervals is not the paired comparison test.

Here n is the number of independent tasks and outcomes are binary. The second expression is the exact one-sided upper confidence bound on failure probability after zero observed failures, assuming independent identically distributed Bernoulli trials; alpha = 0.05 gives 95% confidence. Correlated tasks or an unrepresentative sample invalidate that interpretation.

Revisit the opening score-and-uncertainty diagram. Its error bars are marginal Wilson intervals; the paired interval below addresses the difference directly.

The bootstrap is a percentile approximation, not a guarantee of calibrated coverage with just twenty examples. If multiple prompts belong to one customer or document, resample customers or documents. More repeated generations per item measure stochastic variability; they do not manufacture more independent customers.

Try the denominator locally

No input leaves your browser. This is an arithmetic illustration, not a production risk estimate.

Runtime: Python 3.10 or newer; standard library only. The evaluation bootstrap uses seed 17 and 10,000 resamples. Observability arithmetic is deterministic and needs no seed. These programs make no network or model calls.

Download the complete Python reference

python3 evaluation-audit.py
"""Synthetic paired evaluation; Python 3.10+, standard library, seed 17.
No API calls. Items, rather than individual judgments, are bootstrap units.
"""
import math
import random
import json
from statistics import mean


def wilson(successes, n, z=1.959963984540054):
    if not 0 <= successes <= n or n <= 0:
        raise ValueError('Require 0 <= successes <= n and n > 0')
    p = successes / n
    den = 1 + z*z/n
    center = (p + z*z/(2*n)) / den
    radius = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / den
    return center-radius, center+radius


def paired_bootstrap(a, b, repetitions=10000, seed=17):
    if len(a) != len(b) or not a:
        raise ValueError('Require nonempty matched items')
    if not isinstance(repetitions, int) or repetitions < 100:
        raise ValueError('Use at least 100 bootstrap repetitions')
    if any(v not in (0, 1) for v in a + b):
        raise ValueError('This example requires binary outcomes')
    rng = random.Random(seed)
    differences = [y-x for x, y in zip(a, b)]
    means = sorted(mean(rng.choices(differences, k=len(a)))
                   for _ in range(repetitions))
    return mean(differences), means[int(.025*repetitions)], means[int(.975*repetitions)]



def grade_response(raw, allowed_sources, expected_status):
    """Original deterministic contract grader, NOT a semantic truth judge.
    Candidate text is parsed as data; never eval(), execute(), or prompt a judge.
    """
    try:
        response = json.loads(raw)
    except (json.JSONDecodeError, TypeError):
        return {"schema": False, "evidence_ids": False, "status": False}
    schema = (isinstance(response, dict)
              and set(response) == {"answer", "sources", "status"}
              and isinstance(response["answer"], str)
              and isinstance(response["sources"], list)
              and all(isinstance(x, str) for x in response["sources"])
              and response["status"] in ("answer", "abstain"))
    if not schema:
        return {"schema": False, "evidence_ids": False, "status": False}
    return {"schema": True,
            "evidence_ids": (all(x in allowed_sources for x in response["sources"])
                             and (response["status"] != "answer" or bool(response["sources"]))),
            "status": response["status"] == expected_status}


valid = '{"answer":"The policy says 30 days.","sources":["policy-v3"],"status":"answer"}'
assert all(grade_response(valid, {"policy-v3"}, "answer").values())
assert not grade_response('not JSON', set(), 'abstain')['schema']
assert not grade_response(valid, {"policy-v4"}, "answer")['evidence_ids']
assert not grade_response(valid, {"policy-v3"}, "abstain")['status']
# A plausible-looking lie can pass these structural checks. Add human/semantic
# review against the actual source; do not call this contract pass "correctness".
lie = valid.replace('30 days', '300 days')
assert all(grade_response(lie, {"policy-v3"}, "answer").values())

# Same 20 synthetic tasks: 12 both pass, 4 only B, 2 only A, 2 neither.
a = [1]*12 + [0]*4 + [1]*2 + [0]*2
b = [1]*12 + [1]*4 + [0]*2 + [0]*2
assert sum(a) == 14 and sum(b) == 16
low, high = wilson(sum(b), len(b))
delta, lower, upper = paired_bootstrap(a, b)
assert lower <= 0 <= upper  # small sample does not establish superiority
assert paired_bootstrap(a, a) == (0, 0, 0)
assert wilson(0, 20)[0] >= -1e-12
assert math.isclose(1 - .05**(1/100), .029513049607039932)
# Trial independence is an assumption, not a property guaranteed by a model.
assert math.isclose(1-(1-.8)**3, .992)
assert math.isclose(.8**3, .512)
print(f'A: {mean(a):.1%}; B: {mean(b):.1%}')
print(f'B Wilson 95% interval: [{low:.3f}, {high:.3f}]')
print(f'Paired delta: {delta:.1%}; bootstrap 95% interval: [{lower:.3f}, {upper:.3f}]')
print('Zero failures in 100: one-sided 95% upper bound 2.95%')
print('All evaluation checks passed; synthetic data, not model performance.')

Expected output:

A: 70.0%; B: 80.0%
B Wilson 95% interval: [0.584, 0.919]
Paired delta: 10.0%; bootstrap 95% interval: [-0.150, 0.350]
Zero failures in 100: one-sided 95% upper bound 2.95%
All evaluation checks passed; synthetic data, not model performance.

Test the failure cases: empty samples are rejected; the evaluation comparison of a model with itself has zero difference; the trace algorithm rejects cycles. Next add clustered tasks or missing spans, and document exactly which assumptions stop holding. Unit checks validate this implementation, not a deployed model.

Fundamentos explicados

Repasa el razonamiento que sostiene el método y utiliza las comprobaciones para detectar lagunas.

Un experimento estima una cantidad bajo condiciones explícitas. Define la unidad: solicitud, problema, semilla o episodio no son equivalentes. La comparación pareada usa los mismos casos. Una ablación cambia un mecanismo conservando datos, presupuesto y evaluación; quitar también recursos a la referencia confunde la conclusión.

Razona antes de abrir la solución

A obtiene [0,1,1] y B [1,0,1] en los mismos casos. ¿Qué revela y oculta la media?

Solución paso a paso

Diferencias [1,−1,0], media cero: hay una mejora y una regresión. Media igual no demuestra equivalencia y tres casos no bastan para inferencia robusta. Examina grupos e incertidumbre; repeticiones correlacionadas no son evidencia independiente.

Conexión con la práctica

Conecta afirmación con tarea, partición, versión, presupuesto, métrica y fallos. Separa resultados publicados de ejemplos didácticos. Repetir aritmética no reproduce entrenamiento; adaptar al benchmark puede contaminar la evaluación.

Cómo demostrar comprensión

Define una afirmación refutable, fija evaluación antes de ajustar y conserva resultados y configuración. Compara una referencia creíble con igual presupuesto. En automejora usa tareas externas reservadas e informa regresiones.

Ejemplo Python completo

Python 3.10+ · solo biblioteca estándar · determinista, sin semilla aleatoria · sin red. Guarda el archivo y ejecuta el comando. Son comprobaciones didácticas, no benchmarks.

Descargar Python · evidence.py
python3 evidence.py
# Python 3.10+; standard library; deterministic; no network.
a, b = [0, 1, 1], [1, 0, 1]
assert len(a) == len(b)
differences = [new - old for old, new in zip(a, b)]
mean_change = sum(differences) / len(differences)
assert mean_change == 0
assert sum(x > 0 for x in differences) == 1
assert sum(x < 0 for x in differences) == 1
print("mean change=0; improved=1; regressed=1; unchanged=1")

Salida esperada

mean change=0; improved=1; regressed=1; unchanged=1

Repasar el fundamento →

Fundamento revisado:

Tu siguiente paso

Fija el conjunto reservado; compara ejecuciones emparejadas, incertidumbre, fallos y presupuesto.

Implementar y verificar → · Continuar la ruta →

Predecir, comprobar y conservar evidencia

Antes de revelar la respuesta, predice un resultado. Después registra diferencias y una prueba de transferencia. Se guarda localmente y se comparte entre idiomas.

Abrir cuaderno / exportar / borrar →

El alcance de revisión matemática se registra por lección. No certifica reproducción empírica del artículo; anota posibles errores. Alcance de revisión