1. Why not treat 1000 examples from one training run as 1000 seeds?
Show answer & scoring criteria
They do not capture independent training randomness. The sampling unit must match the uncertainty being estimated.
Experiment Design Lab
Analyze paired run means, uncertainty and a predeclared practical improvement threshold.
Local calculations and synthetic examples. No paid API or live-model benchmark.Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
01 · Understand & derive
Prerequisites · Means, variance, independent units, paired comparisons.
Pair runs only when they share a meaningful experimental unit. Bootstrap whole pairs so covariance is retained. This lab treats each row as an independent run/seed mean, not each example from one run. Compare the interval with a pre-specified practical threshold.
Text / 原始公式
dᵢ = Bᵢ − Aᵢ (reverse sign for lower-is-better) Δ = mean(dᵢ) Resample n pairs with replacement, 2000 times CI = [percentile₂.₅, percentile₉₇.₅]
Step-by-step derivation and interpretation
- Specify the estimand first: mean improvement across independent training runs is different from uncertainty across evaluation examples. Match the resampling unit to the claim.
- For paired runs,
,B). Resample whole pairs so their relationship survives. Pairing IDs alone cannot establish that a pairing is scientifically meaningful. - Separate practical effect size, interval uncertainty and hypothesis testing. This tool provides a percentile interval, not a p-value. It does not correct for repeatedly selecting the best experiment.
Common pitfall: A narrow interval is not protection against leakage, correlated seeds, benchmark selection or an unfair tuning budget.
02 · Check your understanding
For paired improvements [0.02,0.04,0.03], what is the mean improvement in metric units?
Show derivation
(0.02+0.04+0.03)/3 = 0.03; for accuracy this is 3 percentage points
03 · Run a controlled experiment
Experiment assignment
Analyze the sample, reverse metric direction, and duplicate a seed ID to observe validation. Then compare a noisy run set with the same mean improvement. Explain why mean alone is insufficient.
Controlled experiment protocol
- With the five default pairs, mean improvement is 0.026; the fixed bootstrap seed gives approximately [0.018,0.034]. Keep the threshold at 0.01 initially.
- Use threshold 0.03 to see an inconclusive comparison. This is a teaching sensitivity check; in a real study predeclare the threshold rather than moving it after seeing results.
- Build a second dataset with the same mean delta and greater spread. Explain the interval change, seed count limits and why duplicate IDs are rejected.
Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.
Show a hint
An interval crossing the threshold is inconclusive under this rule. An interval above it is evidence conditional on sampling assumptions, not proof of causation or freedom from leakage.
04 · Inspect evidence
Run the experiment to see evidence.
Only compare runs with controlled inputs. Exports include inputs and methods.
05 · Defend your reasoning
Self-assessment: explain the mechanism, identify assumptions, and propose a discriminating test. No automatic LLM grading.
2. Why predeclare the practical threshold?
Show answer & scoring criteria
Choosing it after seeing results encourages selective conclusions. Thresholds should reflect the task and cost of deployment.
3. What else must a credible ablation control?
Show answer & scoring criteria
Data and splits, training and tuning budget, evaluation protocol, seed pairing and selection of the reported checkpoint.
Assumptions & limits
95% paired percentile bootstrap, 2000 resamples, at most 200 rows. Small samples and constant deltas can be misleading. No multiple-testing correction, power calculation or automatic causal inference.
References
arXiv:1709.06560 · 2017
Deep Reinforcement Learning that Matters
Henderson et al. · Year denotes first arXiv submission
Read: Sources of experimental variability and reporting recommendations.
Reading checkpoint: Separate randomness from implementation and tuning differences. RL findings motivate controls; do not blindly transfer its numerical results to LLMs.
BibTeX
@misc{arxiv170906560,
title = {Deep Reinforcement Learning that Matters},
year = {2017},
eprint = {1709.06560},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1709.06560}
}arXiv:1909.03004 · 2019
Show Your Work: Improved Reporting of Experimental Results
Dodge et al. · Year denotes first arXiv submission
Read: Expected validation performance under a computational budget; model selection.
Reading checkpoint: Report search space, trial count and selection rule alongside the final score.
BibTeX
@misc{arxiv190903004,
title = {Show Your Work: Improved Reporting of Experimental Results},
year = {2019},
eprint = {1909.03004},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1909.03004}
}Assumptions, implementation & acceptance
95% paired percentile bootstrap, 2000 resamples, at most 200 rows. Small samples and constant deltas can be misleading. No multiple-testing correction, power calculation or automatic causal inference.
Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.
Before running: predict a boundary case
Analyze the sample, reverse metric direction, and duplicate a seed ID to observe validation. Then compare a noisy run set with the same mean improvement. Explain why mean alone is insufficient.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node experiment-design.mjs. No third-party runtime dependencies.
import {runPhd} from './phd-core.mjs';
const params={
"csv": "id,a,b\n1,0.72,0.75\n2,0.74,0.76\n3,0.71,0.75\n4,0.75,0.76\n5,0.73,0.76",
"direction": "higher",
"threshold": 0.01,
"seed": 42
};
console.log(runPhd('experiment-design',params));Test and transfer
Analyze the sample, reverse metric direction, and duplicate a seed ID to observe validation. Then compare a noisy run set with the same mean improvement. Explain why mean alone is insufficient.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Download complete kit (.tar.gz) · Complete reference kit and regression checksPredict, test and keep evidence
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
Open notebook / export / clear →The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope