1. Why not treat 1000 examples from one training run as 1000 seeds?
참고 답변과 평가 기준
They do not capture independent training randomness. The sampling unit must match the uncertainty being estimated.
실험 설계
Analyze paired run means, uncertainty and a predeclared practical improvement threshold.
로컬 계산과 합성 예제입니다. 유료 API나 실제 모델 벤치마크를 사용하지 않습니다.기술 본문은 영어와 중국어로 제공됩니다. 이 언어에서는 영어 기술 본문을 표시합니다.
Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
01 · 이해와 유도
사전 지식 · Means, variance, independent units, paired comparisons.
Pair runs only when they share a meaningful experimental unit. Bootstrap whole pairs so covariance is retained. This lab treats each row as an independent run/seed mean, not each example from one run. Compare the interval with a pre-specified practical threshold.
Text / 原始公式
dᵢ = Bᵢ − Aᵢ (reverse sign for lower-is-better) Δ = mean(dᵢ) Resample n pairs with replacement, 2000 times CI = [percentile₂.₅, percentile₉₇.₅]
Step-by-step derivation and interpretation
- Specify the estimand first: mean improvement across independent training runs is different from uncertainty across evaluation examples. Match the resampling unit to the claim.
- For paired runs,
,B). Resample whole pairs so their relationship survives. Pairing IDs alone cannot establish that a pairing is scientifically meaningful. - Separate practical effect size, interval uncertainty and hypothesis testing. This tool provides a percentile interval, not a p-value. It does not correct for repeatedly selecting the best experiment.
Common pitfall: A narrow interval is not protection against leakage, correlated seeds, benchmark selection or an unfair tuning budget.
02 · 이해도 확인
For paired improvements [0.02,0.04,0.03], what is the mean improvement in metric units?
유도 과정 보기
(0.02+0.04+0.03)/3 = 0.03; for accuracy this is 3 percentage points
03 · 통제 실험 실행
실험 과제
Analyze the sample, reverse metric direction, and duplicate a seed ID to observe validation. Then compare a noisy run set with the same mean improvement. Explain why mean alone is insufficient.
Controlled experiment protocol
- With the five default pairs, mean improvement is 0.026; the fixed bootstrap seed gives approximately [0.018,0.034]. Keep the threshold at 0.01 initially.
- Use threshold 0.03 to see an inconclusive comparison. This is a teaching sensitivity check; in a real study predeclare the threshold rather than moving it after seeing results.
- Build a second dataset with the same mean delta and greater spread. Explain the interval change, seed count limits and why duplicate IDs are rejected.
Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.
힌트 보기
An interval crossing the threshold is inconclusive under this rule. An interval above it is evidence conditional on sampling assumptions, not proof of causation or freedom from leakage.
04 · 근거 확인
실행하면 결과가 표시됩니다.
변수를 통제하여 결과를 비교하세요. 내보내기에는 입력과 방법이 포함됩니다.
05 · 추론 설명
자기 평가: 원리와 가정을 설명하고 구별 가능한 실험을 제안하세요. LLM 자동 채점은 없습니다.
2. Why predeclare the practical threshold?
참고 답변과 평가 기준
Choosing it after seeing results encourages selective conclusions. Thresholds should reflect the task and cost of deployment.
3. What else must a credible ablation control?
참고 답변과 평가 기준
Data and splits, training and tuning budget, evaluation protocol, seed pairing and selection of the reported checkpoint.
가정과 한계
95% paired percentile bootstrap, 2000 resamples, at most 200 rows. Small samples and constant deltas can be misleading. No multiple-testing correction, power calculation or automatic causal inference.
참고 문헌
arXiv:1709.06560 · 2017
Deep Reinforcement Learning that Matters
Henderson et al. · Year denotes first arXiv submission
Read: Sources of experimental variability and reporting recommendations.
Reading checkpoint: Separate randomness from implementation and tuning differences. RL findings motivate controls; do not blindly transfer its numerical results to LLMs.
BibTeX
@misc{arxiv170906560,
title = {Deep Reinforcement Learning that Matters},
year = {2017},
eprint = {1709.06560},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1709.06560}
}arXiv:1909.03004 · 2019
Show Your Work: Improved Reporting of Experimental Results
Dodge et al. · Year denotes first arXiv submission
Read: Expected validation performance under a computational budget; model selection.
Reading checkpoint: Report search space, trial count and selection rule alongside the final score.
BibTeX
@misc{arxiv190903004,
title = {Show Your Work: Improved Reporting of Experimental Results},
year = {2019},
eprint = {1909.03004},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1909.03004}
}Assumptions, implementation & acceptance
95% paired percentile bootstrap, 2000 resamples, at most 200 rows. Small samples and constant deltas can be misleading. No multiple-testing correction, power calculation or automatic causal inference.
Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.
Before running: predict a boundary case
Analyze the sample, reverse metric direction, and duplicate a seed ID to observe validation. Then compare a noisy run set with the same mean improvement. Explain why mean alone is insufficient.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node experiment-design.mjs. No third-party runtime dependencies.
import {runPhd} from './phd-core.mjs';
const params={
"csv": "id,a,b\n1,0.72,0.75\n2,0.74,0.76\n3,0.71,0.75\n4,0.75,0.76\n5,0.73,0.76",
"direction": "higher",
"threshold": 0.01,
"seed": 42
};
console.log(runPhd('experiment-design',params));Test and transfer
Analyze the sample, reverse metric direction, and duplicate a seed ID to observe validation. Then compare a noisy run set with the same mean improvement. Explain why mean alone is insufficient.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Download complete kit (.tar.gz) · Complete reference kit and regression checksPredict, test and keep evidence
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
Open notebook / export / clear →The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope