PHD / RESEARCH ENGINEERING

논문 재현 워크벤치

Record a falsifiable hypothesis, inspect controls and export a reviewable experiment report.

로컬 계산과 합성 예제입니다. 유료 API나 실제 모델 벤치마크를 사용하지 않습니다.

기술 본문은 영어와 중국어로 제공됩니다. 이 언어에서는 영어 기술 본문을 표시합니다.

Study guide & lesson outline

LEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →

Understand: state assumptions, shapes/units and a counterexample.
Implement: reproduce a numerical example and inspect boundaries.
Investigate: compare one alternative under a matched budget.

01 · 이해와 유도

사전 지식 · Experimental controls, versioning, paired metrics.

A reproduction claim needs a traceable relationship between the paper, implementation, data and observations. The workbench checks declared fields, budget equality and top-level changed configuration keys. It never certifies that the declarations are true.

Text / 原始公式

Hypothesis → baseline → one-factor intervention → paired runs → uncertainty → limitations
Required: paper, hypothesis, metric, direction, threshold, datasetVersion, splitHash, baseline, treatment, runs

Step-by-step derivation and interpretation

Choose one falsifiable paper claim and define what observation would contradict it. Separate exact reproduction, approximate reproduction with declared deviations, and testing generalization.
Freeze dataset/split identifiers, implementation commit, environment, evaluation script, checkpoint selection and search budget. Record failures and exclusions as well as successful runs.
A manifest validates declared consistency, not provenance. Matching budget numbers cannot establish matching compute, and one changed top-level JSON key may hide multiple interventions.

Common pitfall: Do not mark a study as successfully reproduced because a checklist is green. Compare the actual protocol and observations with the original claim.

02 · 이해도 확인

A baseline and treatment share seed IDs [1,2,3,4,5]. How many paired run observations enter the analysis?

유도 과정 보기

Five pairs, not ten independent paired observations.

03 · 통제 실험 실행

실험 과제

Replace the demo metadata with actual identifiers, paste paired seed results, then introduce a second configuration change or budget mismatch. Export the report and explain which concerns remain manual.

Controlled experiment protocol

Replace demo paper and metadata with the actual protocol. Use real dataset versions and source commit identifiers; retain the explicit metric direction.
Paste independent paired run means. Change budget or a second top-level configuration key to inspect review flags; explain any justified deviations in the report.
Export JSON and Markdown. Attach raw logs separately and document claim, method, controls, uncertainty, failure cases and the narrowest conclusion supported by your evidence.

Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.

힌트 보기

Equal step counts need not imply equal compute. Hashes and commit IDs are only declarations here. Read the paper and inspect the actual split membership before claiming successful reproduction.

Experiment manifest JSON

{
  "paper": "https://arxiv.org/abs/2106.09685",
  "hypothesis": "Increasing rank improves a held-out metric at matched training budget.",
  "metric": "accuracy",
  "direction": "higher",
  "threshold": 0.01,
  "datasetVersion": "toy-v1",
  "splitHash": "demo-split-not-a-real-hash",
  "baseline": {
    "config": {
      "rank": 2,
      "learningRate": 0.1
    },
    "budget": 1000,
    "budgetUnit": "optimizer steps"
  },
  "treatment": {
    "config": {
      "rank": 4,
      "learningRate": 0.1
    },
    "budget": 1000,
    "budgetUnit": "optimizer steps"
  },
  "environment": {
    "codeCommit": "fill-with-real-commit",
    "runtime": "record exact runtime",
    "hardware": "record device"
  },
  "runs": [
    {
      "seed": 1,
      "a": 0.72,
      "b": 0.75
    },
    {
      "seed": 2,
      "a": 0.74,
      "b": 0.76
    },
    {
      "seed": 3,
      "a": 0.71,
      "b": 0.75
    },
    {
      "seed": 4,
      "a": 0.75,
      "b": 0.76
    },
    {
      "seed": 5,
      "a": 0.73,
      "b": 0.76
    }
  ]
}

로컬 JSON/CSV 파일 읽기

04 · 근거 확인

실행하면 결과가 표시됩니다.

변수를 통제하여 결과를 비교하세요. 내보내기에는 입력과 방법이 포함됩니다.

05 · 추론 설명

자기 평가: 원리와 가정을 설명하고 구별 가능한 실험을 제안하세요. LLM 자동 채점은 없습니다.

1. Does a green manifest check prove reproducibility?

추론 작성 (저장되지 않음)

참고 답변과 평가 기준

No. It checks internal declarations. Verify code, data versions, environment, execution and agreement with the target protocol.

2. How should a negative reproduction result be reported?

추론 작성 (저장되지 않음)

참고 답변과 평가 기준

State the attempted protocol, deviations, budget, uncertainty and failed hypotheses; do not hide null results or assume the original claim is false.

3. What separates reproducibility from generalization?

추론 작성 (저장되지 않음)

참고 답변과 평가 기준

Reproducibility repeats a result under a specified setup; generalization tests transfer to different data, environments or tasks.

가정과 한계

No paper fetch, cloud training or provenance verification. The included runs are synthetic. Nested configuration differences are counted by top-level key; several causal changes can hide inside one key. Manual review remains required.

참고 문헌

arXiv:1709.06560 · 2017

Deep Reinforcement Learning that Matters

Henderson et al. · Year denotes first arXiv submission

Read: Sources of experimental variability and reporting recommendations.

Reading checkpoint: Separate randomness from implementation and tuning differences. RL findings motivate controls; do not blindly transfer its numerical results to LLMs.

Analysis & visualization →arXiv abstract ↗PDF ↗

BibTeX

@misc{arxiv170906560,
  title = {Deep Reinforcement Learning that Matters},
  year = {2017},
  eprint = {1709.06560},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.06560}
}

arXiv:1909.03004 · 2019

Show Your Work: Improved Reporting of Experimental Results

Dodge et al. · Year denotes first arXiv submission

Read: Expected validation performance under a computational budget; model selection.

Reading checkpoint: Report search space, trial count and selection rule alongside the final score.

Analysis & visualization →arXiv abstract ↗PDF ↗

BibTeX

@misc{arxiv190903004,
  title = {Show Your Work: Improved Reporting of Experimental Results},
  year = {2019},
  eprint = {1909.03004},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.03004}
}

MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.

Claim, split, budget and environment → Validate manifest and paired seeds → Check one-factor and matched budget → Statistics + unresolved provenance — Mechanism map: follow the transformations from top to bottom.

Before running: predict a boundary case

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node paper-reproduction.mjs. No third-party runtime dependencies.

import {runPhd} from './phd-core.mjs';
const params={
  "manifest": "{\n  \"paper\": \"https://arxiv.org/abs/2106.09685\",\n  \"hypothesis\": \"Increasing rank improves a held-out metric at matched training budget.\",\n  \"metric\": \"accuracy\",\n  \"direction\": \"higher\",\n  \"threshold\": 0.01,\n  \"datasetVersion\": \"toy-v1\",\n  \"splitHash\": \"demo-split-not-a-real-hash\",\n  \"baseline\": {\n    \"config\": {\n      \"rank\": 2,\n      \"learningRate\": 0.1\n    },\n    \"budget\": 1000,\n    \"budgetUnit\": \"optimizer steps\"\n  },\n  \"treatment\": {\n    \"config\": {\n      \"rank\": 4,\n      \"learningRate\": 0.1\n    },\n    \"budget\": 1000,\n    \"budgetUnit\": \"optimizer steps\"\n  },\n  \"environment\": {\n    \"codeCommit\": \"fill-with-real-commit\",\n    \"runtime\": \"record exact runtime\",\n    \"hardware\": \"record device\"\n  },\n  \"runs\": [\n    {\n      \"seed\": 1,\n      \"a\": 0.72,\n      \"b\": 0.75\n    },\n    {\n      \"seed\": 2,\n      \"a\": 0.74,\n      \"b\": 0.76\n    },\n    {\n      \"seed\": 3,\n      \"a\": 0.71,\n      \"b\": 0.75\n    },\n    {\n      \"seed\": 4,\n      \"a\": 0.75,\n      \"b\": 0.76\n    },\n    {\n      \"seed\": 5,\n      \"a\": 0.73,\n      \"b\": 0.76\n    }\n  ]\n}"
};
console.log(runPhd('paper-reproduction',params));

Download this example phd-core.mjs academy-core.mjs research-core.mjs

Test and transfer

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

My predictionNotes / code revision / measured evidenceMistakes and concepts to revisit I independently explained, implemented and evaluated this lesson.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope