InstructGPT: change the training signal to change behavior
Connect demonstrations, preference labels, reward learning and policy optimization.
Training language models to follow instructions with human feedback · Ouyang et al. · arXiv:2203.02155
Original editorial analysis by Token Lab · Source checked 2026-10-25 · Paper claims and synthetic examples are separated.
01 · What the paper establishes
InstructGPT uses supervised demonstrations, human comparisons and reinforcement learning to improve instruction following. The paper reports human preferences on its prompt distribution and analyzes remaining limitations. The result is not that parameter count ceases to matter or that all human values have been aligned.
Primary source: paper abstract ↗Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
02 · The mechanism
Text / 原始公式
Demonstrations → SFT policy Ranked responses → learned reward rφ Policy objective ≈ E[rφ]−β KL(πθ||πref) Some variants additionally mix pretraining updates.
A next-token objective and a user-preference objective reward different behavior. A learned reward model supplies an imperfect proxy that the policy can optimize. The reference penalty discourages unrestricted drift, while independent evaluation checks whether proxy improvement corresponds to better user-facing outputs. Each stage has different data requirements and failure modes.
Notation and intuition are teaching explanations. Consult the original method for complete assumptions and implementation details.
Mechanism map & worked derivation
| Symbol | Meaning |
|---|---|
| rφ | Learned reward proxy |
| πref | Reference policy for the drift penalty |
| β | Trade-off coefficient under the stated convention |
- Distinguish three data products: demonstrations for SFT, comparisons for reward learning, and sampled trajectories for policy optimization. They are not interchangeable labels.
- For two policies on the same prompt distribution, (expected reward, expected KL)=(1,0.4) and (0.8,0.1),
gives regularized values 0.6 and 0.7. A lower raw reward can win after the drift penalty. - Evaluate the selected policy with independent task labels and blinded human comparisons. Otherwise reward-model errors can become both the training target and the apparent success criterion.
Report who labeled the data and which prompt distribution was evaluated. A single reward score cannot represent every user objective.
Change one variable. Explain what moves.
Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.
04 · Work through an example
Original numerical example: hold raw reward at 1 and an illustrative KL value at 0.2. The regularized score is 1−0.2β. At
05 · Design your experiment
Keep a blinded, held-out prompt set and compare base, SFT and preference-trained checkpoints at equal decoding budgets. Audit correctness separately from preference. For reward-model validation, measure pairwise agreement on new prompts and inspect disagreements. Include examples where verbosity, confidence or refusal style changes without factual improvement.
State one hypothesis; save the configuration and raw results; identify a competing explanation; report the narrowest supported conclusion. Use the linked lab as a local starting point, not a full paper reproduction.
Open related lesson / lab →06 · Limits & reviewer questions
Editorial caution: the evaluator population and prompt distribution delimit the claim. Reward hacking, distribution shift and capability regressions require dedicated tests. Local teaching diagrams do not execute model training, collect human feedback or establish an alignment result.
Why keep an evaluation separate from the reward model?
Optimizing a proxy can exploit its errors. Reusing the same proxy as the only evaluator cannot distinguish genuine task improvement from overfitting to the proxy.
07 · Read the original
Training pipeline, human evaluation and limitations; track the data source at each stage.
Year is the first arXiv submission year. Unversioned links follow revisions; record the exact version in your reproduction report. DeepSeek-R1 here is explicitly pinned to v1.
Continue this topic
PPO: inspect the clipped objective one sample at a time
Understand the sign of advantage, the rollout-policy ratio and the limits of clipping.
Schulman et al. · arXiv:1707.06347
DeepSeek-R1: rewards, reasoning and what the evidence establishes
Study R1-Zero and R1 separately, then build a controlled test of reasoning improvement. This analysis is pinned to arXiv v1.
DeepSeek-AI · arXiv:2501.12948