DeepSeek-R1: rewards, reasoning and what the evidence establishes
Study R1-Zero and R1 separately, then build a controlled test of reasoning improvement. This analysis is pinned to arXiv v1.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · DeepSeek-AI · arXiv:2501.12948v1
Original editorial analysis by Token Lab · Source checked 2026-10-25 · Paper claims and synthetic examples are separated.
01 · What the paper establishes
The v1 report presents R1-Zero trained with large-scale RL without a preliminary SFT stage, and R1 with cold-start and multistage training. It describes group-relative policy optimization and verifiable reward signals for reasoning tasks. The reported benchmark outcomes belong to that model and evaluation setup.
Primary source: paper abstract ↗Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
02 · The mechanism
Text / 原始公式
For a response group: Aᵢ≈(rᵢ−mean(r))/(std(r)+ε) Group-relative signal → policy update Independent held-out evaluation → capability claim
A group creates a relative comparison among sampled solutions for the same prompt. Higher-reward trajectories receive a favorable relative signal. This is not equivalent to proving that every generated reasoning statement is correct. A verifier can evaluate an outcome while missing reasoning flaws, exploitation or contamination. Distillation is a separate transfer step, not the same training process as RL on the student.
Notation and intuition are teaching explanations. Consult the original method for complete assumptions and implementation details.
Mechanism map & worked derivation
| Symbol | Meaning |
|---|---|
| rᵢ | Reward of response i in the sampled group |
| μ, σ | Group mean and stated standard-deviation convention |
| Âᵢ | Relative normalized signal, not an absolute quality score |
- For [0,0,1,1],
and population Centering and scaling give [−1,−1,1,1], ignoring the tiny stabilizer. - Changing the last reward to 2 gives
and ≈0.829156. Relative advantages become approximately [−0.904534,−0.904534,0.301511,1.507557]. - If every reward equals 1, centering makes every numerator zero. Adding ε prevents division by zero but does not create learning contrast. Audit reward saturation and group diversity.
Document verifier behavior and false positives. More reasoning tokens or reward are not independent proof of better reasoning.
Change one variable. Explain what moves.
Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.
04 · Work through an example
Original group: rewards [0,0,1,1] have mean 0.5 and population standard deviation 0.5, yielding advantages approximately [−1,−1,1,1]. Changing the final reward changes every group member’s normalized value. If every reward is equal, centered advantages are zero. The visualization teaches normalization only; it does not implement GRPO training or reproduce R1.
05 · Design your experiment
Choose a small verifiable task with uncontaminated held-out problems. Compare training recipes at matched rollout and optimization budgets, then match inference tokens and sampling counts. Audit verifier false positives manually, preserve generated trajectories and measure pass@1 separately from best-of-k. Add a distribution-shift test and report failures, not only aggregate gains.
State one hypothesis; save the configuration and raw results; identify a competing explanation; report the narrowest supported conclusion. Use the linked lab as a local starting point, not a full paper reproduction.
Open related lesson / lab →06 · Limits & reviewer questions
Editorial caution: improved benchmark performance alone cannot settle whether RL discovers new capabilities or better elicits existing ones. Long answers and reflection-like wording are not sufficient evidence. Stronger causal claims need controlled baselines and analyses of successful-solution coverage.
Why distinguish pass@1 from best-of-k?
Best-of-k changes inference compute and often adds selection or verification. A gain can come from more attempts rather than a better single-sample policy, so report both protocols explicitly.
07 · Read the original
v1 Sections 2.2–2.4 and evaluation. Keep R1-Zero, R1 and distilled-model claims separate.
Year is the first arXiv submission year. Unversioned links follow revisions; record the exact version in your reproduction report. DeepSeek-R1 here is explicitly pinned to v1.
Continue this topic
PPO: inspect the clipped objective one sample at a time
Understand the sign of advantage, the rollout-policy ratio and the limits of clipping.
Schulman et al. · arXiv:1707.06347
InstructGPT: change the training signal to change behavior
Connect demonstrations, preference labels, reward learning and policy optimization.
Ouyang et al. · arXiv:2203.02155