Token LabSearch

← RL knowledge base
RL / 6 OF 8

GRPO, verifiable rewards & reasoning

See how an outcome signal can reinforce successful solution behavior.

Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →
  • Understand: state assumptions, shapes/units and a counterexample.
  • Implement: reproduce a numerical example and inspect boundaries.
  • Investigate: compare one alternative under a matched budget.

Understand the mechanism

For a prompt, sample a group of responses and compare their rewards. Group-relative normalization can form advantage estimates without a separate learned value model in the discussed setup. Verifiable rewards, such as executable tests or exact mathematical checks, can provide scalable feedback, but only within what the verifier actually tests. Distinguish R1-Zero’s setup from R1’s multistage training.

Text / 原始公式
Âᵢ=(rᵢ−mean(r_group))/(std(r_group)+ε)
Training reward ≠ independent held-out capability
Equal rewards → zero centered signal

Mechanism map & worked derivation

Sample responses for one prompt → Verify and score outcomes → Normalize within group → Update; evaluate held-out tasks
Original schematic; read the steps from top to bottom. SVG ↓
Symbols and assumptions
SymbolMeaning
rᵢReward of response i in the sampled group
μ, σGroup mean and stated standard-deviation convention
ÂᵢRelative normalized signal, not an absolute quality score
  1. For [0,0,1,1], μ and population σ Centering and scaling give [−1,−1,1,1], ignoring the tiny stabilizer.
  2. Changing the last reward to 2 gives μ and σ≈0.829156. Relative advantages become approximately [−0.904534,−0.904534,0.301511,1.507557].
  3. If every reward equals 1, centering makes every numerator zero. Adding ε prevents division by zero but does not create learning contrast. Audit reward saturation and group diversity.
Experiment checkpoint

Document verifier behavior and false positives. More reasoning tokens or reward are not independent proof of better reasoning.

LOCAL / SYNTHETIC / INTERACTIVE

Change one variable. Explain what moves.

Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.

Original exercise & analysis

Normalize rewards [0,0,1,1]: advantages are approximately [−1,−1,1,1]. Then use [1,1,1,1]; no within-group contrast remains. Build a verifier adversary that passes formatting checks but returns a wrong answer, and add an independent correctness check.

Longer reasoning traces are not a capability metric. Control test-time tokens, sampling count and verifier leakage before attributing gains to training.

Does RL create new reasoning skills or elicit existing ones?

A benchmark gain alone cannot decide. Compare matched-compute base and trained models, analyze coverage of successful trajectories and test genuinely held-out tasks. State the narrower claim supported by evidence.

Study evidence & continue

DeepSeek-R1, pinned v1 ↗
Detailed paper analysis →Interview questions & answers →
MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

Group normalization uses population variance (divide by G). Equal rewards remove the centered relative signal, not necessarily all gradients of an objective containing KL. This is not a reproduction of the full GRPO training algorithm.

Formula conditions, units and worked arithmetic reviewed. The implementation isolates one numerical mechanism; it does not train a policy or control real hardware.

Before running: predict a boundary case

These exact values omit epsilon. Equal rewards require an explicit zero-variance branch or stabilizer.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node grpo-verifiable.mjs. No third-party runtime dependencies.

const solve=()=>{
const r=[0,0,1,1], mean=r.reduce((a,b)=>a+b)/r.length;
const sd=Math.sqrt(r.reduce((s,x)=>s+(x-mean)**2,0)/r.length);
return r.map(x=>(x-mean)/sd);
};
const actual=solve();
const expected=[-1,-1,1,1];
if(JSON.stringify(actual)===JSON.stringify(expected)){console.log('PASS',actual)}else{
const a=[actual].flat(), b=[expected].flat();
if(a.length!==b.length||a.some((v,i)=>!Number.isFinite(v)||Math.abs(v-b[i])>1e-10))throw Error('Numerical mismatch');
console.log('PASS within 1e-10',actual);
}
Download this example
Reference output — reveal after predicting
[
  -1,
  -1,
  1,
  1
]

Test and transfer

These exact values omit epsilon. Equal rewards require an explicit zero-variance branch or stabilizer.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope