Token LabSearch

← RL knowledge base
RL / 5 OF 8

SFT → RLHF → DPO: different training signals

Choose the signal that matches the available feedback.

Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →
  • Understand: state assumptions, shapes/units and a counterexample.
  • Implement: reproduce a numerical example and inspect boundaries.
  • Investigate: compare one alternative under a matched budget.

Understand the mechanism

SFT learns from demonstrations of desired outputs. A common RLHF pipeline fits a reward model from preferences and optimizes a policy with a reference penalty. DPO instead uses pairwise preferences and reference log probabilities directly in its loss. DPO is preference optimization, not an online environment-interaction loop simply because it is related to an RL objective.

Text / 原始公式
SFT: −log πθ(y_demo|x)
RLHF: E[rφ(x,y)]−β KL(πθ||πref)
DPO: −log σ(βΔ_log-ratio)

Mechanism map & worked derivation

Human demonstrations → SFT → Preference pairs → reward model → Policy optimization + reference penalty → Independent behavior evaluation
Original schematic; read the steps from top to bottom. SVG ↓
Symbols and assumptions
SymbolMeaning
rφLearned reward proxy
πrefReference policy for the drift penalty
βTrade-off coefficient under the stated convention
  1. Distinguish three data products: demonstrations for SFT, comparisons for reward learning, and sampled trajectories for policy optimization. They are not interchangeable labels.
  2. For two policies on the same prompt distribution, (expected reward, expected KL)=(1,0.4) and (0.8,0.1), β gives regularized values 0.6 and 0.7. A lower raw reward can win after the drift penalty.
  3. Evaluate the selected policy with independent task labels and blinded human comparisons. Otherwise reward-model errors can become both the training target and the apparent success criterion.
Experiment checkpoint

Report who labeled the data and which prompt distribution was evaluated. A single reward score cannot represent every user objective.

Original exercise & analysis

Use the same held-out prompts to compare a base model, SFT checkpoint and preference-tuned checkpoint. Control generation settings and response length. Audit helpfulness, factuality and refusal separately; a single overall preference score can hide regressions.

Human preference is neither a perfect truth label nor a universal representation of all users.

Can RLHF improve behavior without increasing parameter count?

Yes. Optimization changes the distribution of outputs under the model. That is a change in behavior on an evaluation distribution, not evidence that every capability improved.

Study evidence & continue

InstructGPT paper ↗
Detailed paper analysis →Interview questions & answers →
MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

β>0; use natural logs and sum response-token log probabilities with a frozen reference. The sigmoid is pairwise preference probability, not response-generation probability. Minimize the loss. RLHF maximizes reward minus KL, while SFT/DPO minimize losses.

Formula conditions, units and worked arithmetic reviewed. The implementation isolates one numerical mechanism; it does not train a policy or control real hardware.

Before running: predict a boundary case

At loss is ln2; increasing preferred log odds decreases this loss.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node rlhf-dpo.mjs. No third-party runtime dependencies.

const solve=()=>{
const beta=.5, delta=2;
return Math.log1p(Math.exp(-beta*delta));
};
const actual=solve();
const expected=0.31326168751822286;
if(JSON.stringify(actual)===JSON.stringify(expected)){console.log('PASS',actual)}else{
const a=[actual].flat(), b=[expected].flat();
if(a.length!==b.length||a.some((v,i)=>!Number.isFinite(v)||Math.abs(v-b[i])>1e-10))throw Error('Numerical mismatch');
console.log('PASS within 1e-10',actual);
}
Download this example
Reference output — reveal after predicting
0.31326168751822286

Test and transfer

At loss is ln2; increasing preferred log odds decreases this loss.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope