Token Lab검색

← 논문 라이브러리
Efficient adaptation & alignment / 2023

DPO: preference learning through relative log probabilities

Derive the pairwise margin and see what β changes in a single preference example.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model · Rafailov et al. · arXiv:2305.18290

Original editorial analysis by Token Lab · Source checked 2026-10-25 · Paper claims and synthetic examples are separated.

Detailed technical articles are available in English and Chinese. This locale shows the English article with localized navigation.

01 · What the paper establishes

DPO reparameterizes a KL-regularized reward objective into a preference classification loss using a reference policy. The paper evaluates sentiment, summarization and dialogue tasks. Its objective avoids the explicit reward-model-plus-online-RL loop used in the compared setup.

Primary source: paper abstract ↗
Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →
  • Understand: state assumptions, shapes/units and a counterexample.
  • Implement: reproduce a numerical example and inspect boundaries.
  • Investigate: compare one alternative under a matched budget.

02 · The mechanism

Text / 原始公式
Δ = log πθ(yw|x) − log πθ(yl|x)
    − log πref(yw|x) + log πref(yl|x)
L = −log σ(βΔ)
∂L/∂Δ = −βσ(−βΔ)

The margin compares how much the chosen response gains relative to the rejected one, after subtracting the reference policy’s preference. It is not merely the chosen response probability. Sequence log probabilities require consistent tokenization, masking and summation. A KL penalty in the derivation motivates staying near the reference, but a finite dataset and finite optimization do not guarantee an arbitrary hard KL ceiling.

Notation and intuition are teaching explanations. Consult the original method for complete assumptions and implementation details.

Mechanism map & worked derivation

Prompt and preference pair → Policy + reference log scores → Reference-adjusted margin Δ → Preference loss → update
Original schematic; read the steps from top to bottom. SVG ↓
Symbols and assumptions
SymbolMeaning
yw, ylPreferred and rejected complete responses
πrefFrozen reference policy
ΔPolicy preference log-odds minus reference log-odds
  1. Compute log probabilities only for the response tokens under a consistent masking convention. Prompt-token probabilities should not accidentally alter a pairwise response comparison.
  2. Let policy scores be −2 and −3, reference scores −2.5 and −3.2. The policy gap is 1, the reference gap 0.7, so At β, β and
  3. Differentiate −logσ(βΔ) to obtain −βσ(−βΔ), approximately −0.708687 here. Swapping the response labels changes Δ to −0.3 and increases the loss to about 1.037488.
Experiment checkpoint

Assert identical-policy 2. Test label swapping and padding masks. A lower pair loss does not establish factual correctness.

LOCAL / SYNTHETIC / INTERACTIVE

Change one variable. Explain what moves.

Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.

04 · Work through an example

Original example: the policy log-odds are 0.8 and reference log-odds are 0.3, so At β, predicted preference probability is about 0.622 and loss is 0.474. At the loss is log 2. The slider illustrates the loss on a fixed margin, not how an entire trained policy will respond to a β sweep.

05 · Design your experiment

Create three synthetic preference pairs and manually compute the loss before training anything. Test that swapping chosen and rejected negates Δ. For a real POC, freeze the reference checkpoint, hold out prompts rather than individual responses, and compare against the supervised baseline. Track held-out preference accuracy, generation quality, length and estimated KL drift.

Research deliverable

State one hypothesis; save the configuration and raw results; identify a competing explanation; report the narrowest supported conclusion. Use the linked lab as a local starting point, not a full paper reproduction.

Open related lesson / lab →

06 · Limits & reviewer questions

Editorial caution: preference labels can encode style bias, inconsistency or distribution gaps. A decreasing training loss does not prove better factuality, safety or user utility. Sequence length can confound comparisons, so document the exact scoring convention instead of silently normalizing scores.

If policy and reference are identical, what is the loss for every pair?

Both log-ratio differences cancel, so and 2 regardless of β. The gradient with respect to Δ is −β/2, so optimization can still begin.

07 · Read the original

The KL-regularized objective, reward reparameterization and DPO loss derivation; check reference-policy terms.

arXiv ↗PDF ↗

Year is the first arXiv submission year. Unversioned links follow revisions; record the exact version in your reproduction report. DeepSeek-R1 here is explicitly pinned to v1.

Continue this topic

MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

β>0; use natural logs and sum response-token log probabilities with a frozen reference. The sigmoid is pairwise preference probability, not response-generation probability. Minimize the loss.

Displayed formula, bilingual explanation and declared slider range. This code reproduces the scalar teaching calculation, not the paper training system.

Before running: predict a boundary case

Create three synthetic preference pairs and manually compute the loss before training anything. Test that swapping chosen and rejected negates Δ. For a real POC, freeze the reference checkpoint, hold out prompts rather than individual responses, and compare against the supervised baseline. Track held-out preference accuracy, generation quality, length and estimated KL drift.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node dpo.mjs. No third-party runtime dependencies.

import {paperCalc} from './paper-core.mjs';
const result=paperCalc("dpo",1);
console.log(result);
Download this examplepaper-core.mjs
Reference output — reveal after predicting
{
  "labels": [
    "Pairwise preference probability",
    "Loss (nats)"
  ],
  "values": [
    0.6224593312018546,
    0.4740769841801067
  ],
  "unit": "value"
}

Test and transfer

Create three synthetic preference pairs and manually compute the loss before training anything. Test that swapping chosen and rejected negates Δ. For a real POC, freeze the reference checkpoint, hold out prompts rather than individual responses, and compare against the supervised baseline. Track held-out preference accuracy, generation quality, length and estimated KL drift.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope