SFT → RLHF → DPO: different training signals

Assumptions, implementation & acceptance

β>0; use natural logs and sum response-token log probabilities with a frozen reference. The sigmoid is pairwise preference probability, not response-generation probability. Minimize the loss. RLHF maximizes reward minus KL, while SFT/DPO minimize losses.

Formula conditions, units and worked arithmetic reviewed. The implementation isolates one numerical mechanism; it does not train a policy or control real hardware.

Before running: predict a boundary case

At loss is ln2; increasing preferred log odds decreases this loss.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node rlhf-dpo.mjs. No third-party runtime dependencies.

const solve=()=>{
const beta=.5, delta=2;
return Math.log1p(Math.exp(-beta*delta));
};
const actual=solve();
const expected=0.31326168751822286;
if(JSON.stringify(actual)===JSON.stringify(expected)){console.log('PASS',actual)}else{
const a=[actual].flat(), b=[expected].flat();
if(a.length!==b.length||a.some((v,i)=>!Number.isFinite(v)||Math.abs(v-b[i])>1e-10))throw Error('Numerical mismatch');
console.log('PASS within 1e-10',actual);
}

0.31326168751822286

Test and transfer

At loss is ln2; increasing preferred log odds decreases this loss.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.