Token Lab検索

← 強化学習ライブラリ
RL / 4 OF 8

PPO, clipping & on-policy data

Limit an update incentive without assuming a hard trust region.

Detailed technical articles are available in English and Chinese. This locale shows the English article with localized navigation.

Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →
  • Understand: state assumptions, shapes/units and a counterexample.
  • Implement: reproduce a numerical example and inspect boundaries.
  • Investigate: compare one alternative under a matched budget.

Understand the mechanism

PPO compares current and rollout-policy action probabilities. Its clipped objective limits certain improvements in a surrogate objective. For positive advantage the upper side clips; for negative advantage the lower side matters. The old policy is a frozen reference for the collected batch, not necessarily the original pretrained model. Reusing increasingly stale trajectories changes the validity of the approximation.

Text / 原始公式
ρ=πθ(a|s)/πold(a|s)
Lclip=min(ρÂ, clip(ρ,1−ε,1+ε)Â)

Mechanism map & worked derivation

Collect policy rollouts → Freeze old scores; estimate advantage → Optimize clipped surrogate → Measure KL and refresh data
Original schematic; read the steps from top to bottom. SVG ↓
Symbols and assumptions
SymbolMeaning
ρCurrent / rollout-policy sampled action probability
ÂEstimated advantage for a sampled action
εClipping threshold, not a guaranteed KL limit
  1. Freeze old log probabilities for a rollout batch. Compute the ratio as exp(logπθ−logπold) to avoid division of tiny probabilities.
  2. At ε and positive advantage, a ratio above 1.2 receives no additional surrogate incentive. At negative advantage, a ratio below 0.8 is the analogous clipped side.
  3. For ρ and Â=−2, raw objective is −1, clipped candidate −1.6, and the minimum is −1.6. The sign of advantage determines which branch is pessimistic.
Experiment checkpoint

Test both signs and clipping boundaries. If all updates clip immediately, inspect learning rate, advantage scale and rollout staleness.

LOCAL / SYNTHETIC / INTERACTIVE

Change one variable. Explain what moves.

Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.

Original exercise & analysis

At ε and ρ, compare Â=+1 and Â=−1. The objectives are 1.2 and −1.5. Then try ρ: they are 0.5 and −0.8. Explain each branch before opening the answer.

Clipping does not guarantee a fixed KL bound or monotonically improved task reward. Monitor both.

What should be logged besides reward?

Approximate KL, entropy, clipping fraction, advantage distribution, value loss, rollout length and held-out task metrics.

Study evidence & continue

Proximal Policy Optimization Algorithms ↗
Detailed paper analysis →Interview questions & answers →
MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

Maximize the clipped surrogate, or minimize its negative. Freeze old log probabilities and advantages during each policy update. The sampled old-policy probability must be nonzero; clipping is not a hard KL constraint.

Formula conditions, units and worked arithmetic reviewed. The implementation isolates one numerical mechanism; it does not train a policy or control real hardware.

Before running: predict a boundary case

Negative advantages require min after multiplying, not clipping the final objective.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node ppo-updates.mjs. No third-party runtime dependencies.

const solve=()=>{
const ratio=1.5, clipped=Math.max(.8,Math.min(1.2,ratio));
return [Math.min(ratio,clipped),Math.min(-ratio,-clipped)];
};
const actual=solve();
const expected=[1.2,-1.5];
if(JSON.stringify(actual)===JSON.stringify(expected)){console.log('PASS',actual)}else{
const a=[actual].flat(), b=[expected].flat();
if(a.length!==b.length||a.some((v,i)=>!Number.isFinite(v)||Math.abs(v-b[i])>1e-10))throw Error('Numerical mismatch');
console.log('PASS within 1e-10',actual);
}
Download this example
Reference output — reveal after predicting
[
  1.2,
  -1.5
]

Test and transfer

Negative advantages require min after multiplying, not clipping the final objective.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope