PPO: inspect the clipped objective one sample at a time
Understand the sign of advantage, the rollout-policy ratio and the limits of clipping.
Proximal Policy Optimization Algorithms · Schulman et al. · arXiv:1707.06347
Original editorial analysis by Token Lab · Source checked 2026-10-25 · Paper claims and synthetic examples are separated.
01 · What the paper establishes
PPO alternates trajectory collection with minibatch optimization of a surrogate objective. The paper compares variants and evaluates control and Atari tasks. Clipping is a practical optimization device, not a theorem that every policy update improves reward.
Primary source: paper abstract ↗Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
02 · The mechanism
Text / 原始公式
ρₜ(θ)=πθ(aₜ|sₜ)/πold(aₜ|sₜ) Lclip=E[min(ρₜÂₜ, clip(ρₜ,1−ε,1+ε)Âₜ)]
For positive advantage, increasing the sampled action probability is desirable, but the surrogate stops rewarding that increase beyond the upper clipping threshold. For negative advantage, the opposite branch matters. The old log probabilities must correspond to the policy that generated the rollout. Treating them as constantly changing during optimization would change the objective.
Notation and intuition are teaching explanations. Consult the original method for complete assumptions and implementation details.
Mechanism map & worked derivation
| Symbol | Meaning |
|---|---|
| ρ | Current / rollout-policy sampled action probability |
| Â | Estimated advantage for a sampled action |
| ε | Clipping threshold, not a guaranteed KL limit |
- Freeze old log probabilities for a rollout batch. Compute the ratio as exp(logπθ−logπold) to avoid division of tiny probabilities.
- At
and positive advantage, a ratio above 1.2 receives no additional surrogate incentive. At negative advantage, a ratio below 0.8 is the analogous clipped side. - For
and Â=−2, raw objective is −1, clipped candidate −1.6, and the minimum is −1.6. The sign of advantage determines which branch is pessimistic.
Test both signs and clipping boundaries. If all updates clip immediately, inspect learning rate, advantage scale and rollout staleness.
Change one variable. Explain what moves.
Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.
04 · Work through an example
Original example:
05 · Design your experiment
Unit-test both advantage signs and both clipping boundaries. For a small environment POC, save rollout log probabilities and terminal flags, calculate advantages with a fixed convention and compare to a no-clipping baseline. Keep environment interactions and evaluation episodes matched. Log reward, entropy, approximate KL and clipping fraction for every update.
State one hypothesis; save the configuration and raw results; identify a competing explanation; report the narrowest supported conclusion. Use the linked lab as a local starting point, not a full paper reproduction.
Open related lesson / lab →06 · Limits & reviewer questions
Editorial caution: several epochs on one rollout batch do not make the data indefinitely reusable. Large updates, value-function errors and stale data can still destabilize learning. A flat clipped term for one sample does not mean the entire neural network receives zero gradient from every sample.
Is the clipping parameter a guaranteed KL limit?
No. It shapes the surrogate incentive on sampled actions. Distribution-wide KL depends on the resulting policy and should be measured independently.
07 · Read the original
Policy-gradient background, clipped surrogate and implementation details.
Year is the first arXiv submission year. Unversioned links follow revisions; record the exact version in your reproduction report. DeepSeek-R1 here is explicitly pinned to v1.
Continue this topic
InstructGPT: change the training signal to change behavior
Connect demonstrations, preference labels, reward learning and policy optimization.
Ouyang et al. · arXiv:2203.02155
DeepSeek-R1: rewards, reasoning and what the evidence establishes
Study R1-Zero and R1 separately, then build a controlled test of reasoning improvement. This analysis is pinned to arXiv v1.
DeepSeek-AI · arXiv:2501.12948