Token LabSearch

← RL knowledge base
RL / 3 OF 8

Policy gradients, baselines & actor–critic

Understand how rewards change action probabilities.

Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →
  • Understand: state assumptions, shapes/units and a counterexample.
  • Implement: reproduce a numerical example and inspect boundaries.
  • Investigate: compare one alternative under a matched budget.

Understand the mechanism

Policy gradients increase the log probability of actions associated with better-than-baseline outcomes. A state-dependent baseline that does not depend on the sampled action can reduce variance without changing the expected score-function gradient under standard assumptions. Actor–critic methods learn a value estimate to build that baseline, introducing estimation and bootstrapping considerations.

Text / 原始公式
∇J = E[∇log πθ(a|s) A(s,a)]
δₜ=rₜ₊₁+γ(1−dₜ)V(sₜ₊₁)−V(sₜ)
GAE: Âₜ=Σ(l=0..T−t−1) (γλ)^l [Π(j=0..l−1)(1−d_(t+j))] δ_(t+l)

Mechanism map & worked derivation

Sample action from policy → Observe reward or return → Subtract valid baseline → Update log-probability weights
Original schematic; read the steps from top to bottom. SVG ↓
Symbols and assumptions
SymbolMeaning
θPolicy logit in the two-action example
σ(θ)Probability of action 1
b(s)Action-independent baseline for this derivation
  1. With action rewards 0.8 and 0.2, θσθσθσθ
  2. Differentiate: σθσθ At θ, probability is 0.5, expected reward 0.5 and gradient 0.15. A small ascent step increases action-1 probability.
  3. The baseline cancels in expectation because Σₐπππ This cancellation assumes the baseline does not depend on the sampled action.
Experiment checkpoint

Compare Monte Carlo gradient averages with the exact 0.15 reference at θ Increasing samples should reduce estimation noise.

LOCAL / SYNTHETIC / INTERACTIVE

Change one variable. Explain what moves.

Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.

Original exercise & analysis

Use a two-action bandit with rewards 0.8 and 0.2 and action-1 probability σ(θ). The exact expected reward is 0.2+0.6σ(θ); its derivative is 0.6σ(θ)(1−σ(θ)). Compare a sampled gradient estimate to this analytic reference across many seeds.

An action-dependent baseline generally needs a correction; subtracting any convenient function is not automatically unbiased.

Does a negative advantage mean the action is always bad?

No. It means worse than the baseline in that context under the estimator. Absolute reward may still be positive.

Study evidence & continue

PPO paper: policy-gradient background ↗
Interview questions & answers →
MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

Use δ__(t+1)+γ(1−d_t)V(s_(t+1))−V(s_t), with d marking true termination, not rollout truncation. GAE is a finite masked sum through T−1. Baselines are action-independent; stop gradients through advantages. State the discounted occupancy normalization when deriving ∇J.

Formula conditions, units and worked arithmetic reviewed. The implementation isolates one numerical mechanism; it does not train a policy or control real hardware.

Before running: predict a boundary case

Compare analytic 0.15 to central finite differences of J at θ

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node policy-gradient.mjs. No third-party runtime dependencies.

const solve=()=>{
const theta=0, p=1/(1+Math.exp(-theta));
return [.2+.6*p, .6*p*(1-p)];
};
const actual=solve();
const expected=[0.5,0.15];
if(JSON.stringify(actual)===JSON.stringify(expected)){console.log('PASS',actual)}else{
const a=[actual].flat(), b=[expected].flat();
if(a.length!==b.length||a.some((v,i)=>!Number.isFinite(v)||Math.abs(v-b[i])>1e-10))throw Error('Numerical mismatch');
console.log('PASS within 1e-10',actual);
}
Download this example
Reference output — reveal after predicting
[
  0.5,
  0.15
]

Test and transfer

Compare analytic 0.15 to central finite differences of J at θ

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope