1. Why check before clipping?
Show answer & scoring criteria
The numerical derivative is of the loss; clipping transforms the update and no longer equals that derivative.
Gradient Debugger
Check gradients numerically, inject a reduction bug, and distinguish instability from a wrong derivative.
Local calculations and synthetic examples. No paid API or live-model benchmark.Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
01 · Understand & derive
Prerequisites · Chain rule, mean squared error, vector norms.
Finite differences test whether your derivative matches the declared objective. Compare raw gradients before clipping; clipping intentionally changes the update. A sum gradient is 32 times the mean gradient on this 32-row fixture.
Text / 原始公式
L = Σ(wxᵢ+b−yᵢ)²/(2n) ∂L/∂w = mean[(wxᵢ+b−yᵢ)xᵢ] g_FD = [L(θ+ε)−L(θ−ε)]/(2ε) relative error = ||g−g_FD|| / max(10⁻¹², ||g||+||g_FD||)
Step-by-step derivation and interpretation
- Start with
Differentiating rᵢ²/2 gives rᵢ; multiply by ∂rᵢ/∂ , then average over the batch. Keep bias and weight gradients separate. - For a quadratic objective, full-batch gradient descent is stable along each positive-curvature direction when 0<η<2/λ. Use the largest Hessian eigenvalue to reason about the step-size limit; data scaling changes that eigenvalue.
- Finite differences balance truncation and floating-point cancellation. Check several parameter points and ε values; do not use a single passing point as a proof. Compare gradients before clipping.
Common pitfall: Decreasing loss can coexist with a wrong reduction. Do not equate numerical stability with correct implementation.
02 · Check your understanding
At
Show derivation
mean[(-2)×1,(-4)×2] = -5
03 · Run a controlled experiment
Experiment assignment
Run the baseline; enable the reduction bug; then disable it and set input scale to 20. Find a learning rate that restores stable training. Compare exports with one variable changed.
Controlled experiment protocol
- Baseline: rate 0.1, scale 1, bug off, 100 steps, seed 42. Export loss and relative gradient error.
- Change only bug to sum gradient: expected mismatch ≈31/33. Restore it; change only scale to 20: check for instability.
- Reduce rate to 0.001 and rerun at scale 20; then test clipping independently. Submit three controlled comparisons and explain why clipping cannot fix a wrong derivative.
Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.
Show a hint
The Hessian in w scales with mean(x²). A stable learning rate at scale 1 may be too large at scale 20. Clipping can stabilize updates but cannot repair the wrong derivative.
04 · Inspect evidence
Run the experiment to see evidence.
Only compare runs with controlled inputs. Exports include inputs and methods.
05 · Defend your reasoning
Self-assessment: explain the mechanism, identify assumptions, and propose a discriminating test. No automatic LLM grading.
2. Loss decreases despite a failed gradient check. Is the implementation correct?
Show answer & scoring criteria
No. A constant gradient scaling can mimic a different learning rate. Check objective reduction, parameter paths and precision.
3. Design a minimal training-failure investigation.
Show answer & scoring criteria
Overfit a tiny batch, verify targets, run finite differences, inspect gradient norms, then vary input scale and step size independently.
Assumptions & limits
Synthetic linear regression, not a deep-network profiler. Training stops if loss exceeds 10¹². Finite-difference checks are local and do not prove every gradient is correct.
References
arXiv:1412.6980 · 2014
Adam: A Method for Stochastic Optimization
Kingma & Ba · Year denotes first arXiv submission
Read: Algorithm 1; moment estimates and bias correction.
Reading checkpoint: Explain why zero-initialized moments need correction. The gradient debugger uses SGD, not Adam.
BibTeX
@misc{arxiv14126980,
title = {Adam: A Method for Stochastic Optimization},
year = {2014},
eprint = {1412.6980},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1412.6980}
}arXiv:1711.05101 · 2017
Decoupled Weight Decay Regularization
Loshchilov & Hutter · Year denotes first arXiv submission
Read: Comparison of L2 regularization and decoupled decay under adaptive updates.
Reading checkpoint: Write the two updates side by side; identify where the preconditioner acts. This is a reading extension, not an implemented AdamW benchmark.
BibTeX
@misc{arxiv171105101,
title = {Decoupled Weight Decay Regularization},
year = {2017},
eprint = {1711.05101},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1711.05101}
}Assumptions, implementation & acceptance
Synthetic linear regression, not a deep-network profiler. Training stops if loss exceeds 10¹². Finite-difference checks are local and do not prove every gradient is correct.
Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.
Before running: predict a boundary case
Run the baseline; enable the reduction bug; then disable it and set input scale to 20. Find a learning rate that restores stable training. Compare exports with one variable changed.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node gradient-debugger.mjs. No third-party runtime dependencies.
import {runPhd} from './phd-core.mjs';
const params={
"rate": 0.1,
"scale": 1,
"clip": 0,
"bug": 0,
"epochs": 100,
"seed": 42
};
console.log(runPhd('gradient-debugger',params));Test and transfer
Run the baseline; enable the reduction bug; then disable it and set input scale to 20. Find a learning rate that restores stable training. Compare exports with one variable changed.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Download complete kit (.tar.gz) · Complete reference kit and regression checksPredict, test and keep evidence
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
Open notebook / export / clear →The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope