Token Lab검색

← 논문 라이브러리
Optimization & training / 2017

AdamW: where you apply weight decay matters

Separate a loss penalty from parameter shrinkage, then test the difference with an intentionally unequal preconditioner.

Decoupled Weight Decay Regularization · Loshchilov & Hutter · arXiv:1711.05101

Original editorial analysis by Token Lab · Source checked 2026-10-25 · Paper claims and synthetic examples are separated.

Detailed technical articles are available in English and Chinese. This locale shows the English article with localized navigation.

01 · What the paper establishes

The paper distinguishes L2 regularization from decoupled weight decay for adaptive optimizers and evaluates the modification on image-classification tasks. It argues that the two operations should not share a name when their updates differ.

Primary source: paper abstract ↗
Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →
  • Understand: state assumptions, shapes/units and a counterexample.
  • Implement: reproduce a numerical example and inspect boundaries.
  • Investigate: compare one alternative under a matched budget.

02 · The mechanism

Text / 原始公式
Fixed diagonal preconditioner D:
L2: θ′ = θ − αD(g + λθ)
Decoupled: θ′ = (1−αλ)θ − αDg
Difference in shrinkage: −αλ(D−I)θ

The key question is whether the regularizer passes through the gradient scaling machinery. In the first equation it does; in the second it does not. For plain SGD with these displayed updates agree. For adaptive methods, the regularization term can also alter future moment estimates. The fixed-D view is therefore a useful local comparison, not a complete simulation of Adam with L2.

Notation and intuition are teaching explanations. Consult the original method for complete assumptions and implementation details.

Mechanism map & worked derivation

Task gradient → Adaptive gradient step → Separate weight shrinkage → Audit norms and validation
Original schematic; read the steps from top to bottom. SVG ↓
Symbols and assumptions
SymbolMeaning
DFixed diagonal scaling for the local comparison
λRegularization or decay strength; conventions differ
Πₜ(1−αλ)Accumulated decoupled shrinkage without task gradients
  1. For a penalty λ||θ||²/2, differentiating gives λθ. Applying a preconditioner to the full gradient also scales this penalty.
  2. Subtracting the decoupled update from the fixed-D L2 update leaves −αλ(D−I)θ. Equality holds when this expression vanishes, not merely when both options are called weight decay.
  3. With αλ and no task gradient, ten decoupled steps retain 0.99¹⁰≈0.904382 of the initial weight. A different step schedule changes this product.
Experiment checkpoint

Use a two-coordinate example before a neural model. Then check optimizer source code for decay placement and excluded parameter groups.

LOCAL / SYNTHETIC / INTERACTIVE

Change one variable. Explain what moves.

Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.

04 · Work through an example

Original example: θ,1], ,0], α, λ and ,10). L2 gives [0.99,0.90], while decoupled decay gives [0.99,0.99]. The same starting parameter magnitudes shrink differently only because the first update applies coordinate scaling. Our slider changes the second diagonal entry while keeping the penalty and learning rate fixed.

05 · Design your experiment

First verify equality for with a unit test. Then compare actual Adam+L2 and AdamW on a small classifier, keeping data order and initialization paired. Search learning rate and decay jointly with equal budgets. Record parameter norms, train/validation loss and excluded parameter groups. State whether biases and normalization parameters receive decay.

Research deliverable

State one hypothesis; save the configuration and raw results; identify a competing explanation; report the narrowest supported conclusion. Use the linked lab as a local starting point, not a full paper reproduction.

Open related lesson / lab →

06 · Limits & reviewer questions

Editorial caution: “decoupled” describes the update mechanism; it does not mean learning rate and decay never interact in an experiment. Accumulated shrinkage depends on the schedule and number of steps. A library option named weight_decay must be checked against its actual optimizer implementation.

Why can equal decay coefficients be an unfair comparison across different training lengths?

With zero task gradients, repeated decoupled updates multiply θ by ∏ₜ(1−αₜλ). Different step counts or schedules change the total shrinkage even when λ is identical.

07 · Read the original

The L2 versus weight-decay comparison and optimizer algorithms; then inspect the tuning experiments.

arXiv ↗PDF ↗

Year is the first arXiv submission year. Unversioned links follow revisions; record the exact version in your reproduction report. DeepSeek-R1 here is explicitly pinned to v1.

Continue this topic

MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

The decay-only product assumes zero adaptive update, including zero historical moment state. Zero current gradients alone are insufficient. Shrinkage assumes 0≤αλ≤1.

Displayed formula, bilingual explanation and declared slider range. This code reproduces the scalar teaching calculation, not the paper training system.

Before running: predict a boundary case

First verify equality for with a unit test. Then compare actual Adam+L2 and AdamW on a small classifier, keeping data order and initialization paired. Search learning rate and decay jointly with equal budgets. Record parameter norms, train/validation loss and excluded parameter groups. State whether biases and normalization parameters receive decay.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node adamw.mjs. No third-party runtime dependencies.

import {paperCalc} from './paper-core.mjs';
const result=paperCalc("adamw",10);
console.log(result);
Download this examplepaper-core.mjs
Reference output — reveal after predicting
{
  "labels": [
    "L2 coordinate 1",
    "L2 coordinate 2",
    "Decoupled coordinate 2"
  ],
  "values": [
    0.99,
    0.9,
    0.99
  ],
  "unit": "remaining θ"
}

Test and transfer

First verify equality for with a unit test. Then compare actual Adam+L2 and AdamW on a small classifier, keeping data order and initialization paired. Search learning rate and decay jointly with equal budgets. Record parameter norms, train/validation loss and excluded parameter groups. State whether biases and normalization parameters receive decay.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope