LoRA: constrain the update, not the whole model
Factor a weight update, count trainable parameters and explain the asymmetric first gradient.
LoRA: Low-Rank Adaptation of Large Language Models · Hu et al. · arXiv:2106.09685
Original editorial analysis by Token Lab · Source checked 2026-10-25 · Paper claims and synthetic examples are separated.
Detailed technical articles are available in English and Chinese. This locale shows the English article with localized navigation.
01 · What the paper establishes
LoRA freezes pretrained weights and learns a low-rank additive update. The paper studies several language models and adaptation tasks. Its empirical success does not establish that every useful update has the same small rank.
Primary source: paper abstract ↗Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
02 · The mechanism
Text / 原始公式
W ∈ Rᵈˣᵏ, A ∈ Rʳˣᵏ, B ∈ Rᵈˣʳ W′ = W + (α/r)BA Trainable parameters = r(k+d) ∂L/∂A = (α/r)BᵀG; ∂L/∂B = (α/r)GAᵀ
The base matrix can remain full rank; only its learned change is constrained. Two thin matrices provide a bottleneck through which the update passes. With B initialized to zero and A nonzero, the initial function is unchanged, the gradient for A is zero, and B can begin moving. Setting both to zero would block both gradients.
Notation and intuition are teaching explanations. Consult the original method for complete assumptions and implementation details.
Mechanism map & worked derivation
| Symbol | Meaning |
|---|---|
| W ∈ ℝᵈˣᵏ | Frozen base weight |
| A ∈ ℝʳˣᵏ, B ∈ ℝᵈˣʳ | Trainable low-rank factors |
| Gradient arriving at the effective weight |
- The chain rule gives
′). Substitute ·A+B·dA) and collect coefficients to obtain the two factor gradients. - Tiny example with α/
: ,2], ,0]ᵀ and Then ∂L/∂ ,2]ᵀ while ∂L/∂ ,0]. A zero initial update can still start learning through B. - With learning rate 0.1, B becomes [−0.1,−0.2]ᵀ, giving
-0.1,-0.2],[-0.2,-0.4]]. Its rank is at most one; the next step can now give A a nonzero gradient.
Assert that W stays unchanged and that the initial output matches the base model. Test gradients before testing downstream quality.
Change one variable. Explain what moves.
Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.
04 · Work through an example
Original example: a 4096×4096 matrix has 16,777,216 entries. Rank 8 adds 8×8192=65,536 trainable entries, or 0.390625% of the dense count. This is a parameter ratio, not a total training-memory ratio. A rank-r optimum for a squared-error matrix approximation leaves the squared singular-value tail as residual; neural task loss is a different objective.
05 · Design your experiment
Start with the site’s small matrix-fit experiment. Sweep rank while holding target matrix and seed fixed, then compare with the singular-value lower bound. In a real language-model study, additionally fix target modules, adapter scaling, token budget and validation selection. Compare rank increases against wider module coverage at similar trainable-parameter budgets.
State one hypothesis; save the configuration and raw results; identify a competing explanation; report the narrowest supported conclusion. Use the linked lab as a local starting point, not a full paper reproduction.
Open related lesson / lab →06 · Limits & reviewer questions
Editorial caution: fewer trainable weights do not remove backbone forward passes or activation storage. Merging BA into a compatible dense weight can remove the separate adapter path, but serving many adapters or a quantized backbone introduces other constraints. Rank alone is an incomplete experiment description.
Why is zero-initializing both factors a bad idea?
Both gradients contain the other factor. With
07 · Read the original
Low-rank parameterization and initialization, followed by rank and target-module experiments.
Year is the first arXiv submission year. Unversioned links follow revisions; record the exact version in your reproduction report. DeepSeek-R1 here is explicitly pinned to v1.
Continue this topic
QLoRA: a memory budget is more than weight precision
Separate stored weights, computation precision, adapters and training state. Build an honest memory ledger.
Dettmers et al. · arXiv:2305.14314
DPO: preference learning through relative log probabilities
Derive the pairwise margin and see what β changes in a single preference example.
Rafailov et al. · arXiv:2305.18290