1. Why can LoRA start with a zero update without getting stuck?
Show answer & scoring criteria
Make one factor zero and the other nonzero. Their product is zero, but the zero factor can receive a nonzero gradient.
LoRA Rank Lab
Train low-rank factors, compare an analytical capacity bound, and inspect initialization failure.
Local calculations and synthetic examples. No paid API or live-model benchmark.Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
01 · Understand & derive
Prerequisites · Matrix multiplication, rank, singular values, gradients.
A low-rank update restricts the space of possible changes. The experiment fits a known 8×8 target with singular values [5,3,1,0.3,0,0,0,0]. The rank-r optimum has loss equal to the sum of discarded squared singular values divided by 128.
Text / 原始公式
ΔW = BA; A∈R^(r×d), B∈R^(d×r) trainable parameters = 2dr (vs d²) L = ||BA−T||²_F / (2×8²) ∇B = (BA−T)Aᵀ/64; ∇A = Bᵀ(BA−T)/64
Step-by-step derivation and interpretation
- Let
From and A+B dA, collect terms to obtain ∇ and ∇ Verify dimensions before implementation. - If
and A is random, ∇A is initially zero but ∇B can be nonzero. If both are zero, both gradients vanish. Zero initial output and zero learning signal are different conditions. - For this known diagonal target, the optimal rank-r Frobenius error discards the tail singular values. Rank 1 loss floor is (9+1+0.09)/128; rank 2 is 1.09/128. A finite-step optimizer need not attain the floor.
Common pitfall: Equal optimizer steps across ranks do not imply equal FLOPs; low matrix reconstruction loss does not establish downstream language-model quality.
02 · Check your understanding
For a 4096×4096 layer and rank 8, how many parameters are in A and B together?
Show derivation
4096×8 + 8×4096 = 65,536; 0.390625% of 4096²
03 · Run a controlled experiment
Experiment assignment
Compare rank 1,2,4 with identical seed and steps. Then initialize both factors to zero. Separate the theoretical rank limit from failure to reach that limit during optimization.
Controlled experiment protocol
- Compare ranks 1,2,4 using the same seed, learning rate and step count. Record loss floor, achieved loss and the remaining gap.
- Repeat with both matrices zero; document unchanged loss. Restore random A and repeat across seeds to inspect optimization variability.
- At dimension 4096 and rank 8, report 65,536 adapter parameters versus 16,777,216 full parameters. Keep this calculator separate from the 8×8 training task.
Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.
Show a hint
If both A and B are zero, both gradients are zero. Random A with zero B gives a zero initial update but a nonzero first gradient for B.
04 · Inspect evidence
Run the experiment to see evidence.
Only compare runs with controlled inputs. Exports include inputs and methods.
05 · Defend your reasoning
Self-assessment: explain the mechanism, identify assumptions, and propose a discriminating test. No automatic LLM grading.
2. Does a better result at higher rank establish a better method?
Show answer & scoring criteria
Not alone. Compare compute, parameter budget, tuning budget and uncertainty; capacity and optimization can both change.
3. What does the analytical loss floor tell you?
Show answer & scoring criteria
It isolates representational limits for this known matrix. A remaining gap may reflect optimization, and does not predict LLM task quality.
Assumptions & limits
Actual local matrix optimization, not fine-tuning a language model. Frozen base
References
arXiv:2106.09685 · 2021
LoRA: Low-Rank Adaptation of Large Language Models
Hu et al. · Year denotes first arXiv submission
Read: Low-rank parameterization, initialization and rank investigations.
Reading checkpoint: Derive parameter count and the first update when B=0; distinguish the toy matrix experiment from language-model fine-tuning.
BibTeX
@misc{arxiv210609685,
title = {LoRA: Low-Rank Adaptation of Large Language Models},
year = {2021},
eprint = {2106.09685},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2106.09685}
}arXiv:2305.14314 · 2023
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers et al. · Year denotes first arXiv submission
Read: Frozen quantized backbone, NF4, double quantization and paged optimizers.
Reading checkpoint: Make separate memory accounts for weights, adapters, optimizer states and activations. The site does not execute QLoRA.
BibTeX
@misc{arxiv230514314,
title = {QLoRA: Efficient Finetuning of Quantized LLMs},
year = {2023},
eprint = {2305.14314},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2305.14314}
}Assumptions, implementation & acceptance
Actual local matrix optimization, not fine-tuning a language model. Frozen base
Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.
Before running: predict a boundary case
Compare rank 1,2,4 with identical seed and steps. Then initialize both factors to zero. Separate the theoretical rank limit from failure to reach that limit during optimization.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node lora-rank.mjs. No third-party runtime dependencies.
import {runPhd} from './phd-core.mjs';
const params={
"rank": 2,
"rate": 0.8,
"epochs": 1000,
"seed": 42,
"dimension": 4096,
"zeroBoth": 0
};
console.log(runPhd('lora-rank',params));Test and transfer
Compare rank 1,2,4 with identical seed and steps. Then initialize both factors to zero. Separate the theoretical rank limit from failure to reach that limit during optimization.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Download complete kit (.tar.gz) · Complete reference kit and regression checksPredict, test and keep evidence
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
Open notebook / export / clear →The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope