1. Why can LoRA start with a zero update without getting stuck?
参考解答と評価観点
Make one factor zero and the other nonzero. Their product is zero, but the zero factor can receive a nonzero gradient.
LoRA ランク実験
Train low-rank factors, compare an analytical capacity bound, and inspect initialization failure.
ローカル計算と合成データです。有料 API や実モデルのベンチマークは使用しません。技術本文は英語と中国語に対応しています。この言語では英語の技術本文を表示します。
Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
01 · 理解と導出
前提知識 · Matrix multiplication, rank, singular values, gradients.
A low-rank update restricts the space of possible changes. The experiment fits a known 8×8 target with singular values [5,3,1,0.3,0,0,0,0]. The rank-r optimum has loss equal to the sum of discarded squared singular values divided by 128.
Text / 原始公式
ΔW = BA; A∈R^(r×d), B∈R^(d×r) trainable parameters = 2dr (vs d²) L = ||BA−T||²_F / (2×8²) ∇B = (BA−T)Aᵀ/64; ∇A = Bᵀ(BA−T)/64
Step-by-step derivation and interpretation
- Let
From and A+B dA, collect terms to obtain ∇ and ∇ Verify dimensions before implementation. - If
and A is random, ∇A is initially zero but ∇B can be nonzero. If both are zero, both gradients vanish. Zero initial output and zero learning signal are different conditions. - For this known diagonal target, the optimal rank-r Frobenius error discards the tail singular values. Rank 1 loss floor is (9+1+0.09)/128; rank 2 is 1.09/128. A finite-step optimizer need not attain the floor.
Common pitfall: Equal optimizer steps across ranks do not imply equal FLOPs; low matrix reconstruction loss does not establish downstream language-model quality.
02 · 理解を確認
For a 4096×4096 layer and rank 8, how many parameters are in A and B together?
導出を表示
4096×8 + 8×4096 = 65,536; 0.390625% of 4096²
03 · 対照実験を実行
実験課題
Compare rank 1,2,4 with identical seed and steps. Then initialize both factors to zero. Separate the theoretical rank limit from failure to reach that limit during optimization.
Controlled experiment protocol
- Compare ranks 1,2,4 using the same seed, learning rate and step count. Record loss floor, achieved loss and the remaining gap.
- Repeat with both matrices zero; document unchanged loss. Restore random A and repeat across seeds to inspect optimization variability.
- At dimension 4096 and rank 8, report 65,536 adapter parameters versus 16,777,216 full parameters. Keep this calculator separate from the 8×8 training task.
Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.
ヒントを表示
If both A and B are zero, both gradients are zero. Random A with zero B gives a zero initial update but a nonzero first gradient for B.
04 · 根拠を確認
実行すると結果が表示されます。
条件を統制して実行結果を比較してください。出力に入力と手法を含みます。
05 · 推論を説明
自己評価：仕組みと前提を説明し、仮説を区別できる実験を提案してください。LLM 自動採点はありません。
2. Does a better result at higher rank establish a better method?
参考解答と評価観点
Not alone. Compare compute, parameter budget, tuning budget and uncertainty; capacity and optimization can both change.
3. What does the analytical loss floor tell you?
参考解答と評価観点
It isolates representational limits for this known matrix. A remaining gap may reflect optimization, and does not predict LLM task quality.
前提と限界
Actual local matrix optimization, not fine-tuning a language model. Frozen base
参考文献
arXiv:2106.09685 · 2021
LoRA: Low-Rank Adaptation of Large Language Models
Hu et al. · Year denotes first arXiv submission
Read: Low-rank parameterization, initialization and rank investigations.
Reading checkpoint: Derive parameter count and the first update when B=0; distinguish the toy matrix experiment from language-model fine-tuning.
BibTeX
@misc{arxiv210609685,
title = {LoRA: Low-Rank Adaptation of Large Language Models},
year = {2021},
eprint = {2106.09685},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2106.09685}
}arXiv:2305.14314 · 2023
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers et al. · Year denotes first arXiv submission
Read: Frozen quantized backbone, NF4, double quantization and paged optimizers.
Reading checkpoint: Make separate memory accounts for weights, adapters, optimizer states and activations. The site does not execute QLoRA.
BibTeX
@misc{arxiv230514314,
title = {QLoRA: Efficient Finetuning of Quantized LLMs},
year = {2023},
eprint = {2305.14314},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2305.14314}
}Assumptions, implementation & acceptance
Actual local matrix optimization, not fine-tuning a language model. Frozen base
Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.
Before running: predict a boundary case
Compare rank 1,2,4 with identical seed and steps. Then initialize both factors to zero. Separate the theoretical rank limit from failure to reach that limit during optimization.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node lora-rank.mjs. No third-party runtime dependencies.
import {runPhd} from './phd-core.mjs';
const params={
"rank": 2,
"rate": 0.8,
"epochs": 1000,
"seed": 42,
"dimension": 4096,
"zeroBoth": 0
};
console.log(runPhd('lora-rank',params));Test and transfer
Compare rank 1,2,4 with identical seed and steps. Then initialize both factors to zero. Separate the theoretical rank limit from failure to reach that limit during optimization.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Download complete kit (.tar.gz) · Complete reference kit and regression checksPredict, test and keep evidence
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
Open notebook / export / clear →The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope