Transformer: turn token compatibility into information flow
Track Q, K and V, interpret a causal attention row, and distinguish parallel training from autoregressive generation.
Attention Is All You Need · Vaswani et al. · arXiv:1706.03762
Original editorial analysis by Token Lab · Source checked 2026-10-25 · Paper claims and synthetic examples are separated.
01 · What the paper establishes
The Transformer replaces recurrence with attention-based encoder and decoder blocks. The paper evaluates translation and parsing. Scaled dot-product attention, multiple heads and positional signals are central components; the original model is not simply today’s decoder-only LLM.
Primary source: paper abstract ↗Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
02 · The mechanism
Text / 原始公式
Q=XW_Q, K=XW_K, V=XW_V S=QKᵀ/√dₖ + M A=softmax(S) row-wise O=AV; Mᵢⱼ=−∞ when j>i (causal case)
A query chooses how to mix values by comparing itself with keys. For N tokens, Q and K have N rows; their product contains N×N compatibility scores. The softmax is applied per query, so each allowed row sums to one. Scaling by √dₖ controls score magnitude under a simple independent-coordinate variance argument. Positional information is necessary because content-only matching does not encode sequence order.
Notation and intuition are teaching explanations. Consult the original method for complete assumptions and implementation details.
Mechanism map & worked derivation
| Symbol | Meaning |
|---|---|
| Q, K ∈ R^(N × d_k) | Queries and keys: one row per token |
| V ∈ R^(N × d_v) | Values to be mixed |
| A ∈ ℝᴺˣᴺ | Row-normalized mixing weights |
- Check dimensions before numbers: QKᵀ is N×N; multiplying by V returns N×dᵥ. A matrix shape test catches many transposition mistakes.
- For logits [2,1,0], subtract the maximum to get [0,−1,−2]. Exponentials are [1,0.367879,0.135335], whose sum is 1.503214. Division yields [0.665241,0.244728,0.090031].
- The weighted sum for values [1,3,5] is 0.665241+0.734185+0.450153≈1.849579. Mask forbidden logits before normalization; masking weights afterward without renormalization changes the operation.
Check causal zeros and future-token perturbation invariance, not only row sums. Handle an all-masked row explicitly to avoid NaNs.
Change one variable. Explain what moves.
Original teaching calculation, not a reproduction of the paper’s benchmark. Bars are relative within this panel; numeric values and units are authoritative.
04 · Work through an example
Original example: focus on the third token with allowed scores [2,1,0] and values [1,3,5]. Softmax produces approximately [0.665,0.245,0.090], giving output 1.850. A fourth, future value must have exactly zero weight in a causal implementation. Adjust the first score below and observe the mixture rather than treating attention as a hard lookup.
05 · Design your experiment
Build a four-token single-head reference implementation. Assert row sums and causal zeros. Perturb a future token and verify that earlier outputs do not change. Compare gradients with finite differences on a tiny input. Only after those checks, add multiple heads and compare batched output with a per-head implementation.
State one hypothesis; save the configuration and raw results; identify a competing explanation; report the narrowest supported conclusion. Use the linked lab as a local starting point, not a full paper reproduction.
Open related lesson / lab →06 · Limits & reviewer questions
Editorial caution: an attention weight is a mixing coefficient, not by itself a causal explanation of a model’s decision. Full-sequence training can compute positions in parallel, but autoregressive generation still depends on previously generated tokens. Dense attention also creates a sequence-length scaling problem.
If every row sums to one, why can a mask bug still leak future information?
Normalization checks only probability mass. A row can sum to one while assigning mass to forbidden columns. Test the allowed support and perturbation invariance separately.
07 · Read the original
Section 3, especially scaled dot-product attention, multi-head attention and positional encoding.
Year is the first arXiv submission year. Unversioned links follow revisions; record the exact version in your reproduction report. DeepSeek-R1 here is explicitly pinned to v1.
Continue this topic
FlashAttention: optimize data movement before counting FLOPs
See why not storing an attention matrix can matter as much as arithmetic, and reconstruct softmax one tile at a time.
Dao et al. · arXiv:2205.14135