集中した注意力
Larger logits concentrate probability on the largest visible score.
ローカル計算・シミュレーション外部 API を使わないローカル教材です。合成データの結果は本番測定ではありません。
Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
02 · 予測と設定
技術説明、パラメータ名、解答は現在英語です。
03 · 結果を確認
方法と参考文献
One synthetic head, four positions,
Text / 原始公式
softmax(scale × QKᵀ / √2 + mask)V
Text / 原始公式
Q = [[1,0],[0,1],[1,1],[0.5,-1]] K = [[1,0],[0,1],[1,1],[-1,1]] V = [1,3,5,7]
A high attention weight does not establish causal feature importance.
Attention Is All You Need · §3.2 ↗練習問題
attention-3 / 01What does “Sharp attention” test?
Larger logits concentrate probability on the largest visible score.
Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗
ラボを開く →attention-3 / 02Which computation produces the result in “Sharp attention”?
softmax(scale × QKᵀ / √2 + mask)V
Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗
ラボを開く →attention-3 / 03What is the baseline configuration for “Sharp attention”?
{"scale":5,"causal":1}
Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗
ラボを開く →attention-3 / 04What numerical result should you check for “Sharp attention”?
{"Mean entropy (bits)":0.19672661245634565,"Last output":1.1271721043873806}
Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗
ラボを開く →attention-3 / 05How would you run a controlled comparison for “Sharp attention”?
Keep the supplied fixture and other settings fixed; change scale. Inspect per-row normalization, entropy and the weighted value output.
Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗
ラボを開く →attention-3 / 06What failure could invalidate a conclusion from “Sharp attention”?
A high attention weight does not establish causal feature importance.
Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗
ラボを開く →attention-3 / 07Is “Sharp attention” a benchmark of a deployed language model?
No. One synthetic head, four positions,
Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗
ラボを開く →attention-3 / 08Which evidence should you report for “Sharp attention”?
Inspect per-row normalization, entropy and the weighted value output. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.
Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗
ラボを開く →attention-3 / 09How can another researcher reproduce “Sharp attention”?
Open attention-3, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.
Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗
ラボを開く →attention-3 / 10What is the next validation step after “Sharp attention”?
A high attention weight does not establish causal feature importance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.
Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗
ラボを開く →Assumptions, implementation & acceptance
Self-attention: X is N×d_model; W_Q,W_K are d_model×d_k; W_V is d_model×d_v. Scores are N×N and output N×d_v. The variance argument assumes independent, zero-mean, unit-variance coordinates. Cross-attention can have different query/key lengths.
Engine formulas, scenario presets and numerical safeguards reviewed. Seeded synthetic inputs are not a production benchmark.
Before running: predict a boundary case
Larger logits concentrate probability on the largest visible score.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node attention-3.mjs. No third-party runtime dependencies.
import {calculate,network} from './academy-core.mjs';
const params={
"scale": 5,
"causal": 1
};
const result=calculate('attention',params);
console.log(result.metrics);Test and transfer
Larger logits concentrate probability on the largest visible score.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Download complete kit (.tar.gz) · Complete reference kit and regression checksPredict, test and keep evidence
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
Open notebook / export / clear →The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope