Understand: state assumptions, shapes/units and a counterexample.
Implement: reproduce a numerical example and inspect boundaries.
Investigate: compare one alternative under a matched budget.
This technical article displays the English edition.
Mechanism & assumptions
LLaDA studies a language diffusion model using forward masking and a learned reverse process. The model predicts masked tokens conditioned on visible context; iterative generation differs from a left-to-right autoregressive factorization. The training objective includes noise-level weighting, so an ordinary masked-token classifier is only a teaching baseline, not a faithful reproduction. Engineering choices include sequence length, unmasking schedule, confidence calibration and whether tokens can be revised. Parallel token updates can still require expensive repeated full-sequence evaluations.
Text / 原始公式
Teaching masked loss = −(1/M) Σ_(i masked) log pθ(x_i | visible context)
M = number of masked positions; handle M=0 explicitly
Compute ledger ≈ steps × cost_per_full_sequence_pass
Worked example
Two masked positions assign probabilities 0.8 and 0.5 to their correct tokens. The mean masked cross-entropy is −(ln 0.8 + ln 0.5)/2≈0.458145. For a length-128 output with 16 full-sequence passes, the ledger counts 2048 token-position evaluations. This is not 2048 autoregressive decoding steps: kernels, caching and parallel work differ. A wall-clock measurement is required.
Implementation exercise
Begin with a tiny discrete sequence reconstruction task. Compare random versus confidence-based unmasking with identical masks and model weights. Report exact-match accuracy, per-position accuracy and sensitivity to step count. Then specify a real-model comparison at equal quality or latency, including stop behavior and output length; never infer general reasoning quality from reconstruction alone.
Deliver code, fixed seeds, raw results and one counterexample. These are experiment specifications, not executed GPU benchmarks.
Why can confidence-based unmasking lock in mistakes?
Confidence can be miscalibrated, especially under heavily masked context. Early wrong tokens alter later conditioning. Compare calibration across noise levels and test revision policies rather than trusting the highest score.
Original teaching examples; sources checked October 2026. These are foundational mechanisms relevant to engineering, not a claim to cover every latest model.
The displayed cross-entropy uses natural logarithms and is measured in nats. A masked-position average must divide by the number of evaluated masks; the toy loss does not reproduce an entire discrete diffusion objective.
Formula conditions, units and worked arithmetic reviewed. The implementation isolates one numerical mechanism; it does not train a policy or control real hardware.
Mechanism map: follow the transformations from top to bottom.
Before running: predict a boundary case
The result is nats. Divide by ln2 for bits; do not divide by unmasked token count.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node diffusion-language.mjs. No third-party runtime dependencies.
The result is nats. Divide by ln2 for bits; do not divide by unmasked token count.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope
