Token Lab검색

SYSTEMS / ENGINEERING BRIDGE

MoE: active compute is not total memory

Audit routing, capacity, communication and expert imbalance.

Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →

This technical article displays the English edition.

Mechanism & assumptions

Sparse expert routing activates a subset of experts for each token. Stored parameters may greatly exceed active parameters, so a low active count does not establish that a model fits on one device. DeepSeek-V2 combines sparse experts and attention-cache compression; isolate these mechanisms when designing ablations. No specific speedup follows from parameter counts alone.

Text / 原始公式
assignments = tokens × top_k
mean_assignments_per_expert = tokens × top_k / experts
raw_weight_bytes = total_parameters × bytes_per_parameter

Worked example

For 128 tokens, 8 experts and top-2 routing, there are 256 assignments, averaging 32 per expert. With capacity 40, an expert receiving 60 assignments exceeds capacity by 20; the implementation must explicitly drop, reroute or accommodate them. A hypothetical 16B-parameter model at 2 bytes per weight stores 32 GB before overhead, even if only 4B parameters are active.

Implementation exercise

Create balanced and skewed routing histograms with the same token count. Report maximum load, overflow, per-expert utilization and communication bytes under a declared placement. Compare quality after any dropped-token policy.

Deliver code, fixed seeds, raw results and one counterexample. These are experiment specifications, not executed GPU benchmarks.

Why can sparse models be slow at small batch sizes?

Routing, communication and small expert matrix multiplications can dominate. Measure device utilization and transfer time rather than assuming FLOP savings equal latency savings.

arXiv ↗ · PDF ↗

Original teaching examples; sources checked October 2026. These are foundational mechanisms relevant to engineering, not a claim to cover every latest model.

AI/LM curriculum & checkpoints →Systems index →
MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

Activated parameter count is not total resident weight memory. Routing capacity, token imbalance, communication and expert precision all affect runtime. Distinguish logical dispatch bytes from measured network traffic.

Formula conditions, units and worked arithmetic reviewed. The implementation isolates one numerical mechanism; it does not train a policy or control real hardware.

Tokens and top-k expert routing → Dispatch token representations → Expert compute + return traffic → Count active compute vs resident memory
Mechanism map: follow the transformations from top to bottom.

Before running: predict a boundary case

256 expert assignments dispatch 2 MiB once; return traffic is additional. This excludes protocol overhead.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node moe-budgets.mjs. No third-party runtime dependencies.

const solve=()=>{
const tokens=128,topK=2,width=4096,bytes=2;
return [tokens*topK,tokens*topK*width*bytes/2**20];
};
const actual=solve();
const expected=[256,2];
if(JSON.stringify(actual)===JSON.stringify(expected)){console.log('PASS',actual)}else{
const a=[actual].flat(), b=[expected].flat();
if(a.length!==b.length||a.some((v,i)=>!Number.isFinite(v)||Math.abs(v-b[i])>1e-10))throw Error('Numerical mismatch');
console.log('PASS within 1e-10',actual);
}
Download this example
Reference output — reveal after predicting
[
  256,
  2
]

Test and transfer

256 expert assignments dispatch 2 MiB once; return traffic is additional. This excludes protocol overhead.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope