KV memory, paging and serving deadlines

Assumptions, implementation & acceptance

Count both K and V, KV heads rather than query heads, and sequence×batch exactly once. The byte formula excludes activations, weights, allocator fragmentation and attention workspace.

Formula conditions, units and worked arithmetic reviewed. The implementation isolates one numerical mechanism; it does not train a policy or control real hardware.

Before running: predict a boundary case

Double batch doubles MiB; change query heads alone and KV storage stays unchanged.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node kv-serving.mjs. No third-party runtime dependencies.

const solve=()=>{
const layers=32,kvHeads=8,headDim=128,sequence=4096,batch=1,bytes=2;
return 2*layers*kvHeads*headDim*sequence*batch*bytes/2**20;
};
const actual=solve();
const expected=512;
if(JSON.stringify(actual)===JSON.stringify(expected)){console.log('PASS',actual)}else{
const a=[actual].flat(), b=[expected].flat();
if(a.length!==b.length||a.some((v,i)=>!Number.isFinite(v)||Math.abs(v-b[i])>1e-10))throw Error('Numerical mismatch');
console.log('PASS within 1e-10',actual);
}

512

Test and transfer

Double batch doubles MiB; change query heads alone and KV storage stays unchanged.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.