AI makes AI inference faster: from generated kernels to reliable serving
A systems survey of AI performance engineering: mechanisms, recent evidence, hardware limits, production architecture and a reproducible local exercise.
Sources & version details
In this article
What is actually improving?
AI-assisted inference optimization uses a model to propose changes to the software or configuration that executes another model, then retains changes using external evidence. It is not a model spontaneously becoming more intelligent while answering. The target weights can remain fixed while the implementation becomes faster. The optimizer itself may be a different model, running on different hardware.
Keep three mechanisms separate. Code optimization changes the implementation of a mathematical operation. Serving optimization changes scheduling, batching, caches or parallel placement. Model-level acceleration changes precision, architecture or the decoding algorithm and may alter the quality contract. “Same model” is not enough to establish equivalence if precision, sampling or token limits changed.
Our running example is an internal SRE assistant that reads incident evidence and generates a report. Its users care about waiting time and supported conclusions, not an isolated matrix multiplication. We will follow one request down to GPU operations and then follow an optimization proposal back to a controlled deployment. This is an original reference design, not a reconstruction of any vendor’s internal infrastructure.
Prerequisites: tensors, memory and latency
A tensor has a shape, dtype and strides. Shape gives logical dimensions; strides describe how those elements are laid out in memory. A kernel that is correct for a contiguous matrix can silently misread a transposed view. Dtype determines both storage size and arithmetic behavior. Floating-point addition is not associative, so changing reduction order can change numerical results without being an obvious programming bug.
HBM is large device memory; registers and shared memory are smaller, nearer storage. A GPU kernel is a program dispatched to many GPU threads. Fusion performs several operations in one dispatch, potentially avoiding intermediate HBM writes. Tiling reuses a block of inputs near the processor. Neither is free: more live values can increase register demand, reduce occupancy or spill to memory.
Latency is elapsed time for one request. Throughput is useful work completed per second across requests. Queueing can dominate latency near saturation even when kernel timings are unchanged. Before optimizing, understand your tensor contracts, the memory hierarchy and which product-level deadline you must meet.
Follow a request: prefill is not decode
The SRE request enters an authenticated API, is tokenized and queued. A scheduler admits it when model and KV-cache capacity permit. Prefill processes the prompt and constructs attention keys and values. Decode repeatedly produces the next-token distribution, samples a token and appends cache state until a stop condition. Streaming sends outputs while generation continues. Retries and cancellation must release request-owned state.
Prefill commonly exposes large matrix operations and substantial parallelism. Small-batch decode often spends more time moving weights and KV data relative to arithmetic; larger batches reuse weights across requests. These are tendencies, not universal diagnoses: long-context attention, sparse models, interconnect traffic and implementation choices change the limiting resource.
Define TTFT from the client request to its first returned token, including queueing and network. Define inter-token latency on the streamed sequence, not a GPU kernel. Report both percentile latency and aggregate output tokens/s at a specified arrival process. A configuration that produces more tokens across a big batch can still make every interactive user wait longer.
| Level | Question | Failure to avoid |
|---|---|---|
| Kernel | Does this operator improve on the target GPU? | Wrong shape, asynchronous timing, weak baseline |
| Engine | Does the forward pass improve under actual batch/context? | Cache/layout mismatch; MoE imbalance |
| Service | Do users meet TTFT and tail-latency targets? | Queue saturation, unfair warm-cache comparison |
| Task | Is a correct result cheaper or faster? | Shorter output or lower quality masquerading as efficiency |
Find the limiting resource before editing code
A simple roofline lower bound compares arithmetic time with memory-transfer time. Let F be floating-point operations, P attainable operations/s, D bytes transferred and B attainable bytes/s. The maximum of F/P and D/B is an optimistic lower bound when these overlap perfectly; launch, synchronization and communication add constraints. Hardware peak specifications are upper bounds, not measured application rates.
For an original teaching example, take a 4096×4096 FP32 array: one read and one write move 134,217,728 bytes. At an assumed sustained 1 TB/s, transfer alone needs about 0.134 ms. If a proposed pointwise implementation claims 0.01 ms while actually reading and writing the whole array from HBM, investigate cache residency, accounting or timing before celebrating. This is a calculation, not a hardware measurement.
An AI optimizer should receive the profile, shapes, strides, allowed error and actual bottleneck. Useful candidate changes include eliminating a redundant copy, combining pointwise operations or changing tile dimensions. Asking only “make this faster” invites work on a small or irrelevant component. Preserve the original baseline so that improvements remain attributable.
A concrete kernel: row-wise RMS normalization
Consider X with shape [M,N] and a learned scale vector w with shape [N]. For each row, compute the mean square, add a positive epsilon, take the reciprocal square root, and multiply each input by that value and its column scale. Unlike LayerNorm, RMS normalization does not subtract the mean. Accumulating squares in FP32 can reduce error when inputs are FP16 or BF16, but the supported value range must still be specified.
A decomposed implementation may materialize squared values and normalized intermediates. A fused implementation keeps a row or tile on-chip, reduces its sum of squares, and writes the final result. Padding to a convenient tile size requires masks: padded elements contribute zero to the sum, while the divisor remains the true N. Using the padded width as the divisor is a fast but wrong program.
Now the tradeoff: a very wide row may exceed a practical register budget. A multi-stage reduction adds launches and intermediate storage but may be faster for such shapes. The agent should select between valid implementations, not hard-code the answer for known inputs. The local exercise later implements the exact mathematical contract and a deliberately faulty padded variant; it does not pretend a Python loop reproduces GPU fusion performance.
The optimizer is a bounded engineering system
The control plane records the baseline commit, target hardware, workload distribution and optimization budget. A coding agent reads the allowed source and profile, proposes a patch and submits it as an immutable candidate. An isolated worker builds and runs it. The evaluator owns correctness cases and timing; the candidate must not rewrite its own judge. Results return as structured feedback, including compilation failures, errors, resource usage and timing distributions. A candidate registry records successes and failures so the loop does not repeatedly rediscover the same dead end.
Selection is a constrained search problem. Minimize latency or cost over candidates subject to numerical, quality, memory and tail-latency limits across the intended workload distribution. A single scalar score can hide a severe regression on a minority shape, so preserve the per-workload vector and use a Pareto frontier when objectives conflict. A compiler error is feedback; a budget exhaustion is a stopping condition, not permission to remove checks.
Keep execution and promotion separate. Sandbox workers receive no production credentials, have bounded GPU time and cannot modify the hidden suite. Candidate binaries and dependencies are versioned. Only a release controller can promote an independently qualified artifact, and only with a known-good rollback target. The architecture diagram shows this boundary; the agent has no direct path to production.
Reference architecture: optimization cannot bypass release control
Recent evidence: different claims, different denominators
Together Compiler’s November 4, 2026 report describes persistent engines and reports up to 2.6× TPS on its specified InferenceX comparisons. OpenAI’s July 29, 2026 engineering report attributes a 20% end-to-end serving-cost reduction to a combination of kernel-related improvements. DeepMind’s May 14, 2025 AlphaEvolve announcement reports a 23% kernel speedup corresponding to 1% lower Gemini training time. These are author-reported results, not Token Lab reproductions. [1] [2] [3]
The first is a throughput comparison, the second a cost statement, the third a training example. They cannot be placed in a single winner ranking. An absolute tokens/s number needs model, hardware count, precision, context lengths, batch or concurrency and latency constraints. A vendor’s best case describes one supported workload, not your entire traffic distribution.
Persistent experimentation addresses a practical bottleneck: loading weights and rebuilding an engine for every proposal consumes time that could be spent exploring candidates. However, keeping a process alive also requires state isolation. A candidate must not inherit invalid caches, allocator state or configuration from a previous one. Fast iteration and trustworthy measurements must be designed together.
Beyond kernels: cache, speculation and parallelism
Prefix caching reuses compatible prior computation; it helps repeated exact prefixes, not arbitrary similar text. Cache keys must bind model revision, relevant numerical configuration and tenant policy. A faster kernel cannot compensate for repeated prefill caused by unstable prefixes. Conversely, a warm-cache result must not be presented as a cold-cache kernel improvement.
Speculative decoding uses a draft mechanism to propose several tokens and a target mechanism to verify them. With the appropriate acceptance/correction algorithm, exact speculative sampling preserves the target distribution; not every heuristic draft scheme does. Benefit depends on acceptance, draft cost, verification cost and hardware utilization. Making the draft bigger can improve agreement while making the whole system slower. AI may search the draft architecture or implementation; the serving algorithm itself is a separate acceleration technique.
Tensor parallelism divides computations; expert parallelism distributes MoE experts; data parallelism replicates serving capacity. More devices may reduce local work while adding communication or imbalance. A short-query service and a long-context agent can prefer different placements. Search must include real routing skew and topology. Optimizing each layer independently can miss a better joint design or create incompatibilities between cache layouts and kernels.
From microseconds to business value
Amdahl’s law limits the end-to-end benefit of accelerating a fraction f of baseline time by factor s, holding the remaining work fixed. If kernels occupy 40% of a request and those kernels become twice as fast, overall speedup is only 1.25×. New time is 0.8 of old time: 20% less latency, not 100% less. This model excludes queueing changes; under load, a capacity change can change waiting time nonlinearly.
For a synthetic SRE request taking 1000 ms, let queue/network be 200 ms, prefill 300 ms, decode 400 ms and application work 100 ms. Halving decode yields 800 ms. Halving prefill as well yields 650 ms, or 1.54×. These numbers illustrate accounting only. A saved 200 ms repeated over 20 strictly sequential calls saves 4 s; parallel calls and tool waits alter the critical path.
Optimization has a price. If an offline search and qualification run costs an assumed $120 and saves $0.0002 per completed equivalent request, break-even is 600,000 requests, before maintenance. Count failed experiments, GPUs, coding-model tokens, human review and rollback risk. The relevant denominator is successful tasks at required quality and SLO, not raw generated tokens.
Who owns which part of the stack?
Wafer presents workload-specific inference optimization as a hosted product; its YC profile explicitly describes agents acting as performance engineers. That positions the product around an application team buying an endpoint, rather than buying a kernel file. Treat marketing examples as scoped claims pending your own workload comparison. [4]
Together’s compiler is a serving-provider optimization system; OpenAI’s example is optimization inside a model provider; DeepMind’s AlphaEvolve is a broader algorithm-discovery system. KernelBench is an evaluation toolkit, not an inference vendor. Apple’s RSI-Inference Bench targets edits to real vLLM/SGLang engines, rather than just isolated operators. The official repository lists 37 tasks across model support, optimization and regression repair. [1–3] [5] [6]
Our industry interpretation: the durable advantage is access to representative workloads, reliable qualification, hardware utilization and delivery—not just code generation. Enterprise buyers should ask who controls telemetry, who approves changes, whether they can export artifacts and how a failed optimization falls back. A hosted service can amortize search across traffic; a private deployment gains control but owns engineering cost. This is a technical positioning comparison, not a revenue, market-share or acquisition ranking; those figures are not established here.
Production boundaries and unresolved problems
Ship an artifact manifest containing source commit, model revision, compiler/runtime versions, GPU architecture, supported shapes/dtypes, error contract, measurements and rollback identifier. The runtime dispatches only supported cases; all others use a valid fallback. A shape guard is an executable contract, not a note in a README. Shadow execution should not double live side effects, and canary traffic must respect the same tenant permissions as normal serving.
Monitor client latency, error rate, OOMs, output-quality signals and resource saturation. If the canary breaches a limit, stop promotion and restore the last qualified artifact. Keep in-flight requests on a consistent implementation or drain them before switching incompatible cache formats. Kubernetes can schedule GPU workers and roll deployments, but it does not prove numerical correctness or make arbitrary hot replacement safe. Separate optimization worker service accounts from serving deployment permissions.
Open problems include transfer across shapes and GPUs, correlated benchmark contamination, numerical drift in long generations, hardware noise, expensive qualification and shifting traffic. Faster search can overfit faster. Repeatedly selecting on the same validation suite turns it into training feedback. Preserve a final independent workload and account for how often it has been consulted. The most useful frontier is not an unlimited agent: it is cheaper search with stronger, independently enforced evidence.
Is this recursive self-improvement?
There is a meaningful feedback loop if an AI system improves the inference stack that later runs its own optimization work. More useful experiments can then fit inside a fixed budget. But one successful kernel patch establishes only one optimization event. To establish recursive amplification, show multiple generations, unchanged evaluation rules, costs of each generation and held-out improvements in the next generation’s ability to optimize.
Do not equate faster inference with higher reasoning capability, or code evolution with weight learning. The evaluator, hardware access, human interventions and search budget can be the true source of gains. A fair recursive study compares a fixed optimizer, a growing-budget fixed optimizer and the proposed improving optimizer. Report where each generation plateaus and which responsibilities remain human-controlled. This distinction makes the topic more concrete: we already have useful engineering automation without needing to claim an autonomous intelligence explosion.
Practice and independent verification
Local experiment: reject a fast but incorrect candidate
Run Python 3.11 or newer with only its standard library: python3 lab.py. Seed 42 fixes the input stream. The reference computes RMS normalization; a candidate reuses the row scale; one faulty implementation divides by padded width; another returns zeros. The padded bug passes development widths 4 and 8 but fails audit widths 3, 7 and 33. This demonstrates shape overfitting without any paid model call or upload. The audit cases are public teaching fixtures, not a hidden production test. The reference rejects empty scale vectors, mismatched widths and non-finite inputs; production candidates must be guarded by the same contract.
Expected result: candidate passes both sets; padded_bug passes development only; zero_bug fails both. Amdahl’s example returns 1.25 and break-even returns 600000. GPU benchmark performed remains false. Execution times are intentionally not used to claim a GPU speedup. Python’s version is recorded in the output; no external dependency versions are needed.
For a GPU extension, implement the same contract in CUDA or Triton and keep the driver, GPU, compiler, framework versions and source hash in the result. Include odd widths, zero rows, large values, supported non-contiguous layouts and dtype boundaries. Compare to an appropriate optimized framework baseline, not just a slow Python decomposition. Warm compilation separately; synchronize GPU work; use device events and repeated independent runs; randomize candidate order; control competing workloads and report median and tail distributions. Audit output shape, dtype, finite values, maximum absolute/relative error and downstream quality. A tolerance check is empirical evidence, not a proof of equivalence.
Only after kernel qualification, replay a held-out serving workload at fixed model, precision, context lengths, arrival pattern, cache conditions and quality constraints. Compare manual/default configuration, parameter search, AI-generated kernels and the full stack, with matched optimization budgets. Preserve compilation failures and rejected candidates in the denominator. Finish with a canary and rollback drill. Token Lab has not run this GPU extension; it is the acceptance protocol for your own hardware.
Download Python implementation"""Token Lab: CPU correctness gate, not a GPU performance benchmark.
Python 3.11+ standard library; no network, model API, or third-party dependency.
"""
import json
import math
import random
import sys
def reference(rows, weights, eps=1e-5):
if eps <= 0 or not weights or any(len(r) != len(weights) for r in rows):
raise ValueError('positive epsilon and rectangular nonempty rows required')
if any(not math.isfinite(x) for r in rows for x in r + weights):
raise ValueError('finite input contract')
return [[x * w / math.sqrt(eps + math.fsum(v*v for v in r)/len(r))
for x, w in zip(r, weights)] for r in rows]
def candidate(rows, weights, eps=1e-5):
output = []
for row in rows:
inv = 1 / math.sqrt(eps + math.fsum(x*x for x in row)/len(row))
output.append([x*w*inv for x, w in zip(row, weights)])
return output
def padded_bug(rows, weights, eps=1e-5):
padded = 1 << (len(weights)-1).bit_length()
return [[x*w/math.sqrt(eps + math.fsum(v*v for v in row)/padded)
for x,w in zip(row,weights)] for row in rows]
def zero_bug(rows, weights, eps=1e-5):
return [[0.0]*len(weights) for _ in rows]
def passes(fn, cases):
for rows, weights in cases:
expected, actual = reference(rows, weights), fn(rows, weights)
if len(actual) != len(expected):
return False
for a,b in zip(actual,expected):
if len(a) != len(b) or any(not math.isfinite(x) or
not math.isclose(x,y,rel_tol=1e-12,abs_tol=1e-12)
for x,y in zip(a,b)):
return False
return True
def main():
rng = random.Random(42)
dev = [([[rng.uniform(-3,3) for _ in range(n)] for _ in range(3)],
[rng.uniform(.5,1.5) for _ in range(n)]) for n in (4,8)]
# Independent shape coverage, public for teaching: NOT a secret benchmark.
audit = [([[rng.uniform(-3,3) for _ in range(n)] for _ in range(3)],
[1.0]*n) for n in (1,3,7,33)]
audit += [([[0.0,0.0,0.0]], [1.0]*3), ([[1000.,-1000.,1.]], [1.0]*3)]
results = {fn.__name__: {'development':passes(fn,dev), 'audit':passes(fn,audit)}
for fn in (candidate,padded_bug,zero_bug)}
assert results['candidate'] == {'development':True,'audit':True}
assert results['padded_bug'] == {'development':True,'audit':False}
assert results['zero_bug'] == {'development':False,'audit':False}
for bad in [([], []), ([[1.,2.]], [1.]), ([[float('nan')]], [1.])]:
try:
reference(*bad)
except ValueError:
pass
else:
raise AssertionError('invalid input accepted')
speedup = 1 / (.6 + .4/2)
assert speedup == 1.25
print(json.dumps({'python':sys.version.split()[0], 'seed':42,
'correctness':results, 'synthetic_end_to_end_speedup':speedup,
'synthetic_break_even_requests':120/.0002,
'gpu_benchmark_performed':False},indent=2))
if __name__ == '__main__':
main()
{
"python": "3.12.14",
"seed": 42,
"correctness": {
"candidate": {
"development": true,
"audit": true
},
"padded_bug": {
"development": true,
"audit": false
},
"zero_bug": {
"development": false,
"audit": false
}
},
"synthetic_end_to_end_speedup": 1.25,
"synthetic_break_even_requests": 600000.0,
"gpu_benchmark_performed": false
}
Original interview questions and layered answers
1. Why did a 2× kernel yield only 1.25× end-to-end?
Hint: identify the accelerated fraction. Answer: with
2. Why does the padded candidate pass development?
Hint: inspect the widths. Answer: 4 and 8 equal their power-of-two padding. At width 3, using 4 underestimates the mean square and inflates normalization. Add independent shapes, not just more values of the same shapes.
3. Can the optimizer edit its timing harness?
Hint: separate proposal and authority. Answer: agent-owned instrumentation may help diagnosis, but the independent final harness must remain protected. Verify completed GPU work, output dependence and fresh inputs; otherwise cached answers or missing synchronization can win.
4. What would establish a recursive improvement loop?
Hint: one patch is not a generation study. Answer: show multiple generations and held-out optimization gains at controlled total budgets, including human intervention, search and qualification costs. Compare with a fixed optimizer receiving the same extra compute.
Original practice, not reported company interview questions. Score 1: names terms; 2: explains the normal path; 3: derives the example and catches the faulty candidate; 4: designs fair workload comparisons and production boundaries; 5: independently reproduces results, diagnoses a new failure and explains uncertainty and transfer limits.
Related learning routes
Foundations explained
Review the shared reasoning behind this method, then use the worked checks to identify gaps in your understanding.
Autoregressive decoding generates one new token conditioned on earlier tokens. In a causal Transformer, earlier keys and values need not be recalculated when the prefix and model are unchanged. A KV cache stores these internal tensors, not a finished answer. To reason about capacity, count layers, batch items, cached positions, KV heads, head dimensions and bytes per number before quoting a memory estimate.
Work it out before reading the solution
For 2 layers, batch 1, 4 cached tokens, 2 KV heads, head dimension 3 and 2 bytes per number, how many bytes store K and V together?
Reasoning, step by step
The count is 2×2×1×4×2×3×2=192 bytes. The first factor is for both K and V. This excludes weights, temporary attention buffers, allocator overhead and padding. At fixed batch and shape, doubling context doubles this payload; it does not imply total GPU memory doubles.
Why this matters in practice
Grouped-query attention uses fewer KV heads than query heads, reducing this storage term. It is an architectural choice, not a permission to discard arbitrary cached tensors from an already trained model. Prefix reuse also depends on matching tokens, positions, model and relevant execution settings.
Evidence of understanding
Compare cached and uncached outputs for the same prefix within numerical tolerance. Change the prefix and verify invalidation. Report useful tensor bytes separately from allocated memory, and prefill latency separately from decode latency. A browser byte calculator is not a hardware throughput benchmark.
Run the complete Python example
Python 3.10+ · standard library only · deterministic, no random seed needed · no network. Save the file and run the command below. These are teaching checks, not model benchmarks.
Download Python · kv.pypython3 kv.py# Python 3.10+; standard library; deterministic; no network.
def kv_bytes(layers, batch, tokens, kv_heads, head_dim, bytes_per_value):
dimensions = [layers, batch, tokens, kv_heads, head_dim, bytes_per_value]
if any(not isinstance(x, int) or x < 0 for x in dimensions):
raise ValueError("nonnegative integer dimensions required")
size = 2 # K and V
for dimension in dimensions:
size *= dimension
return size
assert kv_bytes(2, 1, 4, 2, 3, 2) == 192
assert kv_bytes(2, 1, 8, 2, 3, 2) == 384
assert kv_bytes(2, 1, 0, 2, 3, 2) == 0
print("KV payload=192 bytes; doubled context=384 bytes")
Expected output
KV payload=192 bytes; doubled context=384 bytes
Foundation revised:
References, papers and implementation resources
- Together Compiler — technical report, 2026-11-04 (not claimed as conference acceptance)https://www.together.ai/together-compiler-paper
- OpenAI — engineering report, 2026-07-29https://openai.com/index/gpt-5-6-frontier-intelligence-efficiency/
- Google DeepMind — AlphaEvolve announcement, 2025-05-14https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/
- Wafer — company profile / product positioninghttps://www.ycombinator.com/companies/wafer
- KernelBench — ICML 2025 proceedingshttps://proceedings.mlr.press/v267/ouyang25a.html
- Apple — RSI-Inference Bench official repository (checked 2026-11-11; no conference status asserted)https://github.com/apple-aiml-research/ml-rsi-inference-bench
- KernelBench — arXiv v1, 2025-02-14https://arxiv.org/abs/2502.10517v1
- KernelBench — arXiv PDF v1https://arxiv.org/pdf/2502.10517v1
- KernelBench — author codehttps://github.com/ScalingIntelligence/KernelBench
- AlphaEvolve — original technical PDFhttps://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf
- Triton — official kernel tutorialshttps://triton-lang.org/main/getting-started/tutorials/
- Wafer — current product descriptionhttps://www.wafer.ai/