AI infrastructure in 2026: from fast chips to useful, reliable capacity
Follow a request through admission, prefill, KV memory, decoding, tools and recovery; connect capacity and cost to frontier hardware and market evidence.
Sources & version details
Contents · lifecycle and reference chapters
In this article
Infrastructure: follow one request through the system
This original case study follows an invoice assistant. It reads tenant-scoped documents and proposes a payment; executing a payment requires separate, exact authorization. The scenario and experiments are synthetic teaching material, not a reported deployment.
Follow one request: the infrastructure behind an invoice agent
An employee asks: “Why is invoice INV-42 overdue, and prepare the next payment action if it is eligible.” Our assistant searches that employee’s tenant documents, checks invoice state, drafts an explanation and proposes an action. It may execute a payment only after an exact approval. This is a synthetic teaching system, not a description of a particular vendor. Its infrastructure must support a durable business workflow rather than merely return fluent tokens.
The request crosses an authenticated gateway, a workflow service, retrieval storage, an inference endpoint, a tool gateway and a business system. These components have different failure semantics. A failed read can usually be repeated. A timed-out payment submission cannot safely be repeated until its outcome is reconciled. The model process is therefore not the owner of the transaction. Durable workflow state records which step is pending, which evidence was used and whether an action has committed.
Before choosing hardware, write the workload contract: arrival rate and burstiness, input and output token distributions, number of model calls per task, retrieval fan-out, tool latency and user deadlines. Separate an interactive answer from a background reconciliation job. “Ten requests per second” is not a capacity specification when one request is a 200-token answer and another triggers eight long-context calls.
Stage 1: admission, context assembly and resource budgets
The gateway authenticates the caller and resolves tenant context before placing work on an inference queue. Rate limits protect a shared service, while admission control decides whether the current system can meet the request’s deadline. Rejecting or deferring work early is often better than accepting it into an unbounded queue whose eventual answer arrives after the user has abandoned the task.
Context assembly has a budget. Reserve space for the system instructions, tool schemas, retrieved evidence, conversation history and maximum output. A declared model context window is an upper limit, not a recommendation to fill it. Retrieval should select relevant, authorized material. Truncation must preserve required policy and provenance; silently dropping the approval requirement to fit more documents is a functional and security defect.
Use estimated tokens for early admission and measured usage for reconciliation. Track per-tenant concurrency and resource occupancy as well as request count. A tenant sending a few enormous prompts can dominate memory despite staying below a request-per-minute limit. Cancellation should propagate through retrieval and inference so abandoned requests do not continue consuming capacity indefinitely.
For a synthetic example, admit at most 24 active generations and queue at most 48 additional interactive requests with a two-second queue deadline. These numbers are illustrative, not recommended defaults. The research question is whether a token-aware or deadline-aware scheduler improves accepted-task success under the same hardware and traffic distribution; a lower mean latency alone is insufficient if one tenant starves.
Stage 2: prefill, decoding, KV memory and scheduling
After retrieval, the inference system first processes the prompt and then generates output tokens. Prefill and decoding have different execution patterns. A long prompt can delay the first generated token; a long answer occupies decoding capacity after the user has already seen the first token. Measure time to first token, inter-token latency, time to final token and total task completion separately.
For a conventional attention cache, a teaching memory estimate is two times layers times KV heads times head dimension times cached tokens times bytes per element, summed across resident sequences. With 32 layers, eight KV heads, head dimension 128, 8,192 cached tokens and two-byte elements, one sequence needs 1 GiB for K and V alone. Sixteen such sequences require 16 GiB. Weights, activations, allocator overhead, fragmentation and implementation-specific state are additional. Do not substitute query-head count for KV-head count in grouped-query attention.
Batching improves utilization by sharing execution, but waiting to form a batch can increase latency. Requests also finish at different times, so a scheduler must manage changing active sequences and memory. Chunking long prefill work can protect decode responsiveness, but the benefit depends on kernels, hardware and workload. Benchmark the actual mixture rather than extrapolating from a uniform synthetic batch.
vLLM documents prefix cachinghttps://docs.vllm.ai/en/stable/features/automatic_prefix_caching/ as reuse of cached computation for shared prefixes. It can reduce repeated prefill, not eliminate the work of generating new output.
Separate three bottlenecks before buying capacity. Prefill performs attention and feed-forward work across prompt tokens; decoding repeatedly reads weights and cache while producing relatively little new work per sequence. Which resource dominates depends on batch size, context, kernels and hardware. Tensor parallelism partitions model computation but adds communication on the critical path; pipeline parallelism partitions layers but requires enough scheduled work to hide stage bubbles. Replicas increase independent serving capacity only when each replica fits and the traffic can be balanced.
Disaggregating prefill and decode can isolate their resource demands, but it introduces a KV-transfer dependency. In the 1 GiB cache example above, an ideal 25 GiB/s path needs at least 40 ms for the cache bytes alone; protocol, serialization, contention and handoff add time. That arithmetic is a lower bound, not measured performance. Separation is unattractive when transfer and idle capacity cost more than the interference it removes. Quantization likewise saves storage or bandwidth only under compatible kernels; lower bit width alone does not prove faster service or unchanged task quality.
Stage 3: routing and fallbacks are changes to the system
The workflow may route a simple invoice lookup to a cheaper model and a disputed invoice to a stronger one. Routing is a policy with its own errors: a difficult task can be misclassified as simple, or every task can be sent to the expensive route. Preserve the routing decision, model version, prompt version and budget in the execution record so a later regression can be attributed to the correct component.
A provider outage may trigger a fallback. The substitute model can differ in tool-call syntax, context limits, instruction following and refusal behavior. A syntactically successful fallback is therefore not automatically an equivalent application. Normalize protocol differences at an adapter, but run behavioral evaluation before enabling that route. Do not allow a fallback to bypass the same tool authorization gateway or approval checks.
Circuit breakers reduce repeated calls to a failing dependency. Retry budgets and exponential backoff reduce load amplification, but neither establishes whether a side effect happened. Separate model-call retries from business-action retries. Model generation can produce a new proposal; executing that proposal still requires the current authorization and idempotency rules. A workflow checkpoint should capture the logical operation and its state, not just the latest natural-language message.
Stage 4: capacity planning connects queues, cost and useful work
Capacity planning begins with a measured service curve: how accepted throughput and latency change as offered load rises. Near saturation, queueing delay can grow rapidly even when average GPU utilization looks healthy. The system may produce more tokens while completing fewer requests within their deadlines. This is why a throughput benchmark must state the quality and latency constraints under which work counts as useful.
In a stable system, Little’s law relates average work in the system to arrival rate times average time in the system. At eight completed requests per second and a mean end-to-end duration of three seconds, average in-flight work is 24 requests. This is an accounting relationship under its stability assumptions, not a proof that 24 concurrent slots will meet a p99 deadline. Bursts, service-time variance and correlated dependency failures still matter.
Use a cost ledger with consistent units. Suppose a synthetic workload completes 1,000 requests for $20 of inference and tool costs, and 800 meet the acceptance definition. Cost per request is $0.02; cost per accepted task is $0.025. If a cheaper route lowers spend to $16 but only 500 tasks are accepted, the latter rises to $0.032. Add reserved capacity, idle time, storage, network and review labor when estimating the full business cost; keep these components visible.
A task is not one model request. Suppose ten user tasks arrive each second and each invokes three model calls on average: the model tier sees 30 calls/s before retries. If 10% of calls independently need one additional attempt, the planning estimate becomes 33 calls/s. This is a synthetic one-retry model; correlated outages and recursive agent loops violate that assumption. Bound attempts per task and across the fleet so recovery does not create more load than normal operation.
Autoscaling must respect startup time and state. If a new replica needs minutes to load weights, a seconds-long burst is served by existing headroom, queues, or admission rejection—not by a replica that becomes ready afterward. Scale on queue age, active tokens and measured capacity as appropriate, rather than GPU utilization alone. For a deadline-constrained service, extra capacity is useful only if it increases accepted completions before deadlines without breaching quality or tenant fairness.
Stage 5: deployment, failure recovery and a defensible operating model
A release bundles more than model weights. Tokenizer, prompt template, tool schema, retrieval index, quantization settings, inference engine and routing policy can all change behavior. Give each artifact a version and keep a compatibility record. Reproducing an old incident may require the historical document snapshot and tool response, not merely the model name.
Use a staged deployment: offline validation, a small shadow workload where actions cannot execute, then a limited canary under real traffic. Compare the canary with a contemporaneous baseline because traffic changes over time. Rollback must restore compatible workflow and schema behavior; reverting a container does not undo a payment already committed. Keep irreversible side effects outside experiments unless an explicit approval and compensation policy exists.
Failure drills should include a full inference-region outage, exhausted quota, a slow retrieval store and a restart during an approval wait. Define recovery time and data-loss objectives for workflow records separately from cache state. KV caches can often be recomputed at a cost; losing approval evidence or committed-action state is a different class of failure.
An engineer finishes this story with four artifacts: a workload model, a measured capacity envelope, a dependency-aware recovery plan and a release report that connects resource changes to task outcomes. The later hardware and company sections should be read against those needs. A faster accelerator is valuable when it removes the measured bottleneck; a larger platform is valuable when it reduces total operational risk and cost. Neither substitutes for understanding what the application must preserve when it is overloaded or partially broken.
How the four disciplines fit together
Security determines which actions are permitted. Infrastructure executes work within capacity and failure constraints. Observability preserves evidence of what happened. Evaluation tests whether a change improves the outcome under a declared contract. None substitutes for the others.
Mechanisms, research and market context — English reference material
The following chapters provide the mathematical examples, experiments, research papers and dated company evidence behind the operational story. Return to the lifecycle when deciding where each mechanism belongs.
1. The unit of progress is a completed workload
An AI infrastructure stack turns electricity, silicon, software and operational labor into training runs or useful responses. A faster accelerator is only one input. A customer experiences the time to a correct answer; an operator pays for reserved capacity, idle time, retries and failures as well as successful computation. The important question is therefore not “which chip has the largest number?” but “what bottleneck prevents this workload from meeting its service objective?”
Definitions. An accelerator is a processor optimized for selected computations. HBM is high-bandwidth memory close to the accelerator; it is not interchangeable with host RAM or remote storage. Scale-up interconnects connect processors within a tightly coupled system; scale-out networking joins systems. A service-level objective (SLO) specifies a measurable target such as a latency percentile. Goodput is the rate of useful completions satisfying specified constraints, rather than every generated token or attempted request.
Before this lesson: understand that matrix multiplication creates reusable parallel work, while autoregressive generation has a dependency between output steps. Understand a byte, a second and a watt: power measures a rate of energy use, and capacity in GiB says nothing by itself about transfer rate. Finally, distinguish average latency from a tail percentile. An average can improve while the slowest users experience worse service. Review attention and KV-cache correctness if those distinctions are unfamiliar.
Learning goals: draw the system boundary; calculate a cache budget with units; identify a misleading benchmark comparison; explain why an acquisition adds an engineering capability; and design a repeatable serving experiment. The company map below is selective, not a complete procurement recommendation or a claim that every listed product is available in every region.
2. Read the stack from the workload down
Start at the application. A short autocomplete request, a long-document question and an agent executing twenty tool steps have different prompt lengths, output lengths and interruption patterns. A speech assistant needs predictable delay; an overnight research job may accept queuing. One aggregate “tokens per second” metric conceals these differences. Record the request distribution before choosing a runtime or hardware platform.
The serving runtime schedules requests and manages state. Continuous batching can admit new work as other requests finish, but mixing a large prompt with latency-sensitive decoding can create interference. Cache allocation determines how many active sequences fit. Parallelism then distributes weights and work across processors. If a model requires frequent collectives or expert routing, communication may erase the benefit of additional compute.
The rack supplies the physical constraints: memory channels, interconnect topology, cooling and electrical delivery. A training job also needs dataset throughput and durable checkpoints. A data pipeline can starve expensive processors; a recovery procedure can waste a large fraction of a run even when individual kernels are fast. Capacity planning must include failed jobs and reloading weights, not just the steady state.
Our engineering interpretation is that differentiation increasingly comes from coordinating these layers. An efficient kernel does not repair a slow scheduler; a faster network does not fix an inaccurate low-precision model; an available GPU does not imply an energized rack. This is why infrastructure is both a distributed-systems problem and an industrial deployment problem.
Industry landscape: control points, buyers and competition
Market evidence checked November 8, 2026. Product documentation establishes an offering, not independently measured efficacy. Competitive theses and unmet opportunities are Token Lab analysis; examples labeled synthetic are not company results.
AI infrastructure: who owns capacity, workflows and responsibility?
Industry review checked November 8, 2026. Official sources establish product scope and transaction milestones; competitive conclusions are our interpretation. This is not a market-share ranking or an independent vendor benchmark. Undated product pages were checked on the review date; dated source entries identify publication or update dates.
The market is a chain of responsibilities
Follow one enterprise assistant from stored documents to a completed task. A data platform supplies authorized evidence; a GPU cloud supplies capacity; an inference operator turns capacity into responses; the application decides what action is allowed. Each layer sells relief from a different bottleneck. Comparing all vendors as interchangeable AI clouds hides the buyer, margin structure and security responsibility. Our competitive judgments below are analytical interpretations, not measured market shares or investment recommendations. A private endpoint can protect transport while an overprivileged application still leaks another tenant’s records.
Reference architecture: serving under latency and memory limits
This diagram chooses split prefill/decode serving to expose the costs; it is not a requirement for every deployment. Admission reserves token and concurrency budgets before queueing. The scheduler dispatches prompt processing to prefill, then transfers the resulting KV state to a compatible decode worker. The registry supplies pinned model artifacts. Both pools emit operational metrics, while cancellation and deadlines must propagate to workers to release reservations.
The transfer is useful only if specialization gains exceed serialization, network and coordination costs. Co-located serving avoids this transfer and may be preferable for small workloads. A cached prefix must match model and cache format as well as the intended isolation boundary. Bound queue length and waiting time; scaling cannot instantly repair a saturated system. Record queue time, first-token latency, token cadence and peak KV occupancy separately, so cheaper capacity is not mistaken for a better completed request.
Company and product positions
A selective map, not market share or a vendor certification. Coverage depends on edition, integration, deployment and enforcement mode.
NVIDIA
- Ownership / organization
- NVIDIA
- Control point
- Accelerators, fabric and orchestration
NVIDIA sells the complementary hardware and software that platform teams need to turn GPUs into useful service. Dynamo addresses distributed inference; Run:ai addresses workload allocation. On October 13, 2026, NVIDIA released the full Run:ai platform as open source, including four modular components. Analyst interpretation: generic scheduling features face price pressure, while integration and operational support remain valuable. Resource quotas and GPU sharing do not establish application authorization or prove strong tenant isolation.
NVIDIA: full Run:ai platform open-source release ↗https://developer.nvidia.com/blog/nvidia-runai-is-now-open-source-as-part-of-nvidia-dsx-ai-factory-platform · NVIDIA Dynamo ↗https://www.nvidia.com/en-us/ai/dynamo/
AWS
- Ownership / organization
- Amazon
- Control point
- Cloud and managed model APIs
AWS serves enterprises already operating within its identity, billing and operational systems. Bedrock combines managed model access with IAM controls; its strategic advantage includes procurement and integration, not only model speed. Security follows a shared-responsibility model. Customers still scope credentials, data access and downstream tools. An identity allowed to invoke a model is not thereby authorized to perform every action suggested by that model. Assess deployment-specific controls rather than treating the cloud brand as a complete security policy.
Amazon Bedrock: data protection and shared responsibility ↗https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html · Amazon Bedrock: invocation permissions ↗https://docs.aws.amazon.com/bedrock/latest/userguide/inference.html
Microsoft Foundry
- Ownership / organization
- Microsoft
- Control point
- Enterprise AI control plane
Microsoft packages models, agent development and connected Azure resources within an enterprise administrative structure. The buyer often already owns Entra identity and Azure commitments, making procurement and governance integration important. Foundry supports resource/project roles, managed identities and private networking. These controls reduce platform exposure but do not automatically isolate every dependency: an AI gateway and connected data services need their own configuration. Application teams must still authorize each sensitive action and reconcile its actual outcome.
Microsoft Foundry architecture ↗https://learn.microsoft.com/en-us/azure/foundry/concepts/architecture · Microsoft Foundry network isolation ↗https://learn.microsoft.com/en-us/azure/foundry/how-to/configure-private-link
Google Cloud
- Ownership / organization
- Alphabet
- Control point
- AI, data and cloud governance
Google Cloud competes where enterprise data and managed AI workflows meet. Its security architecture combines identity, service perimeters and encryption/key controls, but support varies by service. Data-residency terms explicitly distinguish certain grounding and RAG features. Analyst interpretation: integration around existing data can be more defensible than a model catalogue. The corresponding buyer risk is assuming that one approved AI service makes every connected feature compliant. Review each processing path and feature rather than inheriting a platform-wide conclusion.
Google Cloud service-specific data residency ↗https://cloud.google.com/terms/data-residency · Google Cloud: securely deploying AI ↗https://cloud.google.com/blog/products/identity-security/mastering-secure-ai-on-google-cloud-a-practical-guide-for-enterprises/
CoreWeave
- Ownership / organization
- CoreWeave
- Control point
- GPU cloud and developer workflow
CoreWeave supplies specialized GPU capacity while expanding into software through Weights & Biases and OpenPipe. The customer buys reliable capacity delivery and a shorter path from development to training and serving. Analyst interpretation: owning experiment and post-training workflows can increase retention and compute demand, but does not eliminate capital intensity or concentration risk. Platform controls protect infrastructure boundaries; enterprise application permissions remain separate. Acquisition completion proves ownership, not successful integration, better model quality or application safety.
CoreWeave completes Weights & Biases acquisition ↗https://www.coreweave.com/blog/coreweave-completes-acquisition-of-weights-biases · CoreWeave 2025 Form 10-K ↗https://www.sec.gov/Archives/edgar/data/1769628/000176962826000104/crwv-20251231.htm
Nebius
- Ownership / organization
- Nebius Group
- Control point
- GPU capacity and production cloud
Nebius combines large dedicated capacity supply with managed cloud services and production governance. Its Microsoft agreement illustrates a crucial market structure: a hyperscaler can be both competitor and customer. The 2025 announcement described approximately $17.4 billion through 2031, conditional on deployment and availability, with expansion potential. That is not annual revenue or cash already collected. IAM and infrastructure governance help enterprise adoption, while customer data policies and agent action permissions remain outside a capacity contract.
Nebius Cloud 3.6 governance release ↗https://nebius.com/newsroom/nebius-ai-cloud-3-6-strengthens-developer-experience-and-governance-for-production-operations · Nebius–Microsoft agreement ↗https://www.sec.gov/Archives/edgar/data/1513845/000110465925088312/tm2525580d1_6k.htm
Together AI
- Ownership / organization
- Together AI
- Control point
- Open-model inference and deployment
Together targets teams that want open models without assembling all serving operations. Dedicated endpoints reserve isolated GPUs, while managed rollout and performance features address production work. The commercial distinction is utilization: reserved GPU-hours can suit steady demand, while token billing avoids idle reservations for irregular traffic. Regional deployment options depend on plan and configuration. Isolation does not remove provider-access questions, and API compatibility does not guarantee portable operational semantics. Validate performance on the actual model and request distribution.
Together dedicated model inference ↗https://www.together.ai/dedicated-model-inference · Together EU deployment requirements ↗https://support.together.ai/articles/8079447813-eu-data-centers-and-dedicated-model-deployment
Fireworks AI
- Ownership / organization
- Fireworks AI
- Control point
- Inference engine and managed fleet operations
Fireworks can operate its serving stack on customer-owned GPUs through Sovereign. This separates infrastructure ownership from inference software and operations. The customer manages the network edge; Fireworks documents an auditable setup credential and scoped component identities. BYOC therefore does not mean the vendor has no access. Optional cloud overflow changes the data boundary. Its advertised throughput and utilization gains are vendor claims, not a fair cross-provider ranking. Defensibility depends on repeatable workload efficiency and operational delivery.
Fireworks Sovereign shared responsibility ↗https://fireworks.ai/product/sovereign · Fireworks enterprise controls and retention exceptions ↗https://fireworks.ai/solutions/for-enterprise
Baseten
- Ownership / organization
- Baseten
- Control point
- Custom-model serving and deployment
Baseten targets AI product teams needing managed model deployment across cloud capacity. Its November 5, 2026 security practices distinguish shared infrastructure with logical segregation, dedicated clusters, self-hosted deployment and hybrid operation. Those are materially different trust boundaries. Analyst interpretation: custom integration and reliable operations can differentiate the platform beyond a model catalogue. Buyers must check workload placement, provider access and product-specific compliance scope; a certification covering one offering should not automatically be extended to another.
Baseten security practices ↗https://www.baseten.co/security-practices
Databricks
- Ownership / organization
- Databricks
- Control point
- Data, model and operational application platform
Databricks competes around enterprise data and the workflows that turn it into models and applications. MosaicML added training capabilities; Tabular strengthened table-format interoperability; Neon technology powers Lakebase operational Postgres. Analyst interpretation: data integration and accumulated workflow state create switching costs beyond model weights. Governance reduces fragmentation but does not replace runtime authorization. A catalogue permission cannot establish that an agent may execute a specific external payment, and a stored trace does not prove that the payment actually settled.
Databricks–MosaicML acquisition agreement ↗https://www.databricks.com/company/newsroom/press-releases/databricks-signs-definitive-agreement-acquire-mosaicml-leading-generative-ai-platform · Databricks–Tabular completed acquisition update ↗https://www.databricks.com/blog/databricks-tabular · Databricks Lakebase launch powered by Neon ↗https://www.databricks.com/company/newsroom/press-releases/databricks-launches-lakebase-new-class-operational-database-ai-apps
A deployment can combine several competitors
These layers often complement rather than replace each other. A company can rent GPU capacity from a specialist cloud, run an inference vendor’s stack on it, and keep data governance in an enterprise platform. The resulting invoice has several suppliers, but the user experiences one service. Map each failure to an owner: unavailable capacity, stalled model rollout, inaccessible evidence, slow generation and unauthorized action are different incidents. Integration can lower coordination cost, but a single commercial contract does not make all components equally reliable. For startups, the attractive opening is a boundary that repeatedly fails across suppliers and is expensive for the buyer to own internally—not simply another dashboard showing GPU utilization.
Technical detail: prefill, decode and the KV budget
Prefill processes the input and builds key/value states; decode repeatedly reads prior states while generating new tokens. Their bottlenecks depend on workload and hardware, so a single throughput number hides the serving tradeoff. Continuous batching can improve utilization, while excessive concurrency increases queueing and memory pressure. Separating prefill and decode permits different scheduling, but introduces KV transfer and network costs. A provider selling fast tokens must still satisfy time-to-first-token and inter-token latency at the required concurrency.
For a conventional cache, payload bytes are approximately 2 × layers × active sequences × cached tokens per sequence × KV heads × head dimension × bytes per element. With 32 layers, 8 sequences, 4096 tokens, 8 KV heads, dimension 128 and 2-byte elements, the cache payload is 4 GiB. This synthetic example excludes weights, activations, allocator fragmentation and metadata. Grouped-query attention reduces KV heads; prefix sharing helps only when reusable prefixes and isolation rules permit it. Capacity planning must include peak occupancy, not just the average request.
Capacity, tokens and accepted-task economics
Capacity businesses face equipment, power, delivery and utilization risk. Inference software competes on useful throughput under latency and quality constraints. Data platforms compete on integration and workflow retention. A synthetic example shows why price alone misleads: provider A costs $20 for 1,000 requests and 800 acceptable completions, or $0.025 each. Provider B costs $16 but yields 500 acceptable completions, or $0.032 each. B is cheaper per request and more expensive per accepted task. Include idle reservations, retries, data movement and review labor before claiming savings; hold the task mix, quality threshold and deadline constant.
A second synthetic example separates capacity risk from software margin. A reserved worker costs $240 per 24-hour day. At 80% productive utilization, its 19.2 useful hours cost $12.50 each; at 25%, six useful hours cost $40 each. The reservation price is unchanged. This is not a GPU price quote, and busy time alone does not establish useful output. It explains why utilization commitments, workload diversity and idle-capacity ownership matter in comparing a capacity owner with a software operator using customer-owned GPUs. Private gross margins are not inferred from this illustration.
Where defensibility and unsolved problems remain
Open source shifts the profit pool rather than eliminating it. NVIDIA’s four Run:ai components commoditize part of the scheduling checklist; community distribution does not include enterprise support. Durable opportunities may lie in heterogeneous-fleet recovery, policy-aware multi-provider routing, cost attribution and portable evidence. A router must respect region, retention, model quality and failover permissions, not merely select the cheapest endpoint. Open weights do not make IAM semantics, evaluation history or deployment state portable. Buyers should demand ownership boundaries and export paths. Investors should ask which recurring operational failure has a clear budget owner, and whether the platform vendor can bundle the proposed fix.
4. Electricity data: scale and uncertainty
The IEA's 2026 update puts global data-centre electricity consumption at 485 TWh in 2025 and projects approximately 950 TWh in 2030 in its central outlook. These are not AI-only figures. The projected increase is roughly 1.96×, not an observation that consumption has already doubled. The report also identifies physical supply constraints, including HBM and power-system bottlenecks. IEA source, scope and assumptionshttps://www.iea.org/reports/key-questions-on-energy-and-ai/executive-summary.
Electricity demand is a useful scale indicator, not an AI capability metric. Lower energy per task can coexist with higher total consumption if adoption grows or tasks become more computationally intensive. A query that initiates repeated planning, retrieval and verification cannot be compared with a single short completion simply by counting user messages.
For a facility, distinguish IT load from total facility load. Power usage effectiveness is total facility energy divided by IT energy over the same measurement interval; it does not describe model quality. A lower value cannot compensate for wasted computation. For deployment decisions, measure energy per accepted task, peak power, cooling constraints and grid availability separately. A global TWh forecast cannot tell you whether a particular site can connect next quarter.
5. The technical frontier: what to optimize, and what can fail
Prefill and decode disaggregation
Prefill processes the input sequence and initializes the cache; decode repeatedly produces another token. Their operating points differ. DistServe studies separating the phases and selecting resource allocations around time-to-first-token and time-per-output-token constraints. Its original preprint appeared January 18, 2024; this reading pins arXiv v3, June 6, 2024. Abstracthttps://arxiv.org/abs/2401.09670v3 · PDFhttps://arxiv.org/pdf/2401.09670v3.
The useful inference is not “separation is always faster.” Moving cache state costs bandwidth and coordination. With short prompts or a constrained network, colocated execution can be simpler and competitive. Measure cache-transfer time, queue delay and both latency objectives. Hold offered load constant; a configuration that rejects difficult requests must not appear faster merely because it serves an easier subset.
KV cache as a first-class resource
Cache memory grows with active sequence tokens. Grouped-query attention reduces the number of key/value heads relative to query heads, changing memory requirements. Prefix caching can reuse shared prompt state, but tenant boundaries and cache keys must prevent unauthorized reuse. Offloading may increase capacity while adding transfer latency. These are separate interventions: benchmark each rather than calling all of them “cache optimization.”
Mixture of experts and interconnects
An MoE layer routes tokens to selected expert networks. Sparse activation reduces the arithmetic used per token relative to activating every expert, but weights still need storage and routing creates communication. Skewed expert demand can create hot spots. Compare expert placement, token imbalance, communication overlap and tail latency. Counting active parameters alone does not determine serving cost.
Quantization and speculative decoding
Lower precision can reduce memory traffic, but scales, outliers and unsupported kernels complicate the result. Verify downstream task quality and numerical behavior, not only model loading. Speculative decoding proposes multiple tokens and verifies them with a target model; useful acceleration depends on acceptance, proposal cost and verification efficiency. Keep the target distribution and implementation assumptions explicit. A fast draft with low acceptance can slow the full system.
Training reliability and elastic scheduling
Large training jobs coordinate many devices. More devices create more opportunities for hardware or software interruption. Checkpoint frequency trades useful computation against recovery exposure; storage contention can make frequent checkpoints expensive. For a shared cluster, optimize fairness and queue wait as well as accelerator occupancy. High utilization from low-priority work is not success if urgent jobs repeatedly miss deadlines.
Acquisitions, capital and strategic consequences
Acquisitions move the control point
CoreWeave announced its W&B agreement on March 4, 2025 and closed on May 5; OpenPipe was announced September 3 and its filing associates closing with September 5. These moves connect capacity with development and post-training. Databricks announced MosaicML on June 26, 2023 and closed July 19; the announced approximately $1.3 billion included retention packages. Its Tabular blog confirms completion June 7, 2024. Lakebase launched June 11, 2025 using Neon technology; this review does not establish Neon’s exact legal closing day. The pattern suggests integration around recurring workflows, not proof of successful cross-selling. Announcement, closing and actual product integration are separate milestones.
Primary product and transaction sources
- NVIDIA: full Run:ai platform open-source release ↗https://developer.nvidia.com/blog/nvidia-runai-is-now-open-source-as-part-of-nvidia-dsx-ai-factory-platform ·
- NVIDIA Dynamo ↗https://www.nvidia.com/en-us/ai/dynamo/ ·
- Amazon Bedrock: data protection and shared responsibility ↗https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html ·
- Amazon Bedrock: invocation permissions ↗https://docs.aws.amazon.com/bedrock/latest/userguide/inference.html ·
- Microsoft Foundry architecture ↗https://learn.microsoft.com/en-us/azure/foundry/concepts/architecture ·
- Microsoft Foundry network isolation ↗https://learn.microsoft.com/en-us/azure/foundry/how-to/configure-private-link ·
- Google Cloud service-specific data residency ↗https://cloud.google.com/terms/data-residency ·
- Google Cloud: securely deploying AI ↗https://cloud.google.com/blog/products/identity-security/mastering-secure-ai-on-google-cloud-a-practical-guide-for-enterprises/ ·
- CoreWeave completes Weights & Biases acquisition ↗https://www.coreweave.com/blog/coreweave-completes-acquisition-of-weights-biases ·
- CoreWeave 2025 Form 10-K ↗https://www.sec.gov/Archives/edgar/data/1769628/000176962826000104/crwv-20251231.htm ·
- Nebius Cloud 3.6 governance release ↗https://nebius.com/newsroom/nebius-ai-cloud-3-6-strengthens-developer-experience-and-governance-for-production-operations ·
- Nebius–Microsoft agreement ↗https://www.sec.gov/Archives/edgar/data/1513845/000110465925088312/tm2525580d1_6k.htm ·
- Together dedicated model inference ↗https://www.together.ai/dedicated-model-inference ·
- Together EU deployment requirements ↗https://support.together.ai/articles/8079447813-eu-data-centers-and-dedicated-model-deployment ·
- Fireworks Sovereign shared responsibility ↗https://fireworks.ai/product/sovereign ·
- Fireworks enterprise controls and retention exceptions ↗https://fireworks.ai/solutions/for-enterprise ·
- Baseten security practices ↗https://www.baseten.co/security-practices ·
- Databricks–MosaicML acquisition agreement ↗https://www.databricks.com/company/newsroom/press-releases/databricks-signs-definitive-agreement-acquire-mosaicml-leading-generative-ai-platform ·
- Databricks issuer release: MosaicML acquisition completed ↗https://www.prnewswire.com/news-releases/databricks-completes-acquisition-of-mosaicml-301881059.html ·
- Databricks–Tabular completed acquisition update ↗https://www.databricks.com/blog/databricks-tabular ·
- Databricks Lakebase launch powered by Neon ↗https://www.databricks.com/company/newsroom/press-releases/databricks-launches-lakebase-new-class-operational-database-ai-apps ·
- CoreWeave announces OpenPipe agreement ↗https://www.coreweave.com/news/coreweave-to-acquire-openpipe-leader-in-reinforcement-learning ·
- CoreWeave 2025 Q3 filing: OpenPipe closing ↗https://www.sec.gov/Archives/edgar/data/1769628/000176962825000062/crwv-20250930.htm ·
Source and maintenance notes
Primary links are placed beside the claims they support. Acquisition amounts retain their original basis; undisclosed values remain undisclosed. Product specifications and release statements are vendor disclosures, not comparative evidence from Token Lab. The diagrams and reference calculations are original educational material. Corrections should identify the section, source and proposed replacement; a source-access date is not a paper publication date.
All frontier research · Top Company Frontier · Editorial standards
Practice and verification
Practice: infrastructure under failure and load
Stop the model after a proposal: restart must preserve pending approval. Stop the workflow after tool acknowledgment: reconcile before retrying. A transcript alone is not a durable checkpoint.
Inject timeouts, invalid tool arguments and model-version mismatch. Require bounded attempts, explicit states and identical authorization boundaries on primary and fallback routes; compare a safe abstention baseline at matched budgets.
Replay representative long contexts, slow tools, cancellations, bursts, cold caches and failures long enough to expose memory leaks or growing queues. Record hardware, precision, model, scheduler and software versions. Compare throughput at matched p95 latency and acceptance; define rollback thresholds first.
Exceed active and queue limits: verify explicit overload responses, bounded memory and progress for a second tenant. Vary prompt length, output length and prefix reuse independently, separating cold and warm caches and checking isolation. Report throughput, tail latency, peak memory and task acceptance.
Recover from region failure, quota exhaustion, slow retrieval and restart during approval. Verify cancellation, revocation and idempotency after recovery. State what was measured locally versus which distributed guarantees remain unproven.
7. Reproduce a memory and cost budget
This is an original teaching calculation, not a vendor benchmark. Assume B active sequences, T cached tokens per sequence, L transformer layers, Hkv key/value heads, head dimension d and b bytes per stored value. K and V each have shape [B, Hkv, T, d] per layer. Ignoring allocation overhead and sharding:
For B=8, T=8192, L=32, Hkv=8, d=128 and b=2, the cache occupies 8 GiB. At 32768 tokens it occupies 32 GiB. Changing Hkv from 8 to 32 also gives 32 GiB at the shorter length. GiB means 2³⁰ bytes; device specifications expressed in GB need conversion before subtraction.
This is logical total cache size, not a claim about per-device allocation. Add weights, temporary activations, runtime reservations, fragmentation and any duplicated/sharded state. Weight quantization does not automatically quantize the cache. In a mixed-length batch, replace B×T with the sum of each active sequence's cached length.
At an illustrative $8 per hour and two accepted requests per second, the capacity cost is $0.001111 per accepted request. One accepted request per second doubles it. These hypothetical prices exclude network, storage, labor and idle reservations. “Accepted” should include a task-quality check and the agreed latency constraint; otherwise a faster but less useful system gets an unfair advantage.
Local capacity experiment
KV cache: 8.000 GiB for the stated eight-sequence configuration.
Predict the result before doubling the context length. No inputs leave your browser.
Download infra-capacity.py. Python 3.10+, standard library only; deterministic arithmetic, no random seed, network requests or model API. Run python3 infra-capacity.py.
"""Teaching arithmetic, not a GPU benchmark. Python 3.10+, standard library."""
from math import isclose
def kv_gib(batch, tokens, layers, kv_heads, head_dim, bytes_per_value):
values = (batch, tokens, layers, kv_heads, head_dim, bytes_per_value)
if any(not isinstance(x, int) or isinstance(x, bool) or x <= 0 for x in values):
raise ValueError("All dimensions and bytes per value must be positive integers")
return 2 * batch * tokens * layers * kv_heads * head_dim * bytes_per_value / 2**30
def cost_per_success(hourly_cost, accepted_per_second):
if hourly_cost < 0 or accepted_per_second <= 0:
raise ValueError("Nonnegative cost and positive accepted throughput required")
return hourly_cost / (3600 * accepted_per_second)
if __name__ == '__main__':
small = kv_gib(8, 8192, 32, 8, 128, 2)
long = kv_gib(8, 32768, 32, 8, 128, 2)
assert small == 8 and long == 32
assert kv_gib(8, 8192, 32, 32, 128, 2) == 32
assert kv_gib(8, 8192, 32, 8, 128, 1) == 4
assert isclose(cost_per_success(8, 2), 8/7200)
assert cost_per_success(8, 1) == 2 * cost_per_success(8, 2)
for args in [(0, 1, 1, 1, 1, 2), (1, -1, 1, 1, 1, 2)]:
try:
kv_gib(*args)
except ValueError:
pass
else:
raise AssertionError('Invalid dimensions accepted')
print(f'KV: 8k = {small:.1f} GiB; 32k = {long:.1f} GiB')
print(f'Cost per accepted request: ${cost_per_success(8, 2):.6f}')
print('8 checks passed')
Expected output:
KV: 8k = 8.0 GiB; 32k = 32.0 GiB
Cost per accepted request: $0.001111
8 checks passedA real benchmark protocol
Pin the model checkpoint, tokenizer, runtime commit, kernels, precision, hardware topology and power configuration. Replay the same timestamped request trace under cold and warm caches. Sweep offered load until an SLO fails. Report arrivals, successful completions, errors, quality, p50/p95/p99 latency and throughput together. Repeat independent runs and publish spread, not just the best run.
Ablate one factor at a time: colocated versus separated phases; cache reuse off versus on; full versus reduced precision. Compare equal hardware budgets first, then equal cost and power budgets if available. Do not drop out-of-memory failures from the denominator. This lesson verifies arithmetic only; it does not run a GPU service or reproduce a paper's headline speedup.
8. Open problems and original interview practice
Unresolved problems: managing long-lived agent state without runaway memory; sharing processors without noisy-neighbor latency; predicting MoE traffic; reliable heterogeneous execution; reducing recovery loss; and reconciling interactive demand with power constraints. Each problem joins at least two layers. A promising research question specifies both an improvement and the condition under which it stops helping.
For example: can a scheduler reduce p99 latency without reducing accepted throughput for another tenant? Can a cache-placement policy save memory without increasing disclosure risk? Can a hardware-independent intermediate representation preserve performance across kernels rather than merely compile? These are measurable questions. “More efficient infrastructure” is not yet an experiment.
The following are original simulations, not reported company interview questions.
1. Why can a 2× faster kernel leave end-to-end latency almost unchanged?
First hint: Separate queue, computation, communication and tool time.
Answer and analysis
If the improved kernel was only 10% of total latency, halving it changes total time from 1 to 0.95: approximately 1.053× speedup. Queueing can further change the result under load. Measure the critical path before optimizing.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
2. A model fits in memory. Why can inference still fail?
First hint: Weights are not the only resident state.
Answer and analysis
Concurrent KV caches, activation workspace and allocator reservations can exhaust memory. Calculate each term with units, inspect actual peak allocations, then test mixed sequence lengths and admission control. Model parameter count alone is insufficient.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
3. When does prefill/decode separation lose?
First hint: Account for the state crossing the boundary.
Answer and analysis
Transfer and coordination can dominate for short requests or slow links. Sweep prompt length and offered load with fixed SLOs, include network contention, and compare equal resources. A paper’s maximum speedup is not a universal prediction.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
4. What would make a cheaper accelerator more expensive per successful request?
First hint: Include software, utilization and failures.
Answer and analysis
Lower rental cost can be offset by unavailable kernels, compilation, retries, low quality or idle reservation. Compare accepted goodput and total ownership costs using the same workload, not hardware price alone.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
Continue with inference systems, the attention project and the companion AI security survey. These topics meet at cache isolation, scheduling privileges and operational reliability.
9. Frontier research agenda: five falsifiable projects
The following are Token Lab's proposed research directions, not claims that these methods have already succeeded or are novel. Search the literature before claiming novelty. Each proposal starts with an engineering failure and ends with evidence that would count against the idea. Select one narrow workload for a first PoC; broad claims require replication across tasks, implementations and operating conditions.
1. Long-context capacity under a latency budget
Why existing approaches are insufficient: A scheduler that treats each request as an identical job ignores the memory-time footprint of a long context. Reserving only current cache size can over-admit work; reserving maximum context can waste capacity.
Testable hypothesis: A reservation policy using predicted remaining tokens can improve accepted throughput at a fixed out-of-memory risk and latency SLO.
Baselines and fair comparison: Compare current-size reservation, worst-case reservation and a simple length-bucket policy before adding a learned predictor. Use the same arrival trace and cap total memory.
Measurements: Report accepted goodput, rejection rate, prediction calibration, out-of-memory events and p99 latency by length bucket. Include deliberately underestimated output lengths.
What would falsify the claim: Reject the hypothesis if the apparent gain comes from rejecting long requests, increasing failures, or obtaining future token lengths unavailable at admission. A useful system should degrade gracefully when predictions are wrong.
2. Disaggregation when the network is shared
Why existing approaches are insufficient: Dedicated-link benchmarks can hide contention. Prefill/decode separation moves a large cache between stages while training, storage and other tenants may use the same fabric.
Testable hypothesis: Topology-aware placement and bounded transfers can preserve the gain under realistic cross-traffic.
Baselines and fair comparison: Compare colocated serving, naive separation and bandwidth-aware placement. Sweep prompt length, batch size and network contention without changing total accelerators.
Measurements: Measure transfer bytes and duration, queue wait, TTFT, TPOT and goodput. Report the crossover region where separation stops helping, not only the best configuration.
What would falsify the claim: Reject if the method needs dedicated bandwidth that the baseline was denied, or reduces one latency metric by violating the other. A robust result includes topology changes and link degradation.
3. Portable performance across accelerator families
Why existing approaches are insufficient: A model can compile on two systems while one falls back to slow kernels or numerical approximations. Portability of source code and portability of performance are different goals.
Testable hypothesis: A small set of workload-aware kernel choices can retain quality and predictable performance across two supported platforms.
Baselines and fair comparison: Use the default runtime, hand-tuned platform paths and the proposed portable selection policy. Pin model weights and compare identical task outputs with appropriate numerical tolerances.
Measurements: Report compile time, fallback operations, end-to-end cost, energy when measurable, quality and maintenance effort. Include dynamic sequence shapes and an unseen model variant.
What would falsify the claim: Reject if gains require per-model manual retuning comparable to the specialized baseline, or if quality drops explain throughput. Document unsupported operators instead of silently excluding them.
4. Power-aware service without exporting the cost to users
Why existing approaches are insufficient: Optimizing energy in isolation can defer work until queues become unacceptable; optimizing latency alone can create demand peaks. Long reasoning workloads further vary per-request compute.
Testable hypothesis: A controller that jointly chooses admission, batching and power limits can reduce energy per accepted task while maintaining a declared latency envelope.
Baselines and fair comparison: Compare fixed power, static batching and an SLO-only scheduler. Replay identical bursts; keep task quality fixed and include controller overhead.
Measurements: Measure joules per accepted task, wall power, p95/p99 latency, throughput, rejection and recovery after a burst. State whether power is device-only or facility-wide.
What would falsify the claim: Reject if saved energy is just unserved work, if the measurement omits another device, or if gains vanish outside a smooth synthetic trace. Grid-level claims require evidence beyond GPU telemetry.
5. Persistent agent state across failures and tenants
Why existing approaches are insufficient: An agent session may outlive a process, move across hardware and hold private context. Cache migration and prefix reuse create both a systems optimization and an authorization problem.
Testable hypothesis: A versioned state protocol with tenant-scoped keys can support recovery without exposing stale or unauthorized context.
Baselines and fair comparison: Compare no reuse, process-local reuse and shared reuse with explicit ownership. Inject restarts, revocations and concurrent updates.
Measurements: Measure recovery time, recomputation, cache hit rate, accepted throughput and isolation violations. Test correctness of outputs and policy decisions after each migration.
What would falsify the claim: Reject if recovery works only without revocation, if a cross-tenant key collision leaks state, or if the method assumes all workers are equally trusted. This is a joint infrastructure/security project.
From a two-week PoC to a defensible report
In the first stage, write the system boundary, assumptions and failure criterion before implementing the proposed method. Build the simplest baseline and a reproducible fixture. In the second stage, add one intervention and run controlled sweeps; retain failed runs and explain exclusions. In the final stage, test an unseen workload or attack family and write a limitations section. Separate what your experiment measured from what you hope will generalize.
A useful report contains a configuration manifest, data-generation procedure, code commit, exact commands, raw observations, aggregation script and a table of threats to validity. Use multiple seeds when randomness matters; repeated deterministic arithmetic is not independent evidence. Publish negative results: the boundary where a technique stops working is often the most transferable lesson.