AI infrastructure in 2026: from fast chips to useful, reliable capacity
Follow a request through admission, prefill, KV memory, decoding, tools and recovery; connect capacity and cost to frontier hardware and market evidence.
来源与版本详情
Contents · lifecycle and reference chapters
本文目录
基础设施:跟随请求穿过系统
以下原创案例贯穿一个发票助手:它读取租户范围内的资料并提出付款建议,实际付款需要单独且精确的授权。场景与实验均为合成教学材料,不代表真实部署。
沿着一个请求理解发票Agent基础设施
员工问:“为什么INV-42逾期?如果满足条件,准备下一步支付动作。”助手查询本租户文档与发票状态,解释并提出动作;执行支付仍需单独、精确的批准。这是合成教学系统,不是任何厂商的真实架构。基础设施必须承载持久业务流程,而不只是返回流畅文字。
请求跨越认证网关、工作流、检索存储、推理端点、工具网关与业务系统。各层失败语义不同:读请求通常可重试,超时的支付提交却必须先核对结果。模型进程不能拥有事务真相。持久工作流保存待办步骤、已用证据和提交状态。
选择硬件前先定义负载契约:到达率及突发、输入输出token分布、每任务模型调用次数、检索扇出、工具延迟与用户截止时间。交互回答与后台核对任务分开。“每秒10个请求”不足以规划容量,因为200-token回答和8次长上下文调用的工作量不同。恢复设计还要决定哪些状态能直接恢复、哪些必须重新执行非确定性推理;聊天记录本身不是检查点。
阶段1:准入、上下文组装与资源预算
网关在入队前认证调用者并解析租户。限流保护共享服务,准入控制则判断当前系统能否满足截止时间。早拒绝或延后,往往好过接入无界队列后在人已离开时才回答。
上下文要为系统指令、工具schema、证据、历史和最大输出预留预算。模型窗口是上限,不是必须填满的目标。检索只选相关且获授权的材料;裁剪必须保留必要策略与出处,不能为装更多文档而删掉批准要求。
早期准入用估算token,实际用量用于结算核对。除了请求数,还要限制每租户并发与资源占用;少数超长请求也能耗尽内存。取消应传到检索和推理,避免用户离开后继续消耗容量。合成示例允许24个活跃生成、48个排队交互请求,队列期限2秒;这不是推荐默认值,而是明确过载处理边界的示例。公平性要求第二个租户仍能取得进展,平均延迟不能替代这一约束。
阶段2:Prefill、解码、KV内存与调度
检索之后,推理系统先处理提示词,再逐步生成token。Prefill与解码的执行模式不同:长提示词影响首token到达时间;首token出现后,长回答仍持续占用解码容量。首token延迟、token间延迟、末token时间和整个业务任务完成时间需要分别定义。
常规注意力缓存的教学估算是:2×层数×KV头数×头维度×缓存token数×每元素字节数,再对所有驻留序列求和。32层、8个KV头、头维度128、8192个token、每元素2字节时,单序列K和V占1 GiB;16条需要16 GiB。权重、激活、分配器、碎片及实现状态另算。分组查询注意力不能把查询头数当成KV头数。
批处理通过共享执行提高利用率,但等待成批会增加延迟。各请求完成时间不同,调度器必须管理动态活跃序列及内存。把长prefill切成块可能改善decode响应,具体收益取决于内核、硬件和流量。不能从均匀合成批量直接推断真实混合负载。
vLLM前缀缓存文档https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/说明如何复用共同前缀的计算。它减少重复prefill,并不消除生成新输出的工作。冷热流量和权限隔离是理解收益的前提。
采购容量之前先区分瓶颈。Prefill对提示词执行注意力和前馈计算;解码则反复读取权重与缓存,每条序列每步只产生少量新工作。瓶颈取决于批量、上下文、内核和硬件。张量并行分割模型计算,却把通信放到关键路径;流水线并行分割网络层,需要足够的调度工作隐藏阶段空泡。复制副本只有在单副本能放下模型且流量可均衡时,才能增加独立服务容量。
分离prefill与decode可以隔离资源需求,却引入KV传输依赖。以上1 GiB缓存为例,理想25 GiB/s链路仅传字节就至少需要40毫秒,协议、序列化、争用和交接还会增加耗时。这是算术下界,不是实测。当传输和闲置容量的成本超过减少的相互干扰时,拆分就不划算。量化也需要兼容内核才能兑现存储或带宽收益;位宽下降本身不能证明服务更快或任务质量不变。
阶段3:路由与降级会改变系统行为
简单查询可走较便宜模型,争议发票可走能力更强的模型,但路由器也会错:复杂问题被判简单,或所有问题都走昂贵路径。运行记录应保留路由决定、模型、提示词版本与预算,让回归可归因。
供应商故障后的替代模型可能在工具语法、窗口、指令遵循和拒答上不同。协议适配能统一语法,却不自动保证业务行为等价。启用前要有行为证据,替代路径仍经过相同授权与批准边界。
熔断减少对坏依赖重复调用;退避与重试预算限制流量放大,但它们不能证明副作用是否发生。模型调用重试与业务动作重试要分开:生成新提议不等于可以再次支付。检查点记录逻辑操作及状态,而不只是自然语言消息。可靠降级需要说明主路径、替代路径和安全不回答各自的语义、成本与失败条件,不能只比较供应商可用率。
阶段4:容量规划连接队列、成本与有效工作
容量规划从服务曲线开始:输入负载升高时,可接受吞吐和延迟如何变化。接近饱和时,即使平均GPU利用率看似正常,排队延迟也可能陡增。系统可能生成更多token,却减少截止时间内完成的请求,因此吞吐必须附带质量和延迟约束。
稳定系统中的Little定律把平均在途工作量与到达率和平均停留时间联系起来。每秒完成8个请求、平均端到端3秒,则平均在途24个。这是稳定假设下的记账关系,不证明24个并发槽位能满足p99截止时间。突发、服务时间方差与相关性故障仍然重要。
成本账本必须统一单位。合成示例中,1000个请求花费20美元推理与工具费用,800个达到验收定义,则每请求0.02美元、每合格任务0.025美元。便宜路由把支出降至16美元,却只有500个合格任务时,后者反而升至0.032美元。业务总成本还应明确计入预留容量、空闲、存储、网络及人工审核。
一个任务不等于一次模型请求。假设每秒到达10个用户任务,平均每个调用模型3次,则重试前模型层已经承受30次/秒。如果各调用独立地有10%概率需要额外一次尝试,规划值变成33次/秒。这是合成的单次重试模型,相关性故障与递归Agent循环不满足假设。必须同时限制单任务及全服务的重试量,避免恢复流量比正常工作还大。
自动扩容要考虑启动时间和状态。如果新副本加载权重需要几分钟,持续几秒的突发只能靠现有余量、队列或准入拒绝处理,不能靠事后才就绪的副本。应按系统需要使用队列年龄、活跃token和实测容量,而非只看GPU利用率。对于有截止时间的服务,只有在质量和租户公平性不恶化时增加按时完成任务,额外容量才真正有用。
阶段5:发布、恢复与可解释的运行模型
一次发布不只有权重。Tokenizer、提示模板、工具schema、索引、量化、推理引擎和路由策略都会影响行为;每项都应有版本和兼容记录。复现旧事故可能需要历史文档快照与工具响应,而不是只有模型名。
离线验证后,先用不能执行动作的影子流量,再做有限真实流量canary。流量随时间改变,因此比较同期基线。回滚应恢复工作流与schema兼容性;回滚容器不会撤销已提交支付。没有明确批准和补偿策略时,不可把不可逆动作放进实验。
工作流记录和缓存应有不同恢复时间、数据丢失目标。KV缓存可付出成本重算,丢失批准证据或已提交动作状态却是另一类故障。恢复架构必须保留取消、撤销与幂等语义。工程交付应包含负载模型、容量边界、依赖恢复计划及把资源变化联系到任务结果的发布报告。硬件与公司选择要据此判断:只有解决实测瓶颈、降低总体风险和成本才有价值。
How the four disciplines fit together
Security determines which actions are permitted. Infrastructure executes work within capacity and failure constraints. Observability preserves evidence of what happened. Evaluation tests whether a change improves the outcome under a declared contract. None substitutes for the others.
机制、研究与市场背景——以下参考材料仍为英文
The following chapters provide the mathematical examples, experiments, research papers and dated company evidence behind the operational story. Return to the lifecycle when deciding where each mechanism belongs.
1. The unit of progress is a completed workload
An AI infrastructure stack turns electricity, silicon, software and operational labor into training runs or useful responses. A faster accelerator is only one input. A customer experiences the time to a correct answer; an operator pays for reserved capacity, idle time, retries and failures as well as successful computation. The important question is therefore not “which chip has the largest number?” but “what bottleneck prevents this workload from meeting its service objective?”
Definitions. An accelerator is a processor optimized for selected computations. HBM is high-bandwidth memory close to the accelerator; it is not interchangeable with host RAM or remote storage. Scale-up interconnects connect processors within a tightly coupled system; scale-out networking joins systems. A service-level objective (SLO) specifies a measurable target such as a latency percentile. Goodput is the rate of useful completions satisfying specified constraints, rather than every generated token or attempted request.
Before this lesson: understand that matrix multiplication creates reusable parallel work, while autoregressive generation has a dependency between output steps. Understand a byte, a second and a watt: power measures a rate of energy use, and capacity in GiB says nothing by itself about transfer rate. Finally, distinguish average latency from a tail percentile. An average can improve while the slowest users experience worse service. Review attention and KV-cache correctness if those distinctions are unfamiliar.
Learning goals: draw the system boundary; calculate a cache budget with units; identify a misleading benchmark comparison; explain why an acquisition adds an engineering capability; and design a repeatable serving experiment. The company map below is selective, not a complete procurement recommendation or a claim that every listed product is available in every region.
2. Read the stack from the workload down
Start at the application. A short autocomplete request, a long-document question and an agent executing twenty tool steps have different prompt lengths, output lengths and interruption patterns. A speech assistant needs predictable delay; an overnight research job may accept queuing. One aggregate “tokens per second” metric conceals these differences. Record the request distribution before choosing a runtime or hardware platform.
The serving runtime schedules requests and manages state. Continuous batching can admit new work as other requests finish, but mixing a large prompt with latency-sensitive decoding can create interference. Cache allocation determines how many active sequences fit. Parallelism then distributes weights and work across processors. If a model requires frequent collectives or expert routing, communication may erase the benefit of additional compute.
The rack supplies the physical constraints: memory channels, interconnect topology, cooling and electrical delivery. A training job also needs dataset throughput and durable checkpoints. A data pipeline can starve expensive processors; a recovery procedure can waste a large fraction of a run even when individual kernels are fast. Capacity planning must include failed jobs and reloading weights, not just the steady state.
Our engineering interpretation is that differentiation increasingly comes from coordinating these layers. An efficient kernel does not repair a slow scheduler; a faster network does not fix an inaccurate low-precision model; an available GPU does not imply an energized rack. This is why infrastructure is both a distributed-systems problem and an industrial deployment problem.
行业格局:控制点、买方与竞争
行业证据核对于 2026 年 11 月 8 日。产品文档只能证明供应商提供了相关能力,不能证明独立实测效果。竞争判断与机会分析为 Token Lab 的分析;标为合成的算例不是公司实测数据。
AI基础设施:谁掌握容量、工作流与责任?
行业核查日期为2026年11月8日。官方来源用于确认产品范围和交易里程碑,竞争结论属于本站分析。这不是市场份额排名,也不是独立厂商基准。无日期产品页按核查日读取;有日期来源标注发表或更新时间。
市场是一条责任链
沿着一个企业助手,从存储文档追踪到任务完成:数据平台提供获准访问的证据,GPU云提供容量,推理运营商把容量变成响应,应用决定允许采取什么行动。每层解决不同瓶颈。把所有厂商都当成可互换的AI云,会掩盖买方、利润结构与安全责任。以下竞争判断是分析推论,不是实测市场份额或投资建议。私有端点可以保护传输,但权限过大的应用仍可能泄露另一租户的记录。
参考架构:在延迟与显存约束下提供推理服务
图中选择 Prefill/Decode 分离式服务以暴露其成本,并不要求所有部署都采用它。准入阶段在排队前预留 token 和并发预算。调度器把输入处理交给 Prefill,再把产生的 KV 状态传给兼容的 Decode 工作节点。版本仓库提供固定的模型制品。两个资源池输出运行指标;取消与截止时间必须传递到工作节点,才能释放预留资源。
只有专业化收益超过序列化、网络与协调开销时,分离才有价值。同机服务免去这次传输,小负载可能更合适。共享前缀必须匹配模型、缓存格式和隔离边界。队列长度与等待时间需要上限,扩容无法瞬间修复饱和系统。分别记录排队时间、首 token 延迟、生成节奏和 KV 峰值占用,避免把更便宜的容量当成更好的请求完成效果。
公司与产品定位
这是选择性行业地图,不是市场份额排名或供应商认证。覆盖范围取决于版本、集成、部署与执行模式。
NVIDIA
- 所有权/组织
- NVIDIA
- 控制点
- 加速器、互联与编排
NVIDIA向平台团队提供把GPU变成有效服务所需的互补硬件与软件。Dynamo面向分布式推理,Run:ai面向工作负载分配。2026年10月13日,NVIDIA将完整Run:ai平台开源,包含四个模块化组件。分析判断:通用调度功能面临价格压力,集成与运维支持仍有价值。资源配额与GPU共享既不建立应用授权,也不能证明租户之间具有强隔离。
NVIDIA: full Run:ai platform open-source release ↗https://developer.nvidia.com/blog/nvidia-runai-is-now-open-source-as-part-of-nvidia-dsx-ai-factory-platform · NVIDIA Dynamo ↗https://www.nvidia.com/en-us/ai/dynamo/
AWS
- 所有权/组织
- Amazon
- 控制点
- 云与托管模型API
AWS服务于已经使用其身份、计费与运维体系的企业。Bedrock将托管模型访问与IAM控制结合,其战略优势包含采购与集成,而不只是模型速度。安全采用共同责任模式。客户仍需限定凭据、数据访问和下游工具权限。身份获准调用模型,并不意味着它能执行模型建议的所有动作。应检查具体部署控制,而不是把云品牌当成完整的安全策略。
Amazon Bedrock: data protection and shared responsibility ↗https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html · Amazon Bedrock: invocation permissions ↗https://docs.aws.amazon.com/bedrock/latest/userguide/inference.html
Microsoft Foundry
- 所有权/组织
- Microsoft
- 控制点
- 企业AI控制平面
Microsoft把模型、Agent开发和关联Azure资源纳入企业管理体系。买方通常已有Entra身份体系与Azure采购承诺,因此采购和治理集成很重要。Foundry支持资源与项目角色、托管身份和私有网络。这些控制减少平台暴露,却不会自动隔离全部依赖:AI网关及关联数据服务仍需各自配置。应用团队仍须授权每次敏感操作,并核对其实际执行结果。
Microsoft Foundry architecture ↗https://learn.microsoft.com/en-us/azure/foundry/concepts/architecture · Microsoft Foundry network isolation ↗https://learn.microsoft.com/en-us/azure/foundry/how-to/configure-private-link
Google Cloud
- 所有权/组织
- Alphabet
- 控制点
- AI、数据与云治理
Google Cloud在企业数据与托管AI工作流交汇处竞争。其安全架构结合身份、服务边界和加密密钥控制,但支持程度因服务而异。数据驻留条款明确区分某些Grounding与RAG功能。分析判断:围绕现有数据的集成可能比模型目录更有壁垒。买方风险则是误以为批准一个AI服务就等于全部关联功能都合规。应逐一检查处理路径和功能,而不是继承平台级结论。
Google Cloud service-specific data residency ↗https://cloud.google.com/terms/data-residency · Google Cloud: securely deploying AI ↗https://cloud.google.com/blog/products/identity-security/mastering-secure-ai-on-google-cloud-a-practical-guide-for-enterprises/
CoreWeave
- 所有权/组织
- CoreWeave
- 控制点
- GPU云与开发工作流
CoreWeave提供专业GPU容量,并通过Weights & Biases和OpenPipe向软件扩展。客户购买的是可靠的容量交付,以及从开发到训练和推理的更短路径。分析判断:掌握实验和后训练工作流可能增强留存与算力需求,但不能消除资本密集性和集中度风险。平台控制保护基础设施边界,企业应用权限仍是独立问题。完成收购证明所有权,不证明集成成功、模型质量更好或应用安全。
CoreWeave completes Weights & Biases acquisition ↗https://www.coreweave.com/blog/coreweave-completes-acquisition-of-weights-biases · CoreWeave 2025 Form 10-K ↗https://www.sec.gov/Archives/edgar/data/1769628/000176962826000104/crwv-20251231.htm
Nebius
- 所有权/组织
- Nebius Group
- 控制点
- GPU容量与生产云
Nebius把大型专用容量供应、托管云服务和生产治理结合起来。其Microsoft合同体现了关键市场结构:超大规模云厂商可以同时是竞争对手和客户。2025年公告描述截至2031年约174亿美元的合同,受部署和可用性条件约束,并有扩展空间。这不是年度收入,也不是已经收取的现金。IAM与基础设施治理有助企业采用,但客户的数据政策与Agent操作权限并不包含在容量合同之内。
Nebius Cloud 3.6 governance release ↗https://nebius.com/newsroom/nebius-ai-cloud-3-6-strengthens-developer-experience-and-governance-for-production-operations · Nebius–Microsoft agreement ↗https://www.sec.gov/Archives/edgar/data/1513845/000110465925088312/tm2525580d1_6k.htm
Together AI
- 所有权/组织
- Together AI
- 控制点
- 开放模型推理与部署
Together面向希望使用开放模型、又不愿自行组装全部推理运维的团队。专用端点预留隔离GPU,托管发布和性能功能服务于生产运行。商业差别在利用率:持续需求可能适合预留GPU小时,不规则流量则可用按Token计费避免闲置预留。区域部署选项取决于套餐与配置。隔离不能消除服务商访问问题,API兼容也不保证运维语义可移植。性能必须按实际模型和请求分布验证。
Together dedicated model inference ↗https://www.together.ai/dedicated-model-inference · Together EU deployment requirements ↗https://support.together.ai/articles/8079447813-eu-data-centers-and-dedicated-model-deployment
Fireworks AI
- 所有权/组织
- Fireworks AI
- 控制点
- 推理引擎与托管集群运维
Fireworks可通过Sovereign在客户自有GPU上运行推理栈,把基础设施所有权与推理软件及运维分开。客户管理网络边界;Fireworks说明了可审计的安装凭据和范围受限的组件身份。因此BYOC并不意味着服务商完全没有访问权限。可选云溢出还会改变数据边界。宣传中的吞吐与利用率提升是厂商主张,不是公平的跨供应商排名。壁垒取决于可重复的工作负载效率和运维交付。
Fireworks Sovereign shared responsibility ↗https://fireworks.ai/product/sovereign · Fireworks enterprise controls and retention exceptions ↗https://fireworks.ai/solutions/for-enterprise
Baseten
- 所有权/组织
- Baseten
- 控制点
- 定制模型推理与部署
Baseten面向需要跨云容量托管模型部署的AI产品团队。其2026年11月5日安全实践区分了具有逻辑隔离的共享基础设施、专用集群、自托管部署与混合运行;这些是实质不同的信任边界。分析判断:定制集成和可靠运维可能带来超越模型目录的差异化。买方须核对负载位置、服务商访问与产品级认证范围,不能把覆盖某项产品的认证自动延伸到另一项产品。
Baseten security practices ↗https://www.baseten.co/security-practices
Databricks
- 所有权/组织
- Databricks
- 控制点
- 数据、模型与业务应用平台
Databricks围绕企业数据及其转化为模型和应用的工作流竞争。MosaicML增加训练能力,Tabular加强表格式互操作,Neon技术支撑Lakebase业务型Postgres。分析判断:数据集成与累积工作流状态构成超越模型权重的切换成本。治理能减少碎片化,却不能替代运行时授权。目录权限不能证明Agent有权执行一笔具体外部付款,保存的追踪记录也不能证明付款已经结算。
Databricks–MosaicML acquisition agreement ↗https://www.databricks.com/company/newsroom/press-releases/databricks-signs-definitive-agreement-acquire-mosaicml-leading-generative-ai-platform · Databricks–Tabular completed acquisition update ↗https://www.databricks.com/blog/databricks-tabular · Databricks Lakebase launch powered by Neon ↗https://www.databricks.com/company/newsroom/press-releases/databricks-launches-lakebase-new-class-operational-database-ai-apps
一次部署可以同时采用多个竞争者
这些层往往互补而非相互替代。企业可以向专业云租GPU容量,在上面运行推理厂商的软件,同时把数据治理留在企业平台。账单有多个供应商,用户却只感知一个服务。应给每种失败明确负责人:容量不可用、模型发布卡住、证据不可访问、生成过慢和越权操作,是不同事件。集成能够降低协调成本,但单一商业合同不意味着全部组件同样可靠。对创业公司而言,更有价值的切口是反复跨供应商失效、买方又难以低成本自行承担的边界,而不仅是再做一个GPU利用率仪表盘。
技术关键:Prefill、Decode 与 KV 预算
Prefill 处理输入并建立 key/value 状态;Decode 在生成新 token 时反复读取已有状态。瓶颈取决于负载与硬件,因此单一吞吐数字会掩盖服务取舍。连续批处理可提高利用率,过高并发却会增加排队和显存压力。分离 Prefill 与 Decode 能采用不同调度策略,同时引入 KV 传输与网络成本。供应商即使 token 很快,也必须在目标并发下满足首 token 和后续 token 的延迟要求。
常规缓存的载荷字节数约为:2 × 层数 × 活跃序列数 × 每序列已缓存 token 数 × KV 头数 × 头维度 × 每元素字节数。取 32 层、8 条序列、4096 tokens、8 个 KV 头、128 维和每元素 2 字节,缓存载荷为 4 GiB。这个合成算例不含权重、激活、内存碎片和元数据。分组查询注意力可减少 KV 头;前缀共享只有在前缀可复用且隔离策略允许时才有效。容量规划要考虑峰值占用,而不只是平均请求。
容量、Token与合格任务经济性
容量业务承担设备、电力、交付与利用率风险。推理软件竞争的是延迟和质量约束下的有效吞吐。数据平台竞争集成与工作流留存。合成算例说明价格为何可能误导:供应商A处理1000次请求花20美元,获得800次合格完成,每次为0.025美元;B花16美元却只有500次合格完成,每次为0.032美元。B每次请求更便宜,每个合格任务却更贵。声称节省之前,应计入闲置预留、重试、数据移动和人工复核,并保持任务分布、质量门槛与截止时间一致。
另一个合成算例区分产能风险与软件利润:某个预留工作节点每24小时固定花费240美元。有效利用率为80%时,有效工作19.2小时,每小时成本12.50美元;只有25%时,有效工作6小时,每小时成本40美元。预留价格没有改变。这不是GPU报价,忙碌时间也不自动代表有效产出。它说明为什么比较自有产能公司与使用客户GPU的软件运营商时,必须看利用率承诺、工作负载多样性,以及谁承担空闲产能成本,不能据此猜测私营公司的毛利率。
壁垒与未解决问题在哪里
开源改变利润分布,而不是消灭利润。NVIDIA的四个Run:ai组件让部分调度清单功能趋于标准化,但社区分发不包含企业支持。持久机会可能存在于异构集群恢复、遵守策略的跨供应商路由、成本归因和可迁移证据。路由必须遵守地域、留存、模型质量与故障切换权限,而不只是选择最便宜端点。开放权重不意味着IAM语义、评测历史或部署状态可以迁移。买方应要求责任边界与导出路径。投资分析应追问:哪种重复运维故障有明确预算负责人,平台厂商能否顺手捆绑同类修复。
4. Electricity data: scale and uncertainty
The IEA's 2026 update puts global data-centre electricity consumption at 485 TWh in 2025 and projects approximately 950 TWh in 2030 in its central outlook. These are not AI-only figures. The projected increase is roughly 1.96×, not an observation that consumption has already doubled. The report also identifies physical supply constraints, including HBM and power-system bottlenecks. IEA source, scope and assumptionshttps://www.iea.org/reports/key-questions-on-energy-and-ai/executive-summary.
Electricity demand is a useful scale indicator, not an AI capability metric. Lower energy per task can coexist with higher total consumption if adoption grows or tasks become more computationally intensive. A query that initiates repeated planning, retrieval and verification cannot be compared with a single short completion simply by counting user messages.
For a facility, distinguish IT load from total facility load. Power usage effectiveness is total facility energy divided by IT energy over the same measurement interval; it does not describe model quality. A lower value cannot compensate for wasted computation. For deployment decisions, measure energy per accepted task, peak power, cooling constraints and grid availability separately. A global TWh forecast cannot tell you whether a particular site can connect next quarter.
5. The technical frontier: what to optimize, and what can fail
Prefill and decode disaggregation
Prefill processes the input sequence and initializes the cache; decode repeatedly produces another token. Their operating points differ. DistServe studies separating the phases and selecting resource allocations around time-to-first-token and time-per-output-token constraints. Its original preprint appeared January 18, 2024; this reading pins arXiv v3, June 6, 2024. Abstracthttps://arxiv.org/abs/2401.09670v3 · PDFhttps://arxiv.org/pdf/2401.09670v3.
The useful inference is not “separation is always faster.” Moving cache state costs bandwidth and coordination. With short prompts or a constrained network, colocated execution can be simpler and competitive. Measure cache-transfer time, queue delay and both latency objectives. Hold offered load constant; a configuration that rejects difficult requests must not appear faster merely because it serves an easier subset.
KV cache as a first-class resource
Cache memory grows with active sequence tokens. Grouped-query attention reduces the number of key/value heads relative to query heads, changing memory requirements. Prefix caching can reuse shared prompt state, but tenant boundaries and cache keys must prevent unauthorized reuse. Offloading may increase capacity while adding transfer latency. These are separate interventions: benchmark each rather than calling all of them “cache optimization.”
Mixture of experts and interconnects
An MoE layer routes tokens to selected expert networks. Sparse activation reduces the arithmetic used per token relative to activating every expert, but weights still need storage and routing creates communication. Skewed expert demand can create hot spots. Compare expert placement, token imbalance, communication overlap and tail latency. Counting active parameters alone does not determine serving cost.
Quantization and speculative decoding
Lower precision can reduce memory traffic, but scales, outliers and unsupported kernels complicate the result. Verify downstream task quality and numerical behavior, not only model loading. Speculative decoding proposes multiple tokens and verifies them with a target model; useful acceleration depends on acceptance, proposal cost and verification efficiency. Keep the target distribution and implementation assumptions explicit. A fast draft with low acceptance can slow the full system.
Training reliability and elastic scheduling
Large training jobs coordinate many devices. More devices create more opportunities for hardware or software interruption. Checkpoint frequency trades useful computation against recovery exposure; storage contention can make frequent checkpoints expensive. For a shared cluster, optimize fairness and queue wait as well as accelerator occupancy. High utilization from low-priority work is not success if urgent jobs repeatedly miss deadlines.
收购、资本与战略影响
收购改变控制点
CoreWeave于2025年3月4日宣布W&B协议,5月5日完成;OpenPipe于9月3日宣布,其申报文件将交割与9月5日联系起来。这些动作连接容量、开发和后训练。Databricks于2023年6月26日宣布MosaicML,7月19日完成;公告约13亿美元包含留任方案。其Tabular博客确认2024年6月7日完成收购。Lakebase于2025年6月11日使用Neon技术推出;本次核查未确定Neon确切法律交割日。该模式体现围绕重复工作流的集成,不证明交叉销售成功。宣布、交割与实际产品集成是不同里程碑。
一手产品与交易来源
- NVIDIA: full Run:ai platform open-source release ↗https://developer.nvidia.com/blog/nvidia-runai-is-now-open-source-as-part-of-nvidia-dsx-ai-factory-platform ·
- NVIDIA Dynamo ↗https://www.nvidia.com/en-us/ai/dynamo/ ·
- Amazon Bedrock: data protection and shared responsibility ↗https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html ·
- Amazon Bedrock: invocation permissions ↗https://docs.aws.amazon.com/bedrock/latest/userguide/inference.html ·
- Microsoft Foundry architecture ↗https://learn.microsoft.com/en-us/azure/foundry/concepts/architecture ·
- Microsoft Foundry network isolation ↗https://learn.microsoft.com/en-us/azure/foundry/how-to/configure-private-link ·
- Google Cloud service-specific data residency ↗https://cloud.google.com/terms/data-residency ·
- Google Cloud: securely deploying AI ↗https://cloud.google.com/blog/products/identity-security/mastering-secure-ai-on-google-cloud-a-practical-guide-for-enterprises/ ·
- CoreWeave completes Weights & Biases acquisition ↗https://www.coreweave.com/blog/coreweave-completes-acquisition-of-weights-biases ·
- CoreWeave 2025 Form 10-K ↗https://www.sec.gov/Archives/edgar/data/1769628/000176962826000104/crwv-20251231.htm ·
- Nebius Cloud 3.6 governance release ↗https://nebius.com/newsroom/nebius-ai-cloud-3-6-strengthens-developer-experience-and-governance-for-production-operations ·
- Nebius–Microsoft agreement ↗https://www.sec.gov/Archives/edgar/data/1513845/000110465925088312/tm2525580d1_6k.htm ·
- Together dedicated model inference ↗https://www.together.ai/dedicated-model-inference ·
- Together EU deployment requirements ↗https://support.together.ai/articles/8079447813-eu-data-centers-and-dedicated-model-deployment ·
- Fireworks Sovereign shared responsibility ↗https://fireworks.ai/product/sovereign ·
- Fireworks enterprise controls and retention exceptions ↗https://fireworks.ai/solutions/for-enterprise ·
- Baseten security practices ↗https://www.baseten.co/security-practices ·
- Databricks–MosaicML acquisition agreement ↗https://www.databricks.com/company/newsroom/press-releases/databricks-signs-definitive-agreement-acquire-mosaicml-leading-generative-ai-platform ·
- Databricks issuer release: MosaicML acquisition completed ↗https://www.prnewswire.com/news-releases/databricks-completes-acquisition-of-mosaicml-301881059.html ·
- Databricks–Tabular completed acquisition update ↗https://www.databricks.com/blog/databricks-tabular ·
- Databricks Lakebase launch powered by Neon ↗https://www.databricks.com/company/newsroom/press-releases/databricks-launches-lakebase-new-class-operational-database-ai-apps ·
- CoreWeave announces OpenPipe agreement ↗https://www.coreweave.com/news/coreweave-to-acquire-openpipe-leader-in-reinforcement-learning ·
- CoreWeave 2025 Q3 filing: OpenPipe closing ↗https://www.sec.gov/Archives/edgar/data/1769628/000176962825000062/crwv-20250930.htm ·
Source and maintenance notes
Primary links are placed beside the claims they support. Acquisition amounts retain their original basis; undisclosed values remain undisclosed. Product specifications and release statements are vendor disclosures, not comparative evidence from Token Lab. The diagrams and reference calculations are original educational material. Corrections should identify the section, source and proposed replacement; a source-access date is not a paper publication date.
All frontier research · Top Company Frontier · Editorial standards
实践与验证
实践:故障与负载下的基础设施
生成提案后停止模型,重启后批准仍应待处理;工具确认后停止工作流,重试前必须核对。聊天记录本身不是持久检查点。
注入超时、错误工具参数和模型版本不匹配,要求尝试有界、状态明确、主备路径授权边界一致;在相同预算下比较安全不回答基线。
重放长上下文、慢工具、取消、突发、冷缓存与故障,运行足够久暴露泄漏或队列增长。记录硬件、精度、模型、调度器与软件版本。在相同p95延迟和验收率下比较吞吐,先定义回滚阈值。
超出活跃与队列限制,检查明确过载、内存有界和第二租户进展。独立改变输入长度、输出长度与前缀复用,区分冷热并检查隔离;报告吞吐、尾延迟、峰值内存和合格任务率。
从区域故障、配额耗尽、慢检索和等待批准时重启恢复,核验取消、撤销和幂等。明确哪些是本地实测、哪些分布式保证仍未证明。
7. Reproduce a memory and cost budget
This is an original teaching calculation, not a vendor benchmark. Assume B active sequences, T cached tokens per sequence, L transformer layers, Hkv key/value heads, head dimension d and b bytes per stored value. K and V each have shape [B, Hkv, T, d] per layer. Ignoring allocation overhead and sharding:
For B=8, T=8192, L=32, Hkv=8, d=128 and b=2, the cache occupies 8 GiB. At 32768 tokens it occupies 32 GiB. Changing Hkv from 8 to 32 also gives 32 GiB at the shorter length. GiB means 2³⁰ bytes; device specifications expressed in GB need conversion before subtraction.
This is logical total cache size, not a claim about per-device allocation. Add weights, temporary activations, runtime reservations, fragmentation and any duplicated/sharded state. Weight quantization does not automatically quantize the cache. In a mixed-length batch, replace B×T with the sum of each active sequence's cached length.
At an illustrative $8 per hour and two accepted requests per second, the capacity cost is $0.001111 per accepted request. One accepted request per second doubles it. These hypothetical prices exclude network, storage, labor and idle reservations. “Accepted” should include a task-quality check and the agreed latency constraint; otherwise a faster but less useful system gets an unfair advantage.
Local capacity experiment
KV cache: 8.000 GiB for the stated eight-sequence configuration.
Predict the result before doubling the context length. No inputs leave your browser.
Download infra-capacity.py. Python 3.10+, standard library only; deterministic arithmetic, no random seed, network requests or model API. Run python3 infra-capacity.py.
"""Teaching arithmetic, not a GPU benchmark. Python 3.10+, standard library."""
from math import isclose
def kv_gib(batch, tokens, layers, kv_heads, head_dim, bytes_per_value):
values = (batch, tokens, layers, kv_heads, head_dim, bytes_per_value)
if any(not isinstance(x, int) or isinstance(x, bool) or x <= 0 for x in values):
raise ValueError("All dimensions and bytes per value must be positive integers")
return 2 * batch * tokens * layers * kv_heads * head_dim * bytes_per_value / 2**30
def cost_per_success(hourly_cost, accepted_per_second):
if hourly_cost < 0 or accepted_per_second <= 0:
raise ValueError("Nonnegative cost and positive accepted throughput required")
return hourly_cost / (3600 * accepted_per_second)
if __name__ == '__main__':
small = kv_gib(8, 8192, 32, 8, 128, 2)
long = kv_gib(8, 32768, 32, 8, 128, 2)
assert small == 8 and long == 32
assert kv_gib(8, 8192, 32, 32, 128, 2) == 32
assert kv_gib(8, 8192, 32, 8, 128, 1) == 4
assert isclose(cost_per_success(8, 2), 8/7200)
assert cost_per_success(8, 1) == 2 * cost_per_success(8, 2)
for args in [(0, 1, 1, 1, 1, 2), (1, -1, 1, 1, 1, 2)]:
try:
kv_gib(*args)
except ValueError:
pass
else:
raise AssertionError('Invalid dimensions accepted')
print(f'KV: 8k = {small:.1f} GiB; 32k = {long:.1f} GiB')
print(f'Cost per accepted request: ${cost_per_success(8, 2):.6f}')
print('8 checks passed')
Expected output:
KV: 8k = 8.0 GiB; 32k = 32.0 GiB
Cost per accepted request: $0.001111
8 checks passedA real benchmark protocol
Pin the model checkpoint, tokenizer, runtime commit, kernels, precision, hardware topology and power configuration. Replay the same timestamped request trace under cold and warm caches. Sweep offered load until an SLO fails. Report arrivals, successful completions, errors, quality, p50/p95/p99 latency and throughput together. Repeat independent runs and publish spread, not just the best run.
Ablate one factor at a time: colocated versus separated phases; cache reuse off versus on; full versus reduced precision. Compare equal hardware budgets first, then equal cost and power budgets if available. Do not drop out-of-memory failures from the denominator. This lesson verifies arithmetic only; it does not run a GPU service or reproduce a paper's headline speedup.
8. Open problems and original interview practice
Unresolved problems: managing long-lived agent state without runaway memory; sharing processors without noisy-neighbor latency; predicting MoE traffic; reliable heterogeneous execution; reducing recovery loss; and reconciling interactive demand with power constraints. Each problem joins at least two layers. A promising research question specifies both an improvement and the condition under which it stops helping.
For example: can a scheduler reduce p99 latency without reducing accepted throughput for another tenant? Can a cache-placement policy save memory without increasing disclosure risk? Can a hardware-independent intermediate representation preserve performance across kernels rather than merely compile? These are measurable questions. “More efficient infrastructure” is not yet an experiment.
The following are original simulations, not reported company interview questions.
1. Why can a 2× faster kernel leave end-to-end latency almost unchanged?
First hint: Separate queue, computation, communication and tool time.
Answer and analysis
If the improved kernel was only 10% of total latency, halving it changes total time from 1 to 0.95: approximately 1.053× speedup. Queueing can further change the result under load. Measure the critical path before optimizing.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
2. A model fits in memory. Why can inference still fail?
First hint: Weights are not the only resident state.
Answer and analysis
Concurrent KV caches, activation workspace and allocator reservations can exhaust memory. Calculate each term with units, inspect actual peak allocations, then test mixed sequence lengths and admission control. Model parameter count alone is insufficient.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
3. When does prefill/decode separation lose?
First hint: Account for the state crossing the boundary.
Answer and analysis
Transfer and coordination can dominate for short requests or slow links. Sweep prompt length and offered load with fixed SLOs, include network contention, and compare equal resources. A paper’s maximum speedup is not a universal prediction.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
4. What would make a cheaper accelerator more expensive per successful request?
First hint: Include software, utilization and failures.
Answer and analysis
Lower rental cost can be offset by unavailable kernels, compilation, retries, low quality or idle reservation. Compare accepted goodput and total ownership costs using the same workload, not hardware price alone.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
Continue with inference systems, the attention project and the companion AI security survey. These topics meet at cache isolation, scheduling privileges and operational reliability.
9. Frontier research agenda: five falsifiable projects
The following are Token Lab's proposed research directions, not claims that these methods have already succeeded or are novel. Search the literature before claiming novelty. Each proposal starts with an engineering failure and ends with evidence that would count against the idea. Select one narrow workload for a first PoC; broad claims require replication across tasks, implementations and operating conditions.
1. Long-context capacity under a latency budget
Why existing approaches are insufficient: A scheduler that treats each request as an identical job ignores the memory-time footprint of a long context. Reserving only current cache size can over-admit work; reserving maximum context can waste capacity.
Testable hypothesis: A reservation policy using predicted remaining tokens can improve accepted throughput at a fixed out-of-memory risk and latency SLO.
Baselines and fair comparison: Compare current-size reservation, worst-case reservation and a simple length-bucket policy before adding a learned predictor. Use the same arrival trace and cap total memory.
Measurements: Report accepted goodput, rejection rate, prediction calibration, out-of-memory events and p99 latency by length bucket. Include deliberately underestimated output lengths.
What would falsify the claim: Reject the hypothesis if the apparent gain comes from rejecting long requests, increasing failures, or obtaining future token lengths unavailable at admission. A useful system should degrade gracefully when predictions are wrong.
2. Disaggregation when the network is shared
Why existing approaches are insufficient: Dedicated-link benchmarks can hide contention. Prefill/decode separation moves a large cache between stages while training, storage and other tenants may use the same fabric.
Testable hypothesis: Topology-aware placement and bounded transfers can preserve the gain under realistic cross-traffic.
Baselines and fair comparison: Compare colocated serving, naive separation and bandwidth-aware placement. Sweep prompt length, batch size and network contention without changing total accelerators.
Measurements: Measure transfer bytes and duration, queue wait, TTFT, TPOT and goodput. Report the crossover region where separation stops helping, not only the best configuration.
What would falsify the claim: Reject if the method needs dedicated bandwidth that the baseline was denied, or reduces one latency metric by violating the other. A robust result includes topology changes and link degradation.
3. Portable performance across accelerator families
Why existing approaches are insufficient: A model can compile on two systems while one falls back to slow kernels or numerical approximations. Portability of source code and portability of performance are different goals.
Testable hypothesis: A small set of workload-aware kernel choices can retain quality and predictable performance across two supported platforms.
Baselines and fair comparison: Use the default runtime, hand-tuned platform paths and the proposed portable selection policy. Pin model weights and compare identical task outputs with appropriate numerical tolerances.
Measurements: Report compile time, fallback operations, end-to-end cost, energy when measurable, quality and maintenance effort. Include dynamic sequence shapes and an unseen model variant.
What would falsify the claim: Reject if gains require per-model manual retuning comparable to the specialized baseline, or if quality drops explain throughput. Document unsupported operators instead of silently excluding them.
4. Power-aware service without exporting the cost to users
Why existing approaches are insufficient: Optimizing energy in isolation can defer work until queues become unacceptable; optimizing latency alone can create demand peaks. Long reasoning workloads further vary per-request compute.
Testable hypothesis: A controller that jointly chooses admission, batching and power limits can reduce energy per accepted task while maintaining a declared latency envelope.
Baselines and fair comparison: Compare fixed power, static batching and an SLO-only scheduler. Replay identical bursts; keep task quality fixed and include controller overhead.
Measurements: Measure joules per accepted task, wall power, p95/p99 latency, throughput, rejection and recovery after a burst. State whether power is device-only or facility-wide.
What would falsify the claim: Reject if saved energy is just unserved work, if the measurement omits another device, or if gains vanish outside a smooth synthetic trace. Grid-level claims require evidence beyond GPU telemetry.
5. Persistent agent state across failures and tenants
Why existing approaches are insufficient: An agent session may outlive a process, move across hardware and hold private context. Cache migration and prefix reuse create both a systems optimization and an authorization problem.
Testable hypothesis: A versioned state protocol with tenant-scoped keys can support recovery without exposing stale or unauthorized context.
Baselines and fair comparison: Compare no reuse, process-local reuse and shared reuse with explicit ownership. Inject restarts, revocations and concurrent updates.
Measurements: Measure recovery time, recomputation, cache hit rate, accepted throughput and isolation violations. Test correctness of outputs and policy decisions after each migration.
What would falsify the claim: Reject if recovery works only without revocation, if a cross-tenant key collision leaks state, or if the method assumes all workers are equally trusted. This is a joint infrastructure/security project.
From a two-week PoC to a defensible report
In the first stage, write the system boundary, assumptions and failure criterion before implementing the proposed method. Build the simplest baseline and a reproducible fixture. In the second stage, add one intervention and run controlled sweeps; retain failed runs and explain exclusions. In the final stage, test an unseen workload or attack family and write a limitations section. Separate what your experiment measured from what you hope will generalize.
A useful report contains a configuration manifest, data-generation procedure, code commit, exact commands, raw observations, aggregation script and a table of threats to validity. Use multiple seeds when randomness matters; repeated deterministic arithmetic is not independent evidence. Publish negative results: the boundary where a technique stops working is often the most transferable lesson.