AI security in 2026: securing models, data and autonomous actions
Follow an enterprise agent from LLM calls and IAM through prompt injection, least privilege, exact approval, tool execution, audit and incident recovery.
来源与版本详情
Contents · lifecycle and reference chapters
本文目录
安全:从用户请求到已提交操作
以下原创案例贯穿一个发票助手:它读取租户范围内的资料并提出付款建议,实际付款需要单独且精确的授权。场景与实验均为合成教学材料,不代表真实部署。
01 · 跟随发票助手,从请求走到实际后果
09:00,应付账款分析员Maya向助手提出:“找出Northstar的逾期发票,解释差异,并准备付款建议。”这是合成教学场景,不是已报告事件。Maya可以读取自己租户的发票并创建建议,但不能悄悄修改供应商银行账户或执行付款。执行前必须由具有独立授权的审批人批准具体交易。业务目标是在这一边界内提供有用帮助,而不是让聊天机器人拒绝所有财务请求。
应用认证Maya、检索允许访问的记录、把选定上下文发送给模型、解释返回的计划、调用工具、必要时请求审批、执行已授权操作,并记录结果。每一条箭头都跨越某种边界:人与服务、指令与证据、租户、服务商、工具、时间或管理域。安全工程要回答每个接收方在边界处究竟有权信任什么。
供应商可以控制上传的PDF,却不能控制Maya的角色。被攻陷的检索连接器可以返回虚假内容,却不应能签发付款凭据。模型可以建议调用工具,却不能给自己授权。被盗的工作进程令牌可能允许部分操作,因此即使所有模型行为都具有敌意,系统也必须限制该凭据可做什么。恶意内容与可用权限的区别贯穿整个架构。
继续之前,要理解认证用于确定主体,授权用于决定该主体能否执行某个具体操作,审计记录则是有关决策或事件的证据。三者互不等价。读完后应能画出信任边界图、编写授权不变量、设计负向测试、重建有争议的操作,并解释剩余不确定性。文章建立系统基础;专家判断仍需要在真实环境中实现和验证这些控制。
02 · 运行前:定义资产、参与者与禁止结果
威胁建模从第一条提示词之前就开始。先列出资产:发票内容、供应商身份、银行信息、审批状态、付款凭据、模型服务商密钥、日志与持久记忆。再列出参与者及其可影响的范围:外部供应商控制文档,普通员工控制请求,租户管理员控制部分集成,攻击者可能盗取会话或攻陷插件。不要假定攻击者早已控制所有服务器,这种全能设定会掩盖正在检查哪项控制。
写出具体不变量:检索文档永远不能扩大权限;租户A的请求不能检索租户B的发票;付款必须匹配有效审批中的租户、收款人、金额、币种与提案版本;重试不能产生另一笔付款;在授权新的敏感操作之前,撤销必须已经生效。模型生成的解释不能替代这些条件。它们有用,是因为可以通过测试展示具体违规。
模型还要包含依赖清单:模型版本、Tokenizer、编排库、工具清单、浏览器自动化包、容器镜像和检索流水线。恶意软件包或被修改的工具描述,与恶意发票从不同通道进入。固定并审查可部署产物,分离开发与生产凭据,对工具注册表实施受控变更。签名在其信任假设下证明产物声称的来源和完整性,不证明行为无害。
架构原则是针对具体资源建立信任,而不是“网络内部一切安全”。2020年8月发布的NIST SP 800-207https://csrc.nist.gov/pubs/sp/800/207/final提供零信任基础。本例把该原则应用于由模型介入的操作。
区分提示注入与模型或数据投毒。提示注入通过运行时内容引导已有模型;有毒训练样本、恶意微调或被改动的模型产物,可能改变学到的行为或实际部署的计算。应分别保护数据集血缘、训练权限、产物来源与发布关卡。即使模型产物可信,也不能绕过业务授权;来源证明缩小供应链疑问,但不授予运行时权限。
03 · 请求入口:保留用户、租户和工作负载身份
Maya通过应用原有的身份系统登录。网关验证会话,并根据服务端成员关系确定租户,而不是采用提示词中的租户名称。请求上下文包含经过认证的主体、租户、会话或委托标识符,以及应用的工作负载身份。这些身份并不相同:人提出任务,服务执行任务,模型生成候选文本。把模型称为“Agent”并不会使它成为独立可信的身份。
第一层攻击面仍是常规应用安全。会话窃取、跨站请求伪造、不安全对象引用,以及由客户端提交的租户标识,都可能在提示注入发生之前破坏系统。应根据会话和传输设计采用相应保护。在已认证租户和操作范围内解析发票标识符,不要先按全局ID获取发票,再假定模型会发现它属于别人。
后台任务应持久化委托上下文以及到期时间或撤销引用,而不是把整个浏览器会话序列化到队列中。长任务恢复时,应重新确认相关权限仍然存在。昨天获得授权的用户今天可能已经离职。持久任务记录不等于永久权限授予。
04 · 调用LLM:推理服务不是授权服务
应用根据用户任务、相关政策指引和数量受限的证据构建模型请求。模型服务商API凭据用于向推理服务认证应用,它既不是Maya的发票读取凭据,也不应变成支付凭据。把它保存在提示词之外,通过受控服务端机制获取。不要让模型在工具之间携带秘密。
必须明确服务商边界:哪些数据允许离开应用环境,批准了哪个端点和地区,请求日志与留存如何工作,备用服务商是否满足相同的数据处理许可。这些检查取决于部署和合同,不能从模型品牌推断。批准端点不可用时,自动切换到未经审查的端点,可能使可用性机制变成数据泄露路径。
用Token、时间和金额预算约束推理请求。长文档或对抗性循环即使从未执行禁止的工具操作,也能耗尽共享容量。脱敏或省略任务不需要的字段,同时认识到简单脱敏也可能删掉正确回答所需的证据。解释发票所需的最少信息,应与实际支付所需的信息分开。
收到响应后,按带版本的Schema解析模型输出并限制大小。有效JSON工具调用只是格式正确的建议,其中的字符串对SQL、HTML、Shell、URL及下游API仍是不可信输入。数据库访问应参数化,工具应使用范围狭窄的类型化适配器。模型仍可能提出有说服力但错误的计划,后续边界必须限制其后果。
05 · 检索:有权读取不代表内容可信
检索器只在Maya已授权的数据范围内搜索。在敏感内容进入模型上下文之前落实文档级访问规则,并在向量搜索之后获取完整文档时保留策略检查。仅过滤最终答案已经太迟,因为模型或外部服务商可能早已收到秘密。缓存键与检索索引必须遵守租户和授权边界;共享语义缓存可能把此前获准的答案泄露给另一个用户。
现在,供应商PDF中出现要求助手忽略原始任务并修改付款信息的文字。Maya可以阅读PDF,但作者无权向她的应用下命令。攻击试图把证据提升为指令。这就是间接提示注入:指令通过为其他目的检索的材料进入模型。直接注入则来自面向用户的输入通道。两者都不同于取得任意执行能力的软件漏洞,但都可能通过脆弱工具造成损害。
保留来源信息:源ID、租户、检索时间、文档版本、访问决策,以及可信应用策略和不可信证据的区别。分隔符、指令层级、内容扫描及抗注入训练可以减少部分失败,却不能证明对抗文档无法影响模型。OWASP LLM01:2025提示注入指南https://genai.owasp.org/llmrisk/llm01-prompt-injection/提供有用的威胁分类;本文固定引用这一版本,不暗示它代表当前风险排名。
06 · 规划与工具调用:把建议和权限分开
模型提出“读取发票”“查询供应商”或“准备支付”,编排器把提议映射到已注册适配器。不能因为三个窄接口难写,就开放管理员shell。工具描述变化会影响模型行为,实现变化会改变实际能力;两者都需要审核和版本记录。
每次特权操作前应设置确定性的策略执行点,向权威策略决策点核验主体、工作负载、租户、动作、资源及当前上下文。模型可以解释理由,但不能篡改身份来源或跳过检查。模型拒绝有助于纵深防御,仅由模型给出“允许”却不是强制访问控制。
本例的提案工具只能创建包含发票和收款人的草稿,不能执行支付或修改供应商主数据。三种动作对应不同权限。允许集合应是用户权限、服务权限、任务委托范围与策略约束的交集,而不是任一参与者拥有的能力之并集。
把策略判定落实到具体数据:模型提议给发票INV-42支付120美元,可信上下文提供Maya、租户T1和支付工作负载身份;资源解析器提供发票所属租户、未付金额和供应商版本。允许执行必须同时满足租户匹配、支付权限、任务预算、发票状态,以及绑定同一金额、币种和收款目标版本的批准。模型只能提出候选动作,不能提供权威成员关系、剩余预算或“已批准”标记。
所有限制必须共同生效。独立授权的审批人一般可以批准500美元,但当前任务只委托100美元时,120美元提议仍应拒绝。拆成两次60美元也不能绕过累计预算。预算应随授权或提交原子预留,结果未知时先核对,只有确定失败才释放预留;否则两个并发工作者都可能看到足够余额而造成超支。这些金额是合成教学数据,不是金融策略建议。
07 · 最小权限:把凭据约束到实际操作
权限过大的服务账户会形成混淆代理:应用拥有请求者或检索文档没有的权限,却可能被诱导替它们行使权限。因此,“Agent有一个令牌”回答了错误的问题。应追问该令牌代表谁的权限,哪个资源服务器可以接受它,允许哪些操作和对象,何时到期,以及能否重放或继续委托。
优先通过可信凭据代理或受支持的授权流程取得短期、适合目标资源的凭据。支持委托时,应保留执行者与主体的区别。发给发票服务的令牌不应被支付服务接受。Audience检查限制接收方,scope限制操作类别,对象级与租户级策略仍负责决定具体哪张发票可读。名为read的scope不自动意味着“只能读这一张发票”。
RFC 9700,2025年1月https://www.rfc-editor.org/rfc/rfc9700.html建议限制访问令牌权限、受众,并采用适当的发送方约束机制。RFC 8693,2020年1月https://www.rfc-editor.org/rfc/rfc8693.html规定令牌交换,包括与委托有关的表示。它们不代表所有服务商都支持任意缩小权限,也不代表请求窄令牌就一定获得窄令牌;授权服务器必须落实最终策略。
集成MCP时,要针对预期服务器验证凭据,并为下游API建立独立且合法的授权关系。固定于2025-11-25文档版本的MCP安全指南https://modelcontextprotocol.io/docs/2025-11-25/tutorials/security/security_best_practices说明混淆代理和令牌透传风险。不要把上游Bearer令牌直接转发到另一个服务。较短有效期能降低暴露时间,却不能消除剩余有效期内的滥用。
08 · 批准:同意必须绑定精确且仍有效的动作
助手向有权批准的人展示支付提案。“批准这个Agent”太模糊;可信界面应展示发票、核验过的供应商、目标账户引用、金额、币种和执行时间。非可信描述按数据渲染,与应用标签区分,不能让攻击者把账户替换藏进说服性文字。
批准记录绑定动作的规范表示、租户、资源版本、批准人、到期时间及唯一nonce。摘要方便比较,却不授予权限;批准服务还要认证并授权批准者,保护记录。规范化必须统一金额单位、币种、可选字段和账户标识,避免批准与执行解释不同。
执行前重新核验权限、撤销、期限和资源版本。批准后账户变化,动作已不再相同;采用事务前置条件或条件写入,不能检查后留下无保护间隙。新信息可能需要重新提案。真实的人也会错误批准,因此界面应基于核验状态,敏感变更有独立检查,必要时分离职责。有人参与不是万能安全证明。
09 · 执行:限制真正接触外部系统的机器
获得授权的动作仍可能实现不安全。账本查询应使用类型明确的客户端,而非把模型文本插入shell。文档分析限制CPU、内存、文件访问和存活时间;隔离模型密钥、应用凭据与无关租户数据。容器只是隔离机制,不保证不会逃逸或配置错误。
网络需单独策略。抓取模型URL时限制协议、目标与跳转,并按网络模型阻止内部管理与metadata端点。DNS解析和重定向可绕过浅层字符串检查。OWASP SSRF指南https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html解释基础风险。业务适配器通常更适合接收供应商ID,再由服务端解析目标,而非接受任意URL。
出站数据同样需要控制。能读发票不代表能把它寄给发票里任意邮箱。按策略限制收件人和渠道,最小化导出字段,重要披露要求批准。分类器可辅助分流审核,但不能替代权威接收方与租户策略。参数验证、授权、沙箱和出站限制保护不同边界,统称guardrails会掩盖实际防护机制。
10 · 重试、长任务与多Agent委托
支付API超时不能区分执行前失败和执行成功但响应丢失。盲目重试可能重复付款。同一逻辑动作使用稳定幂等键、权威操作记录和服务商支持的核对机制;不能因为模型生成了不同说明就更换键。
幂等有业务范围和有效期,不等于恰好一次传递。服务端应把键绑定规范动作并拒绝参数变化。持久保存提议、批准、提交、确认和未知状态;未知结果进入核对或人工审核,不能自动标成功。补偿可能有成本,也未必能逆转效果。
委托子Agent提取发票时,只授予必要文档访问和限时任务上下文。子Agent不能声称紧急而扩大权限,返回的指令不能成为父Agent授权依据。恢复任务或切换操作时重新检查权限;撤销应按明确策略阻断新授权和敏感动作。关闭聊天不一定取消队列任务。
11 · 记忆与学习:攻击可能跨越会话持续存在
助手若把“Northstar付款改用新账户”保存为权威偏好,恶意PDF就跨过两道边界:证据变成指令,临时上下文变成持久策略。原文消失后,下次会话仍可能受影响。
按用途与信任等级分开用户偏好、检索事实、推断摘要、批准记录和策略配置。保留来源、租户、版本、期限,以及有权创建或修改记录的身份。模型摘要只是派生证据,不能自动覆盖核验过的供应商数据或安全规则。
生产反馈能改善提示词与检索,也能被攻击者操纵。审核进入训练或优化的数据,保留优化器无法查看的留出集,改变工具或权限前设置独立发布门槛。同一受污染裁判给出更高奖励,不证明能力或安全提高。
12 · Agent审计:重建权限、动作与真实结果
财务询问为什么尝试支付时,仅有聊天记录不够。应记录认证主体与工作负载、租户、任务和尝试ID、策略及资源版本、规范化动作、授权结果、批准引用、执行ID和观察结果。失败尝试与后续重试分别记录,减少敏感正文,并限制审计访问者。
动作回执连接三件事:请求了什么、为什么允许、下游报告了什么。数字签名在密钥信任成立时能够认证签发者并检测字节被改动,却不能证明签发者诚实、记录完整、策略正确或款项已结算。结算证据来自权威业务系统,可能稍后才出现。
哈希链只有在攻击者无法替换可信锚点和全部副本时,才能检测某些修改。它不会自动防止控制整个日志路径的管理员删除、遗漏或重写记录。应定义独立保留、完整性检查、访问控制和失效处理。OWASP日志指南https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html提供事件选择、敏感数据处理与日志系统保护的基础。
普通遥测丢失可采用有界缓冲和可见丢失计数;敏感操作的必需审计证据可能要求拒绝执行或持久提交。应按操作决定,不能误把偏重可用性的日志策略用于支付授权。模型可以总结证据,但必须保留原始记录链接、标注不确定性,不能编造因果故事。
按动作后果选择审计持久性。诊断性的token事件可以在明确说明的前提下采样,但支付所需的授权与批准证据不能指望从采样文本还原。一种设计是在同一个本地数据库事务中写入动作意图和审计outbox,再由发送器使用稳定的下游幂等键提交。这只能保证本地意图持久,不能让远端支付与本地事务自动成为一个原子事务。
响应丢失后,应把操作ID与支付服务的权威状态关联,再追加观察结果。“已提交、结果未知”必须是真实状态。如果服务既不支持幂等,也没有可靠状态查询,就不能承诺自动重试不会重复支付;人工核对是架构的一部分。这样才能把持久意图、获准执行和结算确认分开,而不是藏进一个成功布尔值。
13 · 检测、遏制与恢复补全生命周期
助手读某份文档后反复提议未授权的供应商修改,应按拒绝模式、集成和来源告警,而不只匹配可疑词。即使授权边界守住,反复被挡的尝试也能揭示攻击;合法发放的凭据被滥用时,获准但异常的动作序列同样值得调查。
遏制具体能力:停用受损连接器、撤销委托、暂停敏感队列、隔离工作者或记忆空间。破坏性清理前保存证据,向账本核对未知交易,界定实际受影响用户和租户,区分确认后果与可能暴露。戏剧性的模型对话不是数据确已外泄的证明。
恢复要重建可信状态:清除污染的派生产物和原文、轮换暴露凭据、审查工具与策略变更,恢复已知配置,再通过有准入标准的受限canary逐步恢复。边界不修,仅封禁旧攻击词只能挡昨天的载荷。
每次事故应形成证据、权限判定和副作用时间线,并转成回归案例。区分起始内容、失效控制和业务后果,避免“模型被越狱”掩盖过权服务账号、缺失对象检查或不安全重试。复盘必须解释修复为何阻止同类攻击且不破坏合法流程。
How the four disciplines fit together
Security determines which actions are permitted. Infrastructure executes work within capacity and failure constraints. Observability preserves evidence of what happened. Evaluation tests whether a change improves the outcome under a declared contract. None substitutes for the others.
机制、研究与市场背景——以下参考材料仍为英文
The following chapters provide the mathematical examples, experiments, research papers and dated company evidence behind the operational story. Return to the lifecycle when deciding where each mechanism belongs.
1. Define the security problem before choosing a product
An enterprise assistant reads a document, drafts a recommendation and calls a tool. Three different systems are involved: a model that predicts useful output, a data system that decides what can be read, and an execution system that decides what may happen. Security fails when one system silently grants authority belonging to another—for example, when text inside an external document is treated as permission to send a confidential file.
Security for AI protects model artifacts, training and retrieval data, applications and agent actions. AI for security uses models to help analysts investigate or respond to attacks. Alignment concerns whether model behavior follows intended goals and constraints; it overlaps with security but is not a replacement for access control. A model can produce a well-intentioned answer while an application exposes the wrong tenant's data.
A threat model names the assets, adversary capabilities, trust boundaries and unacceptable outcomes. Prompt injection occurs when attacker-influenced content redirects the model away from the authorized task. An indirect injection arrives through a document, website or tool response. A confused deputy has legitimate authority but is manipulated into using it for someone else's purpose. These definitions are more useful than treating every unwanted answer as the same kind of jailbreak.
Before this lesson: understand authentication versus authorization. Identifying an agent does not permit every action it proposes. Understand precision versus recall: detecting most attacks can still create too many false alarms. Finally, distinguish a content classifier's probabilistic judgment from a deterministic permission check. The learning goal is to combine them without allowing the model to rewrite its own boundary.
This survey compares disclosed product roles and transactions, not independently measured vendor efficacy. A procurement claim needs tests on the buyer's workflows. A public benchmark needs its own attacker budget, task set, software version and scoring procedure. Neither can be replaced by an acquisition price.
2. A trust-boundary architecture for an agent
Consider an invoice assistant authorized to prepare reminders for one business unit. The model reads invoice content and proposes an action containing an invoice ID, recipient ID and message. The execution service resolves those identifiers against authoritative records, checks that the current principal can act on the invoice, validates the recipient and enforces any required approval. Only the execution service holds the sending credential.
An instruction embedded in the invoice cannot broaden the principal's scope. The model also cannot declare that its own proposal has already been approved. Approval should bind the exact action, resource and relevant parameters, expire when appropriate, and be invalidated when the proposal changes. Retries need idempotency keys so a network timeout does not create duplicate sends. This turns vague “human oversight” into a specific state transition that can be tested.
Logs should record the model and tool versions, proposal, effective identity, policy decision and execution receipt. Protect those logs: copying all prompts and documents into an observability service can create another disclosure path. Minimize retained content and separate access to operational metadata from access to sensitive payloads.
The boundary is useful even when a detector misses malicious text. A model may still produce a bad proposal, but it cannot execute an unauthorized action through the controlled service. This is not a complete guarantee: an overly broad policy, compromised credentials, an unmediated tool or a bug in the executor can bypass the intended design. Test the enforcement path itself.
3. Follow the attack surface across the lifecycle
Model and dependency supply chain
- Failure mechanism
- A model artifact, serializer or dependency can execute code or alter behavior before inference starts.
- Control and residual question
- Inventory and pin artifacts; verify provenance; isolate loading; scan dependencies. A signature authenticates a publisher, not the semantic safety of the model.
Retrieval and memory
- Failure mechanism
- Poisoned content can be retrieved repeatedly or stored as a persistent instruction.
- Control and residual question
- Preserve provenance, enforce tenant filtering before retrieval, bound memory writes and expire untrusted state. How should legitimate corrections differ from poisoned memory?
Multimodal input
- Failure mechanism
- Text, images, audio or layout can carry instructions that survive extraction differently.
- Control and residual question
- Test the whole preprocessing pipeline and modality conversions. A text-only filter does not establish coverage of the original input.
Tool use and delegation
- Failure mechanism
- The agent can misuse a legitimate tool or transfer excessive scope to another agent.
- Control and residual question
- Use explicit identities, least privilege, scoped capabilities, approval gates and revocation. Natural-language intent remains harder to specify than an API permission.
Outputs and data movement
- Failure mechanism
- A response, URL, attachment or log can disclose data to an unauthorized destination.
- Control and residual question
- Check recipients and destinations, apply data policies and minimize secrets in context. A detector’s refusal is not proof that all side channels are closed.
Evaluation and updates
- Failure mechanism
- Repeatedly tuning on known attacks creates benchmark overfitting; an update can invalidate prior defenses.
- Control and residual question
- Separate development and held-out attack families, record versions, test utility regressions and rehearse rollback. An unchanged benchmark score may conceal changed deployment risk.
The difficult cases are often individually plausible steps. Reading a page, summarizing it and saving a note may each be allowed, yet their composition can move confidential information across a boundary. Conversely, blocking every unusual document destroys useful work. A good threat model describes prohibited outcomes, rather than relying only on a list of suspicious phrases.
For an RSI or self-improvement loop, treat new prompts, tools, policies and training data as untrusted candidate changes until independently evaluated. The evaluator must not share writable state with the candidate. A candidate that edits the benchmark, expands its tools or selectively suppresses failures has changed the experiment rather than demonstrated improvement. See the RSI tutorial for promotion gates and provenance.
行业格局:控制点、买方与竞争
行业证据核对于 2026 年 11 月 8 日。产品文档只能证明供应商提供了相关能力,不能证明独立实测效果。竞争判断与机会分析为 Token Lab 的分析;标为合成的算例不是公司实测数据。
谁掌握 AI 安全边界:公司格局、并购与真正的空白
这份分析覆盖十一家厂商与平台,核对日期为 2026 年 11 月 8 日。产品文档用于确认所述能力与部署边界,不代表独立验证了防护效果或排名。以下商业判断属于本站分析;成本算例为假设,不是厂商实测。
市场竞争的核心是控制点
AI 安全不是一组可互换的产品。模型扫描器控制发布检查点,网关控制请求路径,身份平台控制凭据,运行时传感器观察执行,SOC 平台承接调查;它们拥有的证据和权力不同。关键是哪个产品能把真实主体、允许的动作、具体资源和已确认后果连接起来。识别可疑语言的检测器很有价值,却未必是授权系统。本站的判断是:持久价值来自难以绕过的部署位置、有效上下文和明确的运维责任,而不是仪表盘上攻击标签的数量。
参考架构:具有明确授权边界的 Agent 执行链路
实线表示建议的执行路径。认证后的请求提供租户与主体上下文;检索文本只供模型参考。模型输出候选动作,可信服务解析资源并提供审批记录。限权执行器接触业务服务之前,强制执行点必须取得策略决定。审计虚线传递的是证据,而不是权限。红色拒绝分支不通往执行器。
凭据不应进入模型上下文。审批绑定动作参数及资源版本,敏感执行前再次检查撤销状态。审计把主体、策略版本、审批和操作 ID 连接到下游结果。本地持久 outbox 可以保存动作意图,却无法与独立远端后果自动原子提交;超时后仍需幂等与权威状态核对。这是本站原创参考设计,不代表任何厂商内部架构。
公司与产品定位
这是选择性行业地图,不是市场份额排名或供应商认证。覆盖范围取决于版本、集成、部署与执行模式。
Palo Alto Networks / Prisma AIRS
- 所有权/组织
- Palo Alto Networks
- 控制点
- 模型供应链 · 网关 · 运行时
Prisma AIRS 覆盖模型及制品检查、红队与运行时控制,收购 Portkey 增加了网关位置。Protect AI 已属于 Palo Alto,CyberArk 与 Koi 扩展周边的身份和端点组合。买方是希望整合供应商的企业安全团队。潜在壁垒是渠道,以及多个控制点之间关联的上下文。风险在整合:共同所有权不证明策略语义一致或全部请求都被拦截。需追问真实租户、用户和审批如何传到每个控制点,以及如何阻止直接工具调用绕过它。
Prisma AIRS documentation ↗https://docs.paloaltonetworks.com/prisma-airs/getting-started/prisma-airs-overview · FY2026 acquisition disclosures ↗https://www.sec.gov/Archives/edgar/data/1327567/000132756726000023/panw-20260731.htm
Cisco / AI Defense
- 所有权/组织
- Cisco
- 控制点
- 网络 · 验证 · Agent 身份
AI Defense 提供模型验证、嵌入网络的运行时防护、AI 资产、访问与供应链控制。Robust Intelligence 已被收购;Astrix 增加非人类身份上下文,Galileo 增加 AI 可观测能力。商业逻辑是依托 Cisco 与 Splunk 现有关系,把网络可见性延伸到身份和调查。覆盖取决于拓扑:网络检查不会自动治理本地或旁路执行。需评估哪些集成已可运行、在哪里解密、特权动作是否真的使用登记身份,以及应用资源授权是否仍由别处负责。
AI Defense capabilities ↗https://www.cisco.com/site/us/en/products/security/ai-defense/index.html · Astrix acquisition completion update ↗https://blogs.cisco.com/news/cisco-announces-intent-to-acquire-astrix-security · Galileo acquisition completion ↗https://www.cisco.com/site/us/en/about/corporate-development/acquisitions/galileo/index.html
CrowdStrike / Pangea
- 所有权/组织
- CrowdStrike
- 控制点
- 提示遥测 · 端点 · SOC
Pangea AI Guard 可报告、阻断或修改检测到的内容,AIDR 把 AI 活动接入安全工作流。CrowdStrike 还描述了 Kubernetes 提示层采集,并在 2026 年 9 月推出侧重端点的 Falcon Guardian。自然买方是已有 Falcon 的 SOC。战略价值在于把可疑 AI 交互关联到进程、文件读取或外联。必须区分采集器与后果发生前的阻断,也要区分产品发布与已核实的部署可用性。需追问哪些远程工具能被阻止,以及端点可见性之外如何落实应用审批。
AI Guard API behavior ↗https://pangea.cloud/docs/ai-guard/overview · AI detection and response ↗https://www.crowdstrike.com/en-us/solutions/ai-detection-and-response/ · Falcon Guardian announcement ↗https://www.crowdstrike.com/en-us/press-releases/crowdstrike-unveils-falcon-guardian-ai-agent-security/
Check Point / Lakera
- 所有权/组织
- Check Point
- 控制点
- 语义防护 · Agent 安全态势
Check Point AI Agent Security 文档明确标为 early access。发现与配置风险评估属于态势功能,运行时防护通过 Guard API 接入,原生平台运行时集成仍在路线图上。开发者可单独购买运行时层,形成应用侧入口;Check Point 提供企业渠道。这不证明已经实现普遍的原生执行控制。文档本身也把身份平台、云和网络控制放在架构其他位置。采购应逐一核实连接器和 API 接入,而不是把全生命周期示意图理解成自动全覆盖。
Agent Security scope and early-access status ↗https://docs.lakera.ai/docs/agent-security
SentinelOne / Prompt Security
- 所有权/组织
- SentinelOne
- 控制点
- 员工 AI · 编程助手 · MCP
Prompt Security 把 SentinelOne 扩展到员工 AI 使用、敏感数据共享及 Agent 交互,在交互点施加策略。采购需求可早于自建 Agent:员工已经在用聊天机器人和编程助手。现有端点客户关系可能降低推广阻力。但托管浏览器覆盖与后端 Agent 执行控制是不同部署。需要询问未托管设备、本地模型、支持的助手、携带凭据的工具和旁路。已知 MCP 服务器列表很长,不证明每次调用都有正确业务授权,也不证明覆盖工具的所有版本。
Prompt Security acquisition rationale and capabilities ↗https://www.sentinelone.com/press/sentinelone-to-acquire-prompt-security-to-advance-genai-security/ · Current platform ↗https://www.sentinelone.com/platform/securing-ai-prompt/
HiddenLayer
- 所有权/组织
- HiddenLayer
- 控制点
- AI 供应链 · 攻击模拟 · 运行时
HiddenLayer 区分供应链扫描、攻击模拟与生产运行时防护,并在 2026 年 8 月宣布通过原生钩子保护编程 Agent 框架。其专业定位适合持有敏感模型、部署环境多样的组织。壁垒假设是 AI 专属对抗研究与发布流程集成。应通过支持的模型格式、攻击类别、误报和绕过条件来检验。无 Agent 或无需训练数据等主张必须按模块理解,不能泛化成完全不需插桩。扫描制品也不证明训练数据无污染,或后续每个动作都已授权。
Platform modules ↗https://www.hiddenlayer.com/platform · Agent harness announcement ↗https://www.hiddenlayer.com/newsroom
Noma Security
- 所有权/组织
- Noma Security
- 控制点
- AI 态势 · 访问 · 会话行为
Noma 组合 AI-SPM、访问控制、运行时检测响应与红队,文档描述会话、身份、数据及基线上下文,并通过现有网关和钩子执行监控、告警、阻断或脱敏。差异化假设是发现“单次调用正常、组合行为危险”的序列。CISO 和平台团队需要共同部署。应验证子 Agent 身份连续、事件丢失、并发会话和延迟后果。自然语言防护不自动等于已验证授权策略。行为分数是证据,但权威租户、资源和预算状态仍需可信来源。
Platform control layers ↗https://noma.security/platform · Runtime enforcement and context ↗https://noma.security/platform/runtime-protection/
Upwind
- 所有权/组织
- Upwind
- 控制点
- 云工作负载 · 运行时上下文 · AI 传感器
Upwind 关联云态势、工作负载遥测与 AI 活动,并扫描 Agent 指令和技能上下文。它从 Agent 动作实际发生的基础设施切入:进程、凭据、数据存储与外联,适合需要按实际风险修复的云原生安全及平台买方。要区分“用 AI 调查威胁”和“保护客户 AI 的控制”。传感器覆盖与无 Agent 回退不同。需追问什么能在执行前阻断、语义意图如何进入运行时策略,以及哪些修复需人工批准。运行时可见性本身不构成业务授权。
Cloud and AI runtime architecture ↗https://www.upwind.io/feed/upwind-ai-security-securing-your-ai-stack-from-the-inside-out · Agent context scanning ↗https://www.upwind.io/feed/announcing-the-upwind-ai-context-scanner-securing-ai-agents-instructions-skills-and-model-context
Microsoft / Entra, Defender, Purview
- 所有权/组织
- Microsoft
- 控制点
- 身份 · 数据治理 · 企业 Agent
Entra Agent ID 提供 Agent 身份能力;Agent 365 安全连接 Entra、Defender 和 Purview,后者负责数据保护及合规。Microsoft 拥有现成的员工、文档与管理员工作流,渠道和既有上下文是核心战略资产,也可能挤压独立资产盘点或基础策略产品。但注册身份不证明每个工具都使用它。要核实许可证、支持平台、委托与应用权限、撤销、跨租户访问及非 Microsoft 动作。客户现有 Microsoft 环境带来集成优势,却不证明一个开关能保护全部自主工作流。
Agent identities ↗https://learn.microsoft.com/en-us/entra/agent-id/what-are-agent-identities · Agent 365 security architecture ↗https://learn.microsoft.com/en-us/security/security-for-ai/agent-365-security · Agent data governance ↗https://learn.microsoft.com/en-us/purview/ai-agents
Google Cloud / Model Armor, Wiz
- 所有权/组织
- 控制点
- 内容路径 · 云风险上下文
Model Armor 检查提示、响应及文档中的注入和敏感数据风险。文档区分只检查模式与执行控制集成,也包含 Google Cloud MCP 路径。2026 年 3 月收购的 Wiz 增加云安全上下文,本身并非通用 Agent 授权服务。Google 可把这些控制附加到托管 AI 与基础设施采购。需核实地区及语言支持:文档说明某些地区的部分非英语检查会跳过检测。记录告警不等于阻断,资源 IAM、租户隔离与精确业务审批仍是独立责任。
Model Armor deployment modes ↗https://docs.cloud.google.com/model-armor/overview · Language and regional detection boundaries ↗https://docs.cloud.google.com/model-armor/sanitize-prompts-responses · Wiz acquisition completed ↗https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/wiz-acquisition/
AWS / Bedrock AgentCore
- 所有权/组织
- AWS
- 控制点
- 工具网关 · 确定性授权
AgentCore Policy 使用 Cedar 进行细粒度工具和资源授权,条件可包含身份及工具输入。战略位置靠近执行,补充而非替代概率性的内容检测,适合已部署 AWS 或统一经 AgentCore Gateway 调用工具的开发团队。经济优势可能来自复用云身份和运维工具。但策略引擎的权威性取决于输入,不能修复模型伪造身份、绕过网关或缺失租户检查。有状态预算、幂等、审批绑定与未知外部后果,仍需应用设计处理。
Policy control boundary ↗https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/policy.html · Cedar design rationale ↗https://aws.amazon.com/blogs/security/why-policy-in-amazon-bedrock-agentcore-chose-cedar-for-securing-agentic-workflows/
沿发票流程追踪一次攻击
供应商 PDF 要求发票助手替换收款账户。检索 ACL 决定员工能否读取文档,却不证明文档中的指令可信。内容防护可能识别注入,身份与网关限制委托工具。支付服务还必须把审批绑定到租户、收款人、金额、币种及发票版本,并防止重复后果。端点和云传感器可识别异常进程或外联;审计把决策与确认结果关联。提示检测漏报时,后续边界仍应限制损害。即使检测器抓住了提示,若后端接受伪造审批,架构依然不安全。
技术关键:策略判定与强制执行分离
策略决策点(PDP)根据可信主体、租户、动作、资源及当前状态作出判断;策略执行点(PEP)拦截实际操作并落实决定。所有通向受保护后果的路径都应经过执行点,包括重试与后台任务。提示分类器提供风险信号,但不能签发身份或授予权限。这也解释了身份平台、内容防护与网关为何可以互补,却不能互相替代。
允许决定在执行前也可能过期。审批必须绑定精确动作与资源版本,并在支持时通过条件写入或事务约束执行。授权依赖不可用时,敏感动作应按明确策略拒绝或延后,不能悄悄回退到模型判断。网关无法替无关的远程服务保证内部原子性。商业分析需要追问由谁承担这些集成缺口,而不只是哪个仪表盘显示了告警。
假设采购算例:把业务阻断也计入成本
假设系统每月处理 100 万个任务,其中 100 个恶意任务。检测率为 90%、正常任务误报率为 0.1% 时,会标记 90 个恶意任务和约 1,000 个正常任务,真正恶意的只占告警约 8.3%。若每条审核 5 分钟,仅初步审核便需约 91 人时,尚未计入事件处理。这是虚构教学输入,不是产品基准。产品可能仍值得购买,但判断还需要预期损失、正常业务阻断、覆盖率、延迟、审核成本和遏制能力。不说明基准发生率与运行阈值,单比检测百分比会掩盖经济代价。
仍可能形成壁垒的未解决问题
难题仍包括:完整覆盖本地与远程 Agent,委托后保持身份连续,长任务中的权限撤销,原子预算、幂等及未知结果核对。多轮意图、记忆投毒和工具结果溯源,也超出了单条提示分类。可信的新进入者应选择一个损失高且范围明确的故障,找到真正执行控制的责任方,并在真实正常流量下证明风险降低。买方应要求失败证据:遥测丢失、不支持的语言、绕过钩子、策略服务故障和恢复如何处理。竞争单位应是可靠治理的工作流,而非扫描次数。这是待检验的分析假设,不是断言某家企业必胜。
收购、资本与战略影响
并购:区分签约与交割
Protect AI 于 2025 年 7 月 22 日加入 Palo Alto Networks;Pangea 于同年 9 月 26 日加入 CrowdStrike;Prompt Security 于 9 月 5 日加入 SentinelOne;Lakera 于 10 月 22 日加入 Check Point。这些交易已交割,不能再把它们列作独立创业公司竞争者。2026 年,Palo Alto 在 4 月 30 日宣布意向后于 5 月 29 日完成 Portkey 收购。Cisco 于 5 月 22 日完成 Galileo、6 月 29 日完成 Astrix 收购;Google 于 3 月 11 日完成 Wiz 收购。日期证明所有权变化,不证明产品全部整合。并购逻辑是让渠道、身份、遥测与执行控制互相增强;整合质量仍需验证。
资本是扩张信号,不是护城河证明
公司公告分别披露:HiddenLayer 在 2026 年 9 月 2 日完成 1 亿美元 B 轮,Noma 在 2025 年 7 月 31 日披露 1 亿美元 B 轮,Upwind 在 2026 年 10 月 28 日披露 4 亿美元 C 轮。这些数字衡量融资,不是产品收入或市场份额。专业厂商必须证明,客户为何愿意在云平台或安全套件的捆绑控制之外继续付费:复杂集成、对抗研究、跨环境可靠执行,或关键工作流控制权。大平台拥有采购关系与已部署遥测,但身份、应用状态与远程后果之间仍可能存在缝隙。融资金额和完成并购都不能回答执行质量问题。
一手产品与交易来源
- Protect AI acquisition completed ↗https://investors.paloaltonetworks.com/news-releases/news-release-details/palo-alto-networks-completes-acquisition-protect-ai ·
- Pangea acquisition closing disclosed in annual filing ↗https://www.sec.gov/Archives/edgar/data/1535527/000153552726000010/crwd-20260131.htm ·
- Prompt Security transaction completed ↗https://www.sec.gov/Archives/edgar/data/1583708/000110465925088079/tm2525181d1_8k.htm ·
- Lakera acquisition accounting disclosure ↗https://www.sec.gov/Archives/edgar/data/1015922/000117891326001932/R11.htm ·
- Portkey acquisition completed ↗https://investors.paloaltonetworks.com/news-releases/news-release-details/palo-alto-networks-completes-acquisition-portkey-secure-ai ·
- Astrix acquisition: completed update ↗https://blogs.cisco.com/news/cisco-announces-intent-to-acquire-astrix-security ·
- Galileo acquisition completed ↗https://www.cisco.com/site/us/en/about/corporate-development/acquisitions/galileo/index.html ·
- Google completes Wiz acquisition ↗https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/wiz-acquisition/ ·
- HiddenLayer Series B announcement ↗https://www.hiddenlayer.com/news/hiddenlayer-100m-series-b-ai-security ·
- Noma Series B announcement ↗https://noma.security/blog/noma-security-raises-100m-to-drive-adoption-of-ai-agent-security ·
- Upwind Series C announcement ↗https://www.upwind.io/blog/the-next-wave-is-here-upwinds-400m-series-c-to-build-the-cloud-ai-security-platform ·
- Prisma AIRS documentation ↗https://docs.paloaltonetworks.com/prisma-airs/getting-started/prisma-airs-overview ·
- FY2026 acquisition disclosures ↗https://www.sec.gov/Archives/edgar/data/1327567/000132756726000023/panw-20260731.htm ·
- AI Defense capabilities ↗https://www.cisco.com/site/us/en/products/security/ai-defense/index.html ·
- AI Guard API behavior ↗https://pangea.cloud/docs/ai-guard/overview ·
- AI detection and response ↗https://www.crowdstrike.com/en-us/solutions/ai-detection-and-response/ ·
- Falcon Guardian announcement ↗https://www.crowdstrike.com/en-us/press-releases/crowdstrike-unveils-falcon-guardian-ai-agent-security/ ·
- Agent Security scope and early-access status ↗https://docs.lakera.ai/docs/agent-security ·
- Prompt Security acquisition rationale and capabilities ↗https://www.sentinelone.com/press/sentinelone-to-acquire-prompt-security-to-advance-genai-security/ ·
- Current platform ↗https://www.sentinelone.com/platform/securing-ai-prompt/ ·
- Platform modules ↗https://www.hiddenlayer.com/platform ·
- Agent harness announcement ↗https://www.hiddenlayer.com/newsroom ·
- Platform control layers ↗https://noma.security/platform ·
- Runtime enforcement and context ↗https://noma.security/platform/runtime-protection/ ·
- Cloud and AI runtime architecture ↗https://www.upwind.io/feed/upwind-ai-security-securing-your-ai-stack-from-the-inside-out ·
- Agent context scanning ↗https://www.upwind.io/feed/announcing-the-upwind-ai-context-scanner-securing-ai-agents-instructions-skills-and-model-context ·
- Agent identities ↗https://learn.microsoft.com/en-us/entra/agent-id/what-are-agent-identities ·
- Agent 365 security architecture ↗https://learn.microsoft.com/en-us/security/security-for-ai/agent-365-security ·
- Agent data governance ↗https://learn.microsoft.com/en-us/purview/ai-agents ·
- Model Armor deployment modes ↗https://docs.cloud.google.com/model-armor/overview ·
- Language and regional detection boundaries ↗https://docs.cloud.google.com/model-armor/sanitize-prompts-responses ·
- Policy control boundary ↗https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/policy.html ·
- Cedar design rationale ↗https://aws.amazon.com/blogs/security/why-policy-in-amazon-bedrock-agentcore-chose-cedar-for-securing-agentic-workflows/ ·
Source and maintenance notes
Primary links are placed beside the claims they support. Acquisition amounts retain their original basis; undisclosed values remain undisclosed. Product specifications and release statements are vendor disclosures, not comparative evidence from Token Lab. The diagrams and reference calculations are original educational material. Corrections should identify the section, source and proposed replacement; a source-access date is not a paper publication date.
All frontier research · Top Company Frontier · Editorial standards
实践与验证
14 · 用攻击与正常任务共同验证控制
建立完全本地的合成环境:两租户发票、mock工具和假支付账本。保持合法流程相同,每次只改变检索过滤、凭据范围、批准绑定、出站策略或审计持久性中的一个边界。明确攻击者可控输入、禁止后果、应执行检查的位置和仍须成功的正常结果。
按实际后果计分:未授权读取、披露或状态改变才是定义中的攻击成功,不是出现吓人的文字。同时报告正常任务成功、误拒绝、延迟和成本;说明分母是攻击尝试、独立任务、载荷还是完整会话。小型精选集零失败不代表生产零风险,相关变体也不是独立证据。
加入正常引用指令、合法重试、获准账户变更和普通外链等负对照。可让攻击者获取有限反馈以做自适应攻击,但最终集不能进入开发循环。比较时匹配预算:十次昂贵裁判与一次廉价检查的差异不能全归因于算法。
研究问题包括动态组合工具时策略是否仍可理解、来源能否跨摘要与委托保留且不过量保存私人数据、自动审计能否区分合理故事与真正有效干预。与确定性简单基线比较并公开失败。目标是跨生命周期的可辩护安全论证,不是单模型或产品包治所有风险。
实践:解释架构,而不仅是术语
以下是本站原创练习,并非任何公司的真实面试题。展开答案前,画出信任边界,并指出实际执行控制的组件。
模型拒绝了 PDF 中的注入指令。支付流程就安全吗?
提示: 找出哪个组件真正能够发起支付。
展开答案与分析
不能据此判断安全。拒绝只是一次观察结果;必须检查确定性的授权、审批绑定和工具凭证。将恶意支付提案交给同一个授权关口,检查账本没有产生被禁止的效果。同时验证合法支付仍能完成;一次拒绝不能证明对所有攻击都安全。
模型 API 密钥与用户委托令牌有什么不同?
提示: 区分推理服务的调用权限和业务资源的操作权限。
展开答案与分析
模型密钥允许调用推理服务;委托令牌限定业务操作的执行者、接收方、范围和有效期。两者都不应进入模型上下文。验证接收方、对象级权限和到期时间,不能用持有模型密钥来替代业务授权。
审批人批准向供应商 A 支付 100 美元,但执行前账户变更,怎么办?
提示: 比较审批时的动作与当前资源版本。
展开答案与分析
根据策略拒绝或重新审批。审批应绑定规范化金额、币种、收款方和相关资源版本,执行时原子地验证这些条件。先检查、随后无保护地写入,会留下并发修改窗口。
支付请求发出后工具超时,代理可以重试吗?
提示: 超时是观测结果,并不证明支付未执行。
展开答案与分析
复用稳定操作标识和幂等键,并通过权威记录对账。只有服务端承诺对应的幂等语义时,才能重试完全相同的绑定请求;否则保留结果未知状态并升级处理。生成新键可能造成重复支付。
签名回执写着支付完成,它证明了什么?
提示: 区分完整性、授权和业务事实。
展开答案与分析
在密钥信任假设成立时,它证明持有签名密钥的一方签发了这些字节;它不能单独证明记录完整、授权正确或资金结算成功。必须关联策略决策、审批记录和独立账本,并处理缺失或矛盾证据。
评分标准:1 分能指出风险;2 分能定义信任边界;3 分能说明身份、动作和资源检查由谁执行;4 分能用正反用例验证实际效果;5 分能处理并发、撤销、证据缺口及剩余风险。
综合项目:实现本地双租户发票模拟器,包含精确审批、文档注入、执行超时和可重建的审计链。证明禁止动作被阻止且合法任务完成,并说明模拟器距离生产系统还缺少什么。
实践:逐个验证信任边界
用同一合法发票流程运行以下合成案例。检查后端状态与审计证据,而不只看助手文字。
身份:两租户同名发票、成员移除和篡改租户字段不得造成越界。
模型与供应链:相同任务下比较正常与可疑版本;拒绝格式错误、超大或未注册工具输出,处理服务商超时。
检索:注入文档不得授予权限,正常引用指令和特殊排版仍应可用。
委托:拒绝错误audience、缺scope、过期授权和子Agent扩权。
批准:改变一分钱、币种、目标或资源版本,以及到期和撤销,均不得产生未授权提交。
执行:拒绝非法目标与路径、限制资源耗尽,同时保留合法附件处理。
重试:重复投递、响应丢失与工作者重启须向业务账本核对,不产生意外重复效果。
记忆:检查跨租户泄露、注入经过摘要仍存在,以及源删除后危险派生记录残留。
审计:缺事件、ID不一致、伪造结果及审计库故障应触发预定失败行为。
同时报告攻击阻断与正常效用。说明身份提供方信任、跨系统并发边界和必须人工核对的情况,不承诺全局原子性。
Run the authority boundary, not just a prompt
The model can propose an Action. The application supplies the trusted Grant, approval record, clock and stable operation key. This simulator checks their relationship before a ledger write. It also rejects a second payment under a different retry key. All records are synthetic and in memory; there is no network, credential or payment service.
Python 3.10+, standard library, deterministic: download agent-authority.py.
python3 agent-authority.py"""Offline policy-and-receipt simulator, Python 3.10+, standard library only.
Not production IAM, a payment API, or a cryptographic audit implementation.
The caller is trusted test code. Untrusted model output supplies Action only.
"""
from dataclasses import dataclass, replace
from hashlib import sha256
import json
@dataclass(frozen=True)
class Action:
tenant: str
invoice: str
revision: int
payee: str
cents: int
operation: str = 'pay'
@dataclass(frozen=True)
class Grant:
user: str
tenant: str
audience: str
invoice: str
max_cents: int
expires: int
revision: int
def digest(action):
return sha256(json.dumps(action.__dict__, sort_keys=True,
separators=(',', ':')).encode()).hexdigest()
class Ledger:
def __init__(self):
self.commits = {} # durable and transactional in a real implementation
self.events = []
self.revoked = set()
self.audit_available = True
self.invoice_revision = 4
def execute(self, action, grant, approval, key, now):
"""approval is a trusted approval-service record, not model text.
Single-threaded transaction simulation; no exactly-once network claim.
"""
reason = None
if grant.user in self.revoked: reason = 'revoked'
elif now >= grant.expires: reason = 'expired'
elif grant.audience != 'ledger': reason = 'audience'
elif action.tenant != grant.tenant: reason = 'tenant'
elif action.invoice != grant.invoice: reason = 'resource'
elif action.operation != 'pay': reason = 'operation'
elif type(action.cents) is not int or not 0 < action.cents <= grant.max_cents: reason = 'amount'
elif action.revision != self.invoice_revision or grant.revision != self.invoice_revision: reason = 'stale_revision'
elif approval != digest(action): reason = 'approval_binding'
elif not self.audit_available: reason = 'audit_unavailable'
elif key in self.commits:
if self.commits[key]['digest'] != digest(action): reason = 'idempotency_conflict'
else: return self.commits[key] # same receipt, no second write
if reason is None and any(r['invoice'] == action.invoice and r['tenant'] == action.tenant for r in self.commits.values()):
reason = 'already_settled'
if reason:
self.events.append({'state': 'denied', 'reason': reason})
return {'state': 'denied', 'reason': reason}
receipt = {'state': 'committed', 'digest': digest(action), 'key': key,
'actor': grant.user, 'tenant': grant.tenant,
'invoice': action.invoice, 'revision': action.revision, 'policy': 'teaching-v1'}
# In production: commit effect and durable audit/outbox atomically,
# or use downstream idempotency + reconciliation; this is in-memory.
self.commits[key] = receipt
self.events.append(dict(receipt))
return receipt
action = Action('tenant-a', 'inv-7', 4, 'supplier-42', 5000)
grant = Grant('user-1', 'tenant-a', 'ledger', 'inv-7', 10000, 100, 4)
approval = digest(action)
ledger = Ledger()
assert ledger.execute(action, grant, approval, 'op-1', 50)['state'] == 'committed'
assert ledger.execute(action, grant, approval, 'op-1', 51)['state'] == 'committed'
assert len(ledger.commits) == 1
assert ledger.execute(action, grant, approval, 'different-key', 51)['reason'] == 'already_settled'
assert ledger.execute(replace(action, tenant='tenant-b'), grant, approval, 'op-2', 50)['reason'] == 'tenant'
assert ledger.execute(replace(action, payee='attacker'), grant, approval, 'op-2', 50)['reason'] == 'approval_binding'
assert ledger.execute(replace(action, cents=10001), grant, approval, 'op-2', 50)['reason'] == 'amount'
assert ledger.execute(action, replace(grant, audience='search'), approval, 'op-2', 50)['reason'] == 'audience'
assert ledger.execute(action, grant, approval, 'op-2', 100)['reason'] == 'expired'
ledger.invoice_revision = 5
assert ledger.execute(action, grant, approval, 'op-2', 50)['reason'] == 'stale_revision'
ledger.invoice_revision = 4
ledger.audit_available = False
assert ledger.execute(action, grant, approval, 'op-2', 50)['reason'] == 'audit_unavailable'
ledger.audit_available = True
ledger.revoked.add('user-1')
assert ledger.execute(action, grant, approval, 'op-2', 50)['reason'] == 'revoked'
assert len(ledger.commits) == 1
print('13 checks passed: approval, tenant, audience, budget, revision, expiry, audit, revocation and replay.')
print('One committed effect. Model-proposed changes cannot expand the trusted grant.')
13 checks passed: approval, tenant, audience, budget, revision, expiry, audit, revocation and replay.
One committed effect. Model-proposed changes cannot expand the trusted grant.Production gaps: a digest is not a signature. Real approval records need authenticated issuance, approver authority, expiry and replay controls. Validate tokens cryptographically; use a transactional database, concurrency controls, authoritative invoice state, downstream idempotency and reconciliation. This single-threaded example does not prove exactly-once delivery or tamper-proof audit storage.
6. Data that actually tells us something
AgentDojo provides a concrete research substrate: 97 tasks and 629 security test cases involving tool-using agents. The original arXiv submission was June 19, 2024; this survey pins v3, November 24, 2024, whose revision notes include an implementation bug fix and travel-suite changes. Paper abstracthttps://arxiv.org/abs/2406.13352v3 · PDFhttps://arxiv.org/pdf/2406.13352v3 · Author codehttps://github.com/ethz-spylab/agentdojo. Those counts describe a benchmark, not the prevalence of attacks in customer deployments.
Three distinct quantities are needed. Clean utility measures completion without attack. Attack success measures a defined attacker objective under a specified opportunity and budget. Defended utility measures legitimate completion with the defense present. Report all three. An agent that cannot complete a banking task in the clean condition may also fail to execute the injected action; that is not necessarily a successful defense.
Denominators must be explicit. Was success counted per attack attempt, per task, or after the adversary tried many candidates and selected the best? Did the attacker see the defense or query its feedback? Was tool execution inspected, or only the final natural-language answer? A refusal message can accompany an earlier unauthorized tool call. A screenshot of a safe-looking answer is weak evidence about the full trajectory.
Assume 100,000 requests, an attack prevalence of 0.1%, a true-positive rate of 95%, and a false-positive rate of 1%. Expected counts are 95 true alerts, 999 false alerts and five missed attacks. Only 8.68% of alerts correspond to malicious requests. A seemingly small false-positive rate can overwhelm operations when attacks are rare.
Here π is attack prevalence, r is true-positive rate and f is false-positive rate. These are probabilities, not percentages entered as whole numbers. The formula follows by dividing expected true positives by all expected positive predictions. It does not imply that deployment prevalence is known or that adaptive attacks remain distributed like test samples.
Local false-alarm experiment
95 true alerts; 999.0 false alerts; alert precision 8.68%.
Population, prevalence and recall stay fixed at the stated teaching assumptions. No inputs leave your browser.
Download security-base-rate.py. Python 3.10+, standard library only; deterministic arithmetic, no random seed, network requests or model API. Run python3 security-base-rate.py.
"""Expected counts in a synthetic population, not vendor performance."""
from math import isclose
def audit(n, prevalence, tpr, fpr):
if not isinstance(n, int) or isinstance(n, bool) or n <= 0:
raise ValueError('n must be a positive integer')
if not all(0 <= x <= 1 for x in (prevalence, tpr, fpr)):
raise ValueError('Rates must be in [0, 1]')
tp = n * prevalence * tpr
fn = n * prevalence * (1 - tpr)
fp = n * (1 - prevalence) * fpr
tn = n * (1 - prevalence) * (1 - fpr)
precision = tp / (tp + fp) if tp + fp else None
return tp, fp, fn, tn, precision
if __name__ == '__main__':
tp, fp, fn, tn, precision = audit(100000, .001, .95, .01)
assert isclose(tp, 95)
assert isclose(fp, 999)
assert isclose(fn, 5)
assert isclose(tp + fp + fn + tn, 100000)
assert isclose(precision, 95/1094)
assert audit(100, 0, 1, 0)[4] is None
assert audit(100, 1, 1, 0)[4] == 1
try:
audit(100, -.1, .95, .01)
except ValueError:
pass
else:
raise AssertionError('Invalid rate accepted')
print(f'TP={tp:.0f}, FP={fp:.0f}, FN={fn:.0f}, TN={tn:.0f}')
print(f'Alert precision={100*precision:.2f}%')
print('8 checks passed')
Expected output:
TP=95, FP=999, FN=5, TN=98901
Alert precision=8.68%
8 checks passedBefore reducing f to 0.1%, predict the outcome: false alerts become 99.9 and precision approximately 48.74% if recall is unchanged. In a real detector, changing the threshold generally changes recall too. The calculator isolates the arithmetic; it is not a threshold-tuning algorithm. A missed low-impact test attack and an unauthorized high-value transaction also have different costs.
7. Build a fair security evaluation
Start with a workflow inventory and concrete forbidden outcomes: sending data to a non-owner, accessing another tenant's record, running an unapproved command, or changing an authorization policy. Define the adversary's entry point and budget. A user who directly owns the account and a third party controlling one retrieved page are different adversaries; do not mix their results.
Maintain a clean task set with validated expected outcomes. Create attack families on separate development fixtures and freeze a held-out set before tuning. Include benign text that resembles an attack, multiple languages, long contexts, interrupted tool sequences and malicious content that appears only after an initial harmless step. Track software and policy versions so that a changed score can be attributed to an intervention.
Use four baselines: the original agent; a detector-only variant; a deterministic tool-policy gate; and their combination. Keep model, tools, tasks, allowed scopes and attack budgets fixed. Measure forbidden actions, clean success, defended success, false blocks, latency, cost and coverage of action paths. An ablation must not give one system fewer privileges or easier tasks without acknowledging the change.
Inspect complete traces. Validate tool arguments and actual side effects, not the model's description of them. Verify that denied actions did not execute, then verify that permitted actions still complete. Include timeouts, retries, expired approvals and revocation during a multi-step run. An observer that alerts after a transfer is a detective control; it is not preventive enforcement.
Repeat stochastic model runs and report uncertainty. Group results by task family to expose correlated failures. Repeated attempts against one template do not supply the same evidence as independent families. For zero observed violations, use a confidence bound with explicit assumptions; see the alignment evidence tutorial. Never turn 0/n into a promise of zero future risk.
8. Original interview practice and engineering decisions
These are original simulations, not verified questions from the companies discussed. Answer with a threat model and an observable test, not a list of product names.
1. Why can 95% attack recall still create an unusable alert queue?
First hint: Use prevalence and the benign population as well as recall.
Answer and analysis
At 0.1% prevalence in 100000 requests, there are 100 malicious and 99900 benign cases. A 1% false-positive rate creates 999 false alerts, exceeding 95 true alerts. Discuss triage cost, threshold trade-offs and the uncertainty of the prevalence estimate.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
2. Why does an ethical explanation not authorize a tool call?
First hint: Separate behavior prediction from authoritative permission.
Answer and analysis
An explanation is generated content. A policy service should resolve identity, resource scope, recipient and approval from authoritative state. Test a persuasive proposal with an out-of-scope recipient and prove that the executor denies it before any side effect.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
3. How could a defense achieve zero attack success while being useless?
First hint: Inspect clean task completion.
Answer and analysis
Refusing everything, disabling every tool, or failing the workflow can prevent attacker objectives while also preventing authorized work. Report utility and attack success together, plus the exact permissions and denominators.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
4. What changes when an agent can write memory or improve its own prompt?
First hint: Think across runs, not only within one request.
Answer and analysis
Poisoned state can persist after the original input disappears. Version memory and prompts, restrict writes and evaluate candidates on untouched tasks with an independent gate. Test rollback and ensure the candidate cannot alter evaluation evidence.
Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.
For an engineer, a strong first project is a small controlled executor with auditable permissions and adversarial fixtures. For a researcher, the next step is a task family that exposes a specific failure in existing defenses without redefining success after seeing results. Continue with bounded tools, authorization after learning and the infrastructure companion.
9. Frontier research agenda: five falsifiable projects
The following are Token Lab's proposed research directions, not claims that these methods have already succeeded or are novel. Search the literature before claiming novelty. Each proposal starts with an engineering failure and ends with evidence that would count against the idea. Select one narrow workload for a first PoC; broad claims require replication across tasks, implementations and operating conditions.
1. Semantic authorization beyond a tool allowlist
Why existing approaches are insufficient: An API can be allowed while a particular recipient, amount or purpose is not. Natural-language requests often underspecify the legitimate scope, and agents revise plans as they work.
Testable hypothesis: A typed action proposal bound to resource-level policy and explicit approvals reduces unauthorized effects without excessive false blocks.
Baselines and fair comparison: Compare prompt-only restrictions, tool-name allowlists and resource-scoped enforcement. Use tasks with the same tool but different ownership and destinations.
Measurements: Measure actual unauthorized effects, authorized completion, approval burden and policy coverage. Mutate parameters after approval and test revocation between planning and execution.
What would falsify the claim: Reject if the evaluator accepts self-reported authorization or if benign success collapses. A typed schema checks structure; it does not by itself prove that the values are permitted.
2. Adaptive prompt injection under a fixed attack budget
Why existing approaches are insufficient: A defense tuned to a static phrase collection can learn surface patterns. An attacker can observe failures, alter content and exploit a different point in the workflow.
Testable hypothesis: Training or detection informed by provenance and task boundaries generalizes better than phrase matching at matched clean utility.
Baselines and fair comparison: Compare a phrase baseline, a model detector and an execution gate, with separate adaptive attack runs against each. Match query and token budgets.
Measurements: Report success per attempt and per task after the full budget, clean utility, attack cost and failures on unseen task families. Preserve all attempted attacks, not only successful examples.
What would falsify the claim: Reject if improvement disappears when attackers adapt, or if one defense gets less attack effort. Do not report a fixed-template score as a guarantee against strategic adversaries.
3. Long-horizon memory poisoning and recovery
Why existing approaches are insufficient: A single harmless-looking note can influence later decisions after the original document is gone. Most short episodes miss persistence, delayed triggers and the cost of removing contaminated state.
Testable hypothesis: Provenance-aware memory writes with quarantine and selective rollback can reduce persistent influence while retaining useful learned preferences.
Baselines and fair comparison: Compare no persistent memory, unrestricted memory and the proposed write policy. Use multi-session legitimate tasks interleaved with controlled contamination fixtures.
Measurements: Measure delayed violations, useful retention, time to detect, rollback completeness and collateral forgetting. Inspect every downstream copy of the tainted state.
What would falsify the claim: Reject if the defense succeeds only by deleting all memory or assumes a known infection time. Recovery must address derived summaries and replicas, not only the original record.
4. Multimodal and multilingual boundary preservation
Why existing approaches are insufficient: A security decision made on extracted text may not cover instructions encoded in an image, an audio segment or layout. Translation and OCR change what the model sees.
Testable hypothesis: Tracking provenance through preprocessing and testing modality-specific transformations improves coverage at matched utility.
Baselines and fair comparison: Compare a text-only detector, independent modality filters and the proposed end-to-end pipeline on paired inputs expressing the same authorized task.
Measurements: Report results by language and modality, extraction errors, false blocks, task quality and latency. Include benign ambiguous layouts and low-quality scans.
What would falsify the claim: Reject if a global average hides failure in one language, or if the defense gets access to a clean transcription unavailable in production. Human labels need agreement checks too.
5. Security evaluation for self-improving agents
Why existing approaches are insufficient: When candidates alter prompts, tools or code, the evaluator and reward become attractive targets. Passing more tests may mean learning the tests rather than improving robust capability.
Testable hypothesis: An isolated promotion gate with held-out task families and immutable execution receipts can detect more invalid improvements than an in-loop score alone.
Baselines and fair comparison: Compare self-scoring, a shared-state evaluator and an independent evaluator. Introduce benign improvements, reward exploits and unauthorized tool-scope changes as distinct candidate types.
Measurements: Measure legitimate gain, exploit acceptance, retained abilities, scope changes and audit reproducibility. Keep a final test set untouched until selection finishes.
What would falsify the claim: Reject if the candidate can edit the score, if test contamination is ignored, or if safety comes only from forbidding all changes. A useful gate must accept genuine improvements as well as reject exploits.
From a two-week PoC to a defensible report
In the first stage, write the system boundary, assumptions and failure criterion before implementing the proposed method. Build the simplest baseline and a reproducible fixture. In the second stage, add one intervention and run controlled sweeps; retain failed runs and explain exclusions. In the final stage, test an unseen workload or attack family and write a limitations section. Separate what your experiment measured from what you hope will generalize.
A useful report contains a configuration manifest, data-generation procedure, code commit, exact commands, raw observations, aggregation script and a table of threats to validity. Use multiple seeds when randomness matters; repeated deterministic arithmetic is not independent evidence. Publish negative results: the boundary where a technique stops working is often the most transferable lesson.