Token LabSearch

English article. Synthetic examples and vendor claims are identified separately.

AI SECURITY / SYSTEMS SURVEY

AI security in 2026: securing models, data and autonomous actions

Follow an enterprise agent from LLM calls and IAM through prompt injection, least privilege, exact approval, tool execution, audit and incident recovery.

Sources & version details
Contents · lifecycle and reference chapters
In this article
PART I / ONE SYSTEM, END TO END

Security: from the user to the committed action

This original case study follows an invoice assistant. It reads tenant-scoped documents and proposes a payment; executing a payment requires separate, exact authorization. The scenario and experiments are synthetic teaching material, not a reported deployment.

Security: from the user to the committed action: conceptual lifecycle diagram
Original architecture illustration. Lines represent dependencies or evidence, not measured performance.

01 · Follow one invoice assistant from request to consequence

At 09:00, Maya, an accounts-payable analyst, asks an assistant: “Find the overdue invoices from Northstar, explain discrepancies, and prepare a payment proposal.” This is a synthetic teaching scenario, not a reported incident. Maya may read her tenant’s invoices and create proposals. She may not silently change a supplier’s bank account or execute a payment. A separately authorized approver must approve the exact transaction before execution. The business objective is useful assistance within that boundary—not a chatbot that refuses every finance-related request.

The application authenticates Maya, retrieves permitted records, sends selected context to a model, interprets the returned plan, invokes tools, requests approval if necessary, executes an authorized operation and records its outcome. Every arrow crosses a boundary: between people and services, instructions and evidence, tenants, providers, tools, time, or administrative domains. Security engineering asks what each recipient is entitled to trust at that boundary.

A supplier can control an uploaded PDF but not Maya’s role. A compromised retrieval connector can return false content but should not mint payment credentials. A model can propose a tool call but should not grant itself permission. A stolen worker token may allow some actions, so the system must constrain what that credential can do even when all model behavior is adversarial. This distinction between malicious content and usable authority drives the entire architecture.

Before proceeding, understand authentication as establishing a principal, authorization as deciding whether that principal may perform a specific operation, and an audit record as evidence about a decision or event. None implies the others. By the end, you should be able to draw a trust-boundary diagram, write an authorization invariant, design negative tests, reconstruct a disputed action and explain the remaining uncertainty. Reading one article creates a systematic foundation; expert judgment also requires implementing and testing these controls in a real environment.

02 · Before runtime: define assets, actors and forbidden outcomes

The threat model starts before the first prompt. List the assets: invoice content, supplier identities, bank details, approval state, payment credentials, model-provider keys, logs and persistent memory. Then name the actors and what they can influence. Our external supplier controls a document; a normal employee controls their request; a tenant administrator controls some integrations; an attacker may steal a session or compromise a plugin. Avoid an omnipotent attacker who already controls every server: that assumption obscures which control is being tested.

Write concrete invariants. A retrieved document must never expand authority. Tenant A’s request must not retrieve Tenant B’s invoice. A payment must match a valid approval’s tenant, payee, amount, currency and proposal version. A retry must not create another payment. Revocation must take effect before a new sensitive action is authorized. A model-generated explanation cannot replace any of these conditions. These statements are useful because a test can demonstrate a violation.

Inventory dependencies as part of this model: model versions, tokenizers, orchestration libraries, tool manifests, browser automation packages, container images and retrieval pipelines. A malicious package or changed tool description enters through a different channel from a malicious invoice. Pin and review deployable artifacts, separate development from production credentials, and require controlled changes to the tool registry. A signature establishes an artifact’s claimed origin and integrity under its trust assumptions; it does not establish benign behavior.

The architectural principle is resource-specific trust, not “everything inside the network is safe.” NIST SP 800-207, published August 2020https://csrc.nist.gov/pubs/sp/800/207/final provides a zero-trust foundation. Our invoice design applies that principle to model-mediated actions.

Distinguish prompt injection from model or data poisoning. Prompt injection steers an existing model through runtime content. Poisoned training examples, a malicious fine-tune or a modified model artifact can change learned behavior or the deployed computation itself. Protect dataset lineage, training permissions, artifact provenance and release gates separately. Even a trusted model artifact must remain unable to bypass business authorization; provenance narrows the supply-chain question but does not grant runtime authority.

03 · Request entry: preserve the human, tenant and workload identities

Maya signs in through the application’s normal identity system. The gateway validates the session and resolves the tenant from server-side membership, not from a tenant name in her prompt. The request context contains an authenticated subject, tenant, session or delegation identifier and the application’s workload identity. These are distinct: the person asked for the task; a service executes it; a model generates candidate text. Calling a model “the agent” does not make it an independently trustworthy identity.

The first attack surface is ordinary application security. Session theft, cross-site request forgery, insecure object references and a client-supplied tenant identifier can defeat the system before prompt injection is relevant. Apply the protections appropriate to the session and transport design. Resolve invoice identifiers under the authenticated tenant and operation. Do not fetch an invoice by global ID and assume the model will notice that it belongs to somebody else.

For a background job, persist the delegation context and its expiry or revocation reference. Do not serialize an entire browser session into the queue. When a long-running task resumes, re-establish that the relevant authority still exists. A user who was authorized yesterday may have left the organization today. A durable task record is not a durable permission grant.

04 · Calling the LLM: an inference service is not an authorization service

The application constructs a model request from the user’s task, relevant policy guidance and a bounded set of evidence. The model-provider API credential authenticates the application to the inference service. It is not Maya’s invoice-reading credential and must not become a payment credential. Store it outside the prompt and retrieve it through a controlled server-side mechanism. Never ask the model to carry secrets between tools.

Treat the provider boundary explicitly. Determine what data may leave the application’s environment, which endpoint and region are approved, how request logging and retention work, and whether a fallback provider has the same permitted data handling. These are deployment- and contract-specific checks, not assumptions inferred from a model’s brand. If the approved endpoint is unavailable, failing over to an unreviewed endpoint can turn an availability mechanism into a data disclosure.

Bound inference requests by token, time and monetary budgets. A long document or adversarial loop can exhaust shared capacity without ever obtaining a prohibited tool action. Redact or omit fields not needed for the task, while recognizing that naive redaction can also remove evidence necessary for correctness. Separate the minimum information needed to explain an invoice from the information needed to execute a payment.

On response, parse the model’s output against a versioned schema and enforce size limits. A valid JSON tool call is only a well-formed proposal. Its strings remain untrusted inputs to SQL, HTML, shells, URLs and downstream APIs. Parameterize database access and use narrow typed adapters. The model can still produce a persuasive wrong plan; the next boundaries must contain its consequences.

05 · Retrieval: permission to read does not make content trustworthy

The retriever searches only within Maya’s authorized data scope. Apply document-level access rules before sensitive content enters the model context, and preserve policy checks when fetching full documents after vector search. Filtering only the final answer is too late: the model or an external provider may already have received the secret. Cache keys and retrieval indexes must respect tenant and authorization boundaries; a shared semantic cache can leak a previously authorized answer to a different user.

Now the supplier’s PDF contains text asking the assistant to ignore the original task and change payment details. Maya is allowed to read the PDF, but its author is not allowed to issue commands to her application. The attack crosses the boundary from evidence into instructions. This is indirect prompt injection: the instruction reaches the model through material retrieved for another purpose. A direct injection comes through the user-facing input channel. Both differ from a software exploit that obtains arbitrary execution, although either can lead to harm through vulnerable tools.

Preserve provenance: source ID, tenant, retrieval time, document version, access decision and the distinction between trusted application policy and untrusted evidence. Delimiters, instruction hierarchy, content scanning and injection-resistant training can reduce some failures, but they are not a proof that an adversarial document cannot influence the model. OWASP’s LLM01:2025 Prompt Injection guidancehttps://genai.owasp.org/llmrisk/llm01-prompt-injection/ is a useful threat taxonomy; this article pins that edition rather than implying a current risk ranking.

06 · Planning and tool calls: separate a suggestion from authority

The model proposes “read invoice,” “look up supplier” and perhaps “prepare payment.” The orchestrator resolves each proposal to a registered adapter. It must not expose a generic administrator shell merely because three narrow business APIs are inconvenient to implement. Tool metadata is configuration that deserves review: a changed description can alter model behavior, while a changed implementation can alter actual authority. Record both versions.

Place a deterministic policy-enforcement point immediately before each privileged operation. It asks an authoritative policy decision point whether this subject, workload, tenant, action, resource and current context are allowed. The model may explain its reasoning, but cannot edit the policy inputs that establish identity or bypass the check. A refusal generated by a model is useful defense in depth; an allow decision generated only by a model is not the same as an enforced access-control boundary.

For this workflow, the proposal tool can create a draft with a selected invoice and intended payee. It cannot execute payment or change the supplier master record. Those are distinct actions with distinct permissions. The allowed action set should be the intersection of the user’s rights, the service’s rights, the task’s delegated scope and applicable policy—not the union of whatever any participant can do.

Make the policy decision concrete. For a proposed payment of 120 USD on invoice INV-42, the trusted context identifies Maya, tenant T1 and the payment worker; the resource resolver supplies the authoritative invoice tenant, outstanding amount and supplier revision. A permit requires matching tenant, payment scope, sufficient task budget, an eligible invoice and an approval bound to the same amount, currency and destination revision. The model supplies only a candidate action. It must not supply authoritative membership, the remaining budget or an approval flag.

Evaluate all applicable restrictions together. If an independently authorized approver may approve up to 500 USD but this task delegates only 100 USD, the 120 USD proposal is denied. Splitting it into two 60 USD requests must not bypass the aggregate task budget. Reserve budget atomically with authorization or commit, reconcile uncertain outcomes, and release reservations only when failure is established. Otherwise two concurrent workers can each observe sufficient budget and overspend. These amounts are synthetic examples, not a recommended financial policy.

07 · Least privilege: scope credentials to the actual operation

A broad service account creates a confused deputy: the application has authority that the requester or retrieved document does not, and can be persuaded to exercise it on their behalf. “The agent has a token” therefore answers the wrong question. Ask whose authority the token represents, which resource server may accept it, which actions and objects it permits, when it expires, and whether it can be replayed or delegated.

Prefer short-lived, resource-appropriate credentials obtained through a trusted broker or supported authorization flow. Preserve the actor and subject distinction where delegation is supported. A token intended for the invoice service should not be accepted by the payment service. Audience checks constrain the recipient; scopes constrain a class of operations; object- and tenant-level policy still determine which invoice may be read. A scope called read does not automatically mean “read only this invoice.”

RFC 9700, January 2025https://www.rfc-editor.org/rfc/rfc9700.html recommends access-token privilege restriction, audience restriction and suitable sender-constraining mechanisms. RFC 8693, January 2020https://www.rfc-editor.org/rfc/rfc8693.html specifies token exchange, including delegation-related representations. Neither means every provider supports arbitrary downscoping, nor that requesting a narrower token makes it narrower: the authorization server must enforce the resulting policy.

For MCP integrations, validate credentials for the intended server and use a separate legitimate authorization relationship for downstream APIs. The MCP security guidance, pinned to the 2025-11-25 documentation editionhttps://modelcontextprotocol.io/docs/2025-11-25/tutorials/security/security_best_practices describes confused-deputy and token-passthrough risks. Do not simply forward an upstream bearer token to another service. Short lifetimes reduce exposure; they do not eliminate misuse during the remaining lifetime.

08 · Approval: bind consent to an exact, current action

The assistant presents a payment proposal to an authorized approver. “Approve this agent” is too vague. Show the invoice, verified supplier identity, destination account reference, amount, currency and intended execution time in a trustworthy interface. Render untrusted descriptions as data, and distinguish them from application-generated labels. An attacker should not be able to hide a changed destination inside persuasive prose.

Create an approval record bound to a canonical representation of the exact action, its tenant, resource versions, approver, expiry and a unique nonce. Hashing that representation can make comparison convenient, but the digest alone grants no authority. The approval service must authenticate the approver, authorize approval of that transaction and protect the approval record. Canonicalization must be unambiguous: amount units, currency, optional fields and account identifiers cannot change interpretation between approval and execution.

Immediately before execution, recheck permission, revocation, expiry and relevant state versions. If the supplier’s bank details changed after approval, the action is no longer the one approved. This is a time-of-check/time-of-use problem. Use transactionally enforced preconditions or conditional writes where available, rather than a check followed by an unprotected gap. New material information may require a new proposal and approval.

09 · Execution: constrain the machine that touches the outside world

Even an authorized action can be implemented unsafely. A ledger lookup should call a known endpoint through a typed client, not interpolate model text into a shell command. A document-analysis tool should run with bounded CPU, memory, file access and lifetime. Keep model-provider secrets, application credentials and unrelated tenant data out of that execution environment. A container is one isolation mechanism, not a complete guarantee against escape or misconfiguration.

Network access needs its own policy. If a tool fetches a model-supplied URL, restrict schemes, destinations and redirects; prevent access to internal management services and metadata endpoints according to the deployment’s network model. DNS resolution and redirects can invalidate a superficial string check. OWASP’s SSRF prevention guidancehttps://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html provides the underlying web-security considerations. The safest business adapter often accepts a supplier ID and resolves the destination server-side rather than accepting an arbitrary URL.

Outbound data deserves the same attention as inbound text. The assistant may be allowed to read an invoice but not email it to any address appearing in that invoice. Constrain recipients and channels according to policy, minimize exported fields and require approval for consequential disclosure. A content classifier can help route review; it cannot substitute for an authoritative recipient and tenant policy.

10 · Retries, long-running jobs and multi-agent delegation

The payment API times out. That observation does not tell the worker whether the payment failed before execution or succeeded before the response was lost. Blindly retrying can duplicate a payment. Use a stable idempotency key for the same logical action, an authoritative operation record and provider-supported reconciliation. A new model attempt must not invent a new key simply because it generated different explanatory text.

Idempotency is a business invariant with a scope and lifetime, not a synonym for exactly-once delivery. The server must bind the key to the canonical action and reject reuse with changed parameters. Persist transitions such as proposed, approved, submitted, confirmed and uncertain. An uncertain outcome belongs in reconciliation or review, not in an automatically “successful” final answer. Compensating actions can have costs and may not reverse the original effect.

If a planner delegates invoice extraction to a sub-agent, give that worker only the necessary document access and an expiring task context. The child cannot obtain more authority merely by claiming urgency, and a returned instruction cannot become the parent’s authorization decision. Re-evaluate permissions when a workflow resumes or crosses into a new operation. Revocation should stop future authority issuance and sensitive actions according to a defined policy; cancelling a chat window does not necessarily cancel queued jobs.

11 · Memory and learning: an incident can persist beyond the prompt

The assistant may summarize the supplier interaction into long-term memory. If it stores “Northstar payments should use this new account” as an authoritative preference, a malicious document has crossed two boundaries: evidence became instruction, then temporary context became persistent policy. The next session can be compromised even after the original PDF is no longer visible.

Separate memory by purpose and trust level. User preferences, retrieved facts, inferred summaries, approval records and policy configuration should not share an undifferentiated text bucket. Preserve provenance, tenant, version, expiry and the identity authorized to create or revise each record. A model-produced summary is derived evidence; it should not automatically overwrite a verified supplier record or security rule.

Learning loops introduce a related risk. Production feedback may be useful for improving prompts or retrieval, but an attacker can shape that feedback. Review what enters training or optimization, maintain a holdout that the optimizer cannot inspect, and require an independent release gate before changing tools or privileges. A higher reward from the same compromised judge is not evidence that the system became safer or more capable.

12 · Agentic audit: reconstruct authority, action and observed outcome

When finance asks why a payment was attempted, a transcript alone is insufficient. Record the authenticated subject and workload, tenant, task and attempt IDs, policy version, relevant resource versions, proposed canonical action, authorization decision, approval reference, execution identifier and observed outcome. Keep the failed attempt and the later retry distinct. Minimize sensitive content and restrict who can inspect the evidence.

A useful action receipt joins three claims: what was requested, why execution was permitted, and what the downstream system reported. A cryptographic signature can authenticate the issuer and protect recorded bytes from undetected modification under its key-management assumptions. It does not prove that the issuer was honest, the record complete, the policy correct or the payment actually settled. Settlement evidence must come from the authoritative business system, and may arrive later.

Likewise, a hash chain can make some alteration detectable if an attacker cannot replace the trusted anchor and all copies. It does not automatically prevent deletion, omission or rewriting by an administrator who controls the whole logging path. Define independent retention, integrity checks, access controls and failure handling. OWASP’s Logging Cheat Sheethttps://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html is a useful baseline for event selection, sensitive-data handling and protecting the logging system itself.

Routine telemetry loss may be handled with bounded buffering and visible dropped-event counters. Mandatory audit evidence for a sensitive action may require a fail-closed or durable-commit design. Decide this per operation; do not accidentally apply an availability-oriented logging policy to payment authorization. An LLM can summarize the resulting evidence, but its fluent explanation must retain links to records and mark uncertainty rather than manufacture a causal story.

Choose audit durability from the action consequence. A diagnostic token event can tolerate documented sampling, while the authorization and approval evidence for a payment cannot be reconstructed from sampled text. One design writes the intended action and audit outbox record in the same local database transaction, then a dispatcher submits it using a stable downstream idempotency key. This makes local intent durable; it does not make a remote payment atomic with that transaction.

After a lost response, join the operation ID to the remote provider status and append the observed result. Preserve “submitted but outcome unknown” as a real state. If the provider has neither idempotency nor reliable status lookup, automatic retries cannot promise duplicate prevention; a human reconciliation path is part of the architecture. This separates durable intent, permitted execution and confirmed settlement rather than hiding them in a single success boolean.

13 · Detection, containment and recovery complete the lifecycle

Prevention is never the whole story. Suppose the assistant repeatedly proposes unauthorized supplier changes after reading a particular document. Alert on the policy-denial pattern, affected integration and source provenance, not merely on suspicious words. Repeated blocked attempts can reveal an attack even though the authorization boundary held. Conversely, a permitted but unusual sequence may require investigation when credentials were legitimately issued but misused.

Contain the relevant capability: suspend the compromised connector, revoke delegated grants, pause queued sensitive actions and isolate the affected worker or memory namespace. Preserve evidence before destructive cleanup. Reconcile uncertain transactions against the ledger, identify which users and tenants were actually exposed, and separate confirmed effects from possible exposure. A dramatic model transcript is not proof that data left the system.

Recovery includes rebuilding trusted state. Remove poisoned derivatives as well as the original document, rotate exposed credentials, review tool and policy changes, and restore from a known configuration. Resume through a constrained canary with explicit acceptance criteria. If the vulnerable boundary remains, merely adding the attack wording to a denylist prepares the system for yesterday’s payload.

For every incident, produce a timeline of evidence, authority decisions and side effects, then convert the failure into a regression test. Distinguish the initiating content, the control that failed and the business consequence. This prevents a vague conclusion such as “the model was jailbroken” from hiding an overprivileged service account, missing object check or unsafe retry policy. The best postmortem explains both why the attack worked and why the proposed fix blocks that class without breaking the legitimate task.

How the four disciplines fit together

Security determines which actions are permitted. Infrastructure executes work within capacity and failure constraints. Observability preserves evidence of what happened. Evaluation tests whether a change improves the outcome under a declared contract. None substitutes for the others.

PART II / REFERENCE CHAPTERS

Mechanisms, research and market context — English reference material

The following chapters provide the mathematical examples, experiments, research papers and dated company evidence behind the operational story. Return to the lifecycle when deciding where each mechanism belongs.

1. Define the security problem before choosing a product

An enterprise assistant reads a document, drafts a recommendation and calls a tool. Three different systems are involved: a model that predicts useful output, a data system that decides what can be read, and an execution system that decides what may happen. Security fails when one system silently grants authority belonging to another—for example, when text inside an external document is treated as permission to send a confidential file.

Security for AI protects model artifacts, training and retrieval data, applications and agent actions. AI for security uses models to help analysts investigate or respond to attacks. Alignment concerns whether model behavior follows intended goals and constraints; it overlaps with security but is not a replacement for access control. A model can produce a well-intentioned answer while an application exposes the wrong tenant's data.

A threat model names the assets, adversary capabilities, trust boundaries and unacceptable outcomes. Prompt injection occurs when attacker-influenced content redirects the model away from the authorized task. An indirect injection arrives through a document, website or tool response. A confused deputy has legitimate authority but is manipulated into using it for someone else's purpose. These definitions are more useful than treating every unwanted answer as the same kind of jailbreak.

Before this lesson: understand authentication versus authorization. Identifying an agent does not permit every action it proposes. Understand precision versus recall: detecting most attacks can still create too many false alarms. Finally, distinguish a content classifier's probabilistic judgment from a deterministic permission check. The learning goal is to combine them without allowing the model to rewrite its own boundary.

This survey compares disclosed product roles and transactions, not independently measured vendor efficacy. A procurement claim needs tests on the buyer's workflows. A public benchmark needs its own attacker budget, task set, software version and scoring procedure. Neither can be replaced by an acquisition price.

2. A trust-boundary architecture for an agent

An untrusted document and model proposal sit outside a protected execution boundary; identity, scoped policy and approvals guard tools, while audit receipts record decisions.
Original architecture diagram. This is a proposed enforcement design, not a depiction of any company’s internal system.

Consider an invoice assistant authorized to prepare reminders for one business unit. The model reads invoice content and proposes an action containing an invoice ID, recipient ID and message. The execution service resolves those identifiers against authoritative records, checks that the current principal can act on the invoice, validates the recipient and enforces any required approval. Only the execution service holds the sending credential.

An instruction embedded in the invoice cannot broaden the principal's scope. The model also cannot declare that its own proposal has already been approved. Approval should bind the exact action, resource and relevant parameters, expire when appropriate, and be invalidated when the proposal changes. Retries need idempotency keys so a network timeout does not create duplicate sends. This turns vague “human oversight” into a specific state transition that can be tested.

Logs should record the model and tool versions, proposal, effective identity, policy decision and execution receipt. Protect those logs: copying all prompts and documents into an observability service can create another disclosure path. Minimize retained content and separate access to operational metadata from access to sensitive payloads.

The boundary is useful even when a detector misses malicious text. A model may still produce a bad proposal, but it cannot execute an unauthorized action through the controlled service. This is not a complete guarantee: an overly broad policy, compromised credentials, an unmediated tool or a bug in the executor can bypass the intended design. Test the enforcement path itself.

3. Follow the attack surface across the lifecycle

Model and dependency supply chain

Failure mechanism
A model artifact, serializer or dependency can execute code or alter behavior before inference starts.
Control and residual question
Inventory and pin artifacts; verify provenance; isolate loading; scan dependencies. A signature authenticates a publisher, not the semantic safety of the model.

Retrieval and memory

Failure mechanism
Poisoned content can be retrieved repeatedly or stored as a persistent instruction.
Control and residual question
Preserve provenance, enforce tenant filtering before retrieval, bound memory writes and expire untrusted state. How should legitimate corrections differ from poisoned memory?

Multimodal input

Failure mechanism
Text, images, audio or layout can carry instructions that survive extraction differently.
Control and residual question
Test the whole preprocessing pipeline and modality conversions. A text-only filter does not establish coverage of the original input.

Tool use and delegation

Failure mechanism
The agent can misuse a legitimate tool or transfer excessive scope to another agent.
Control and residual question
Use explicit identities, least privilege, scoped capabilities, approval gates and revocation. Natural-language intent remains harder to specify than an API permission.

Outputs and data movement

Failure mechanism
A response, URL, attachment or log can disclose data to an unauthorized destination.
Control and residual question
Check recipients and destinations, apply data policies and minimize secrets in context. A detector’s refusal is not proof that all side channels are closed.

Evaluation and updates

Failure mechanism
Repeatedly tuning on known attacks creates benchmark overfitting; an update can invalidate prior defenses.
Control and residual question
Separate development and held-out attack families, record versions, test utility regressions and rehearse rollback. An unchanged benchmark score may conceal changed deployment risk.

The difficult cases are often individually plausible steps. Reading a page, summarizing it and saving a note may each be allowed, yet their composition can move confidential information across a boundary. Conversely, blocking every unusual document destroys useful work. A good threat model describes prohibited outcomes, rather than relying only on a list of suspicious phrases.

For an RSI or self-improvement loop, treat new prompts, tools, policies and training data as untrusted candidate changes until independently evaluated. The evaluator must not share writable state with the candidate. A candidate that edits the benchmark, expands its tools or selectively suppresses failures has changed the experiment rather than demonstrated improvement. See the RSI tutorial for promotion gates and provenance.

Industry landscape: control points, buyers and competition

Market evidence checked November 8, 2026. Product documentation establishes an offering, not independently measured efficacy. Competitive theses and unmet opportunities are Token Lab analysis; examples labeled synthetic are not company results.

Who controls the AI security boundary? Companies, consolidation and defensible gaps

An analyst’s map of eleven vendors and platforms, checked November 8, 2026. Product documentation establishes advertised capabilities and deployment boundaries, not comparative effectiveness. Commercial conclusions below are editorial analysis; the worked economics are hypothetical, not vendor results.

Who controls the AI security boundary? Companies, consolidation and defensible gaps
Conceptual map of responsibilities; not a vendor score or measured coverage percentage.

The market is a competition for control points

AI security is not one interchangeable product category. A model scanner owns a release checkpoint; a gateway owns request transit; an identity provider owns credentials; a runtime sensor observes execution; a SOC platform owns investigation. Their evidence and authority differ. The strategic question is which product can connect a real principal, an allowed action, a particular resource and a confirmed effect. A detector that recognizes suspicious language may be valuable without being an authorization system. Our thesis is that durable value comes from hard-to-bypass placement, useful context and operational ownership—not the number of attack labels in a dashboard.

Reference architecture: an authorized agent execution path

Read the solid arrows as the proposed execution path. The authenticated request provides tenant and principal context; retrieved text only informs the model. The model emits a candidate action, while a trusted service resolves the resource and supplies the approval record. The enforcement gate obtains a policy decision before the scoped executor can reach the business service. The dashed audit connections carry evidence, not permission. The red rejection branch does not reach execution.

Keep credentials out of the model context. Bind approval to action parameters and resource version, and recheck revocation before sensitive execution. Audit records connect principal, policy version, approval and operation ID to the downstream outcome. A durable local outbox can preserve intent, but it cannot atomically commit an unrelated remote effect: timeouts require idempotency and authoritative reconciliation. This is an original reference design, not any vendor’s internal architecture.

Reference architecture: an authorized agent execution path
Conceptual map of responsibilities; not a vendor score or measured coverage percentage.

Company and product positions

A selective map, not market share or a vendor certification. Coverage depends on edition, integration, deployment and enforcement mode.

Palo Alto Networks / Prisma AIRS

Ownership / organization
Palo Alto Networks
Control point
Model supply chain · gateway · runtime

Prisma AIRS covers model/artifact inspection, red teaming and runtime controls; acquired Portkey adds gateway placement. Protect AI is already part of Palo Alto, while CyberArk and Koi extend the surrounding identity/endpoint portfolio. The buyer is an enterprise security organization seeking consolidation. The potential moat is distribution plus correlated context at multiple enforcement points. The key risk is integration: shared ownership does not prove shared policy semantics or complete interception. Ask how an authoritative tenant, user and approval reach each control, and how direct tool calls are prevented from bypassing it.

Prisma AIRS documentation ↗https://docs.paloaltonetworks.com/prisma-airs/getting-started/prisma-airs-overview · FY2026 acquisition disclosures ↗https://www.sec.gov/Archives/edgar/data/1327567/000132756726000023/panw-20260731.htm

Cisco / AI Defense

Ownership / organization
Cisco
Control point
Network · validation · agent identity

AI Defense describes model validation, network-embedded runtime protection, AI inventory, access and supply-chain controls. Robust Intelligence is acquired; Astrix adds non-human identity context and Galileo adds AI observability capabilities. The commercial hypothesis is an enterprise path from network visibility to identity and investigation through existing Cisco/Splunk relationships. Coverage is topology-dependent: network inspection does not automatically govern local or out-of-band execution. Evaluate which integrations are operational, where decryption happens, whether every privileged action uses the recorded identity, and whether application-specific resource authorization still resides elsewhere.

AI Defense capabilities ↗https://www.cisco.com/site/us/en/products/security/ai-defense/index.html · Astrix acquisition completion update ↗https://blogs.cisco.com/news/cisco-announces-intent-to-acquire-astrix-security · Galileo acquisition completion ↗https://www.cisco.com/site/us/en/about/corporate-development/acquisitions/galileo/index.html

CrowdStrike / Pangea

Ownership / organization
CrowdStrike
Control point
Prompt telemetry · endpoint · SOC

Pangea AI Guard can report, block or modify detected content; AIDR connects AI activity with the security workflow. CrowdStrike also documents Kubernetes prompt-layer collection and introduced endpoint-focused Falcon Guardian in September 2026. Its natural buyer is a SOC already using Falcon. The strategic value is linking suspicious AI interaction to a process, file access or outbound connection. Distinguish a collector from pre-effect enforcement, and an announced product from verified deployment availability. Ask which remote tools can be blocked and how application approval is enforced beyond endpoint visibility.

AI Guard API behavior ↗https://pangea.cloud/docs/ai-guard/overview · AI detection and response ↗https://www.crowdstrike.com/en-us/solutions/ai-detection-and-response/ · Falcon Guardian announcement ↗https://www.crowdstrike.com/en-us/press-releases/crowdstrike-unveils-falcon-guardian-ai-agent-security/

Check Point / Lakera

Ownership / organization
Check Point
Control point
Semantic guardrails · agent posture

Check Point AI Agent Security documentation explicitly labels the product early access. Discovery and configuration risk assessment are posture functions; runtime protection uses the Guard API. Native platform runtime integrations remain on the roadmap. Developers can buy the runtime layer separately, creating an application-level entry point, while Check Point supplies enterprise distribution. This is not proof of universal native enforcement. The documentation itself places identity-provider, cloud and network controls elsewhere in the architecture. Procurement should verify each connector and API integration rather than interpreting a full-lifecycle diagram as automatic coverage.

Agent Security scope and early-access status ↗https://docs.lakera.ai/docs/agent-security

SentinelOne / Prompt Security

Ownership / organization
SentinelOne
Control point
Employee AI · coding assistants · MCP

Prompt Security extends SentinelOne into employee AI use, sensitive-data sharing and agent interactions, with policy controls at interaction points. The buying trigger can arrive before a company builds its own agent: employees already use chatbots and coding assistants. Existing endpoint relationships may lower distribution friction. But managed-browser coverage and backend agent enforcement are different deployments. Ask about unmanaged devices, local models, supported assistants, credential-bearing tools and bypass paths. A large list of known MCP servers does not prove every invocation has correct business authorization or covers every version of a tool.

Prompt Security acquisition rationale and capabilities ↗https://www.sentinelone.com/press/sentinelone-to-acquire-prompt-security-to-advance-genai-security/ · Current platform ↗https://www.sentinelone.com/platform/securing-ai-prompt/

HiddenLayer

Ownership / organization
HiddenLayer
Control point
AI supply chain · attack simulation · runtime

HiddenLayer separates supply-chain scanning, attack simulation and production runtime protection, and announced coding-agent harness security using native hook surfaces in August 2026. Its specialist proposition suits organizations with sensitive models and heterogeneous deployment environments. The moat hypothesis is AI-specific adversarial research combined with release-workflow integration. Validate that hypothesis through supported model formats, attack classes, false positives and bypass conditions. Agentless or no-training-data claims must be interpreted per module, not as universal absence of instrumentation. Scanning an artifact does not prove its training data uncontaminated or every later action authorized.

Platform modules ↗https://www.hiddenlayer.com/platform · Agent harness announcement ↗https://www.hiddenlayer.com/newsroom

Noma Security

Ownership / organization
Noma Security
Control point
AI posture · access · session behavior

Noma combines AI-SPM, access control, runtime detection/response and red teaming. Its documentation describes session, identity, data and baseline context, with monitor, alert, block or mask actions through existing gateways and hooks. The differentiation hypothesis is identifying dangerous sequences whose individual calls look ordinary. CISO and platform teams must jointly deploy it. Verify identity continuity across subagents, lost events, concurrent sessions and delayed effects. Natural-language guardrails are not automatically verified authorization policies. A behavioral score is useful evidence, but authoritative tenant, resource and budget state still needs a trusted source.

Platform control layers ↗https://noma.security/platform · Runtime enforcement and context ↗https://noma.security/platform/runtime-protection/

Upwind

Ownership / organization
Upwind
Control point
Cloud workloads · runtime context · AI sensors

Upwind links cloud posture, workload telemetry and AI activity, with context scanning for agent instructions and skills. Its cloud-security entry point is the infrastructure where an agent’s action becomes real: processes, credentials, data stores and egress. This can fit cloud-native security and platform buyers who need prioritized remediation. Separate AI used to investigate threats from controls securing the customer’s AI. Sensor coverage and agentless fallbacks differ. Ask what can be blocked before execution, how semantic intent reaches runtime policy, and which remediation actions require human approval. Runtime visibility alone does not establish business authorization.

Cloud and AI runtime architecture ↗https://www.upwind.io/feed/upwind-ai-security-securing-your-ai-stack-from-the-inside-out · Agent context scanning ↗https://www.upwind.io/feed/announcing-the-upwind-ai-context-scanner-securing-ai-agents-instructions-skills-and-model-context

Microsoft / Entra, Defender, Purview

Ownership / organization
Microsoft
Control point
Identity · data governance · enterprise agents

Entra Agent ID supplies agent identity capabilities; Agent 365 security connects Entra, Defender and Purview, while Purview addresses data protection and compliance. Microsoft starts with established employees, documents and administrator workflows, making distribution and existing context its major strategic assets. This can pressure standalone inventory or basic policy products. Yet registering an identity does not prove each tool uses it. Verify licensing, supported platforms, delegated versus application permissions, revocation, cross-tenant access and non-Microsoft actions. The buyer’s existing Microsoft estate is an integration advantage, not proof that one switch secures every autonomous workflow.

Agent identities ↗https://learn.microsoft.com/en-us/entra/agent-id/what-are-agent-identities · Agent 365 security architecture ↗https://learn.microsoft.com/en-us/security/security-for-ai/agent-365-security · Agent data governance ↗https://learn.microsoft.com/en-us/purview/ai-agents

Google Cloud / Model Armor, Wiz

Ownership / organization
Google
Control point
Content path · cloud risk context

Model Armor screens prompts, responses and documents for injection and sensitive-data risks. Its documentation distinguishes inspection-only behavior from enforcement integrations, including Google Cloud MCP paths. Wiz, acquired in March 2026, adds cloud-security context; it is not itself a universal agent authorization service. Google can attach these controls to managed AI and infrastructure procurement. Verify regional and language support: documentation describes skipped detection for certain non-English checks in some regions. A logged finding is not a block, and resource IAM, tenant isolation and exact business approval remain separate responsibilities.

Model Armor deployment modes ↗https://docs.cloud.google.com/model-armor/overview · Language and regional detection boundaries ↗https://docs.cloud.google.com/model-armor/sanitize-prompts-responses · Wiz acquisition completed ↗https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/wiz-acquisition/

AWS / Bedrock AgentCore

Ownership / organization
AWS
Control point
Tool gateway · deterministic authorization

AgentCore Policy uses Cedar for fine-grained tool/resource authorization, including identity and tool-input conditions. The strategic position is close to execution, complementing rather than replacing probabilistic content detection. It fits builders already deploying on AWS or consistently routing tools through AgentCore Gateway. Its economic advantage can be reusing cloud identities and operational tooling. But a policy engine is only as authoritative as its inputs. It cannot repair model-supplied identities, bypassed gateways or missing tenant checks. Stateful budgets, idempotency, approval binding and uncertain external effects still require application design.

Policy control boundary ↗https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/policy.html · Cedar design rationale ↗https://aws.amazon.com/blogs/security/why-policy-in-amazon-bedrock-agentcore-chose-cedar-for-securing-agentic-workflows/

Follow one attack across the invoice workflow

A supplier PDF asks an invoice agent to replace the beneficiary account. Retrieval ACLs decide whether the employee may read the document, not whether its instructions are trustworthy. A content guard may flag injection. Identity and gateway controls constrain the delegated tools. The payment service must separately bind approval to tenant, beneficiary, amount, currency and invoice version, then prevent duplicate effects. Endpoint/cloud sensors can detect unexpected processes or egress; audit correlates decisions with confirmed outcomes. If the prompt detector misses, later boundaries should still contain damage. If it catches the prompt but the backend accepts a forged approval, the architecture remains insecure.

Technical detail: separate policy decisions from enforcement

A policy decision point (PDP) evaluates trusted principal, tenant, action, resource and current state. A policy enforcement point (PEP) intercepts the operation and applies that decision. Put the PEP on every path to the protected effect, including retries and background jobs. A prompt classifier supplies a risk signal; it does not mint identity or grant permission. This separation explains why an identity vendor, a content guard and a gateway can complement one another without being substitutes.

An allow decision can become stale before execution. Bind approval to the exact action and resource version; use conditional writes or transactions where available. If an authorization dependency is unavailable, sensitive actions should follow an explicit deny-or-defer policy, not silently fall back to a model judgment. A gateway cannot enforce atomicity inside an unrelated remote service. The commercial question is which team owns these integration gaps, not simply which dashboard displays the alert.

Technical detail: separate policy decisions from enforcement
Conceptual map of responsibilities; not a vendor score or measured coverage percentage.

A hypothetical buying decision: count disruption as well as detection

Suppose a system processes one million tasks monthly, of which 100 are malicious. At 90% detection and a 0.1% benign false-positive rate, it flags 90 malicious and about 1,000 benign tasks: only about 8.3% of alerts are true positives. At five minutes of review each, that is roughly 91 analyst-hours before incident work. These are invented teaching inputs, not a product benchmark. A vendor may still be worth buying, but the decision needs expected loss, blocked legitimate work, coverage, latency, review cost and containment capability. Comparing detection percentages without base rates or operating thresholds hides the economics.

Where defensible opportunities remain

The hard remaining problems are complete interception across local and remote agents, identity continuity through delegation, revocation during long tasks, atomic budgets, idempotency and reconciliation of uncertain effects. Multi-turn intent, memory poisoning and tool-result provenance also exceed isolated prompt classification. A credible entrant should choose a narrow high-cost failure, identify the enforcing owner and show a measurable reduction under realistic benign traffic. Buyers should demand failure-mode evidence: lost telemetry, unsupported languages, bypassed hooks, policy-service outages and recovery. The winning unit is a reliably governed workflow, not a scan count. That is an analytical hypothesis to test, not a forecast that any named company will win.

Acquisitions, capital and strategic consequences

Consolidation: distinguish an agreement from a completed deal

Protect AI joined Palo Alto Networks on July 22, 2025; Pangea joined CrowdStrike on September 26, 2025; Prompt Security joined SentinelOne on September 5, 2025; Lakera joined Check Point on October 22, 2025. These are completed transactions, not independent startup competitors. In 2026, Palo Alto closed Portkey on May 29 after announcing intent April 30. Cisco completed Galileo on May 22 and Astrix on June 29. Google completed Wiz on March 11. These dates establish ownership, not complete product integration. The acquisition thesis is that distribution, identity, telemetry and enforcement can reinforce one another; integration quality must still be demonstrated.

Capital is an expansion signal, not proof of a moat

Company announcements report HiddenLayer’s $100M Series B on September 2, 2026, Noma’s $100M Series B on July 31, 2025, and Upwind’s $400M Series C on October 28, 2026. These figures measure financing, not product revenue or market share. Specialists need a reason customers will pay beyond bundled cloud or security-suite controls: difficult integrations, strong adversarial research, reliable cross-environment enforcement, or workflow ownership. Large platforms benefit from existing procurement and installed telemetry, but broad portfolios can still leave gaps between identity, application state and remote effects. Neither a funding total nor a completed acquisition settles that execution question.

Primary product and transaction sources

Source and maintenance notes

Primary links are placed beside the claims they support. Acquisition amounts retain their original basis; undisclosed values remain undisclosed. Product specifications and release statements are vendor disclosures, not comparative evidence from Token Lab. The diagrams and reference calculations are original educational material. Corrections should identify the section, source and proposed replacement; a source-access date is not a paper publication date.

All frontier research · Top Company Frontier · Editorial standards

Practice and verification

14 · Prove the controls with an attack-and-utility experiment

Build a local test environment with synthetic invoices, two tenants, mock tools and a fake payment ledger. Keep the business workflow unchanged while varying one boundary: retrieval filtering, scoped credentials, approval binding, egress policy or audit durability. For each case, define the attacker-controlled input, the desired forbidden effect, the expected enforcement point and the permitted outcome that should still succeed.

Measure actual effects. Attack success is an unauthorized read, disclosure, state change or other specified consequence—not simply an alarming sentence. Report task utility, false denials, latency and cost alongside attack prevention. State the denominator: attempted attacks, independent tasks, generated payloads or complete sessions are different populations. Zero failures in a small curated set does not establish zero production risk; correlated variants provide less evidence than independent scenarios.

Use negative controls: legitimate invoices with quoted instructions, normal retries, authorized account changes and benign external links. Add adaptive tests in which the attacker can observe limited feedback, but keep a final set outside the development loop. Match budgets when comparing defenses. A method that runs ten expensive judges should not be credited with purely algorithmic superiority over one cheap check without disclosing the resource difference.

Three research questions remain especially important. Can authorization policies remain understandable as agents compose tools dynamically? Can provenance survive retrieval, summarization and delegation without excessive data retention? Can automated audit distinguish plausible causal narratives from interventions that truly prevent failures? Compare each proposal with a simple deterministic baseline and publish failure cases. The goal is a defensible security argument across the entire lifecycle, not a promise that one model or product makes every stage safe.

15 · Practice the architecture, not just the terminology

These are original practice questions, not reported interview questions from any company. Before opening an answer, draw the relevant trust boundary and name the component that enforces it.

A model refuses the injected PDF. Is the payment workflow secure?

Hint: Ask which component can actually execute a payment.

Reveal answer and analysis

No. The refusal is one observed behavior. Inspect deterministic permission enforcement, approval binding and tool credentials; then force an adversarial proposal through the same gate and verify no forbidden ledger effect. A strong answer also tests legitimate utility and avoids treating one refusal as a general guarantee.

Why are a model API key and a user’s delegated token different?

Hint: Separate inference billing from authority over business resources.

Reveal answer and analysis

The API key lets the application invoke an inference service. Delegated authority identifies a permitted actor, recipient, scope and lifetime for business operations. Neither should be placed in model context. Explain audience checks, object-level authorization and what happens when delegation expires.

An approver accepted $100 to supplier A; the account changes before execution. What now?

Hint: Compare the approved action and current resource version.

Reveal answer and analysis

Reject or require renewed approval according to the policy. Bind approval to canonical action fields and relevant versions, then enforce the precondition atomically where possible. A pre-execution check followed by an unprotected write can still race.

The tool timed out after a payment request. Can the agent retry?

Hint: A timeout is an observation, not proof of non-execution.

Reveal answer and analysis

Reconcile using a stable operation/idempotency key and authoritative status. A retry with identical bound parameters may be safe if the server guarantees the required semantics; otherwise mark the result uncertain and escalate. Do not generate a new payment key for the same logical intent.

A signed receipt says “payment completed.” What does it prove?

Hint: Separate integrity, authority and business truth.

Reveal answer and analysis

It can establish that a holder of the signing key issued those bytes, subject to key trust. It does not prove completeness, correct authorization or settlement. Join the receipt to policy and approval records and independent ledger evidence, and explain how missing or contradictory evidence is handled.

Scoring: 1/5 names a risk; 2/5 defines the boundary; 3/5 specifies the enforcing identity, action and resource check; 4/5 designs a positive and negative test that inspects real effects; 5/5 handles concurrency, revocation, evidence gaps and residual risk. The capstone is a local two-tenant invoice simulator with one exact-action approval, an injected document, a timed-out execution and a reconstructable audit trail. Demonstrate prevention and legitimate task completion, then document which production guarantees your simulator does not provide.

Practice: verify each trust boundary

Run these synthetic cases against the same legitimate invoice workflow. Inspect backend state and audit evidence, not only assistant text.

Identity: two tenants with identical invoice IDs, removed membership and tampered tenant fields must not cross scope.

Model and supply chain: compare clean and suspicious revisions under matched tasks; reject malformed, oversized and unregistered tool outputs; handle provider timeouts.

Retrieval: an injected document must not grant authority, while normal quoted instructions and unusual formatting remain usable.

Delegation: reject wrong audience, missing scope, expired grants and sub-agents requesting broader privilege.

Approval: change one cent, currency, destination or resource revision; expire or revoke approval. No unauthorized effect may commit.

Execution: reject disallowed destinations and paths, bound resource exhaustion, and retain legitimate attachment processing.

Retries: duplicate delivery, response loss and worker restart must reconcile to the business ledger without unintended duplicate effects.

Memory: check cross-tenant leakage, injected instructions surviving summaries and unsafe derivatives surviving source deletion.

Audit: missing events, inconsistent IDs, forged outcomes and audit-store outage must produce the declared failure behavior.

Report both blocked attacks and preserved task utility. Record remaining identity-provider trust, concurrent-system boundaries and cases requiring human reconciliation; do not promise global atomicity.

Run the authority boundary, not just a prompt

The model can propose an Action. The application supplies the trusted Grant, approval record, clock and stable operation key. This simulator checks their relationship before a ledger write. It also rejects a second payment under a different retry key. All records are synthetic and in memory; there is no network, credential or payment service.

Python 3.10+, standard library, deterministic: download agent-authority.py.

python3 agent-authority.py
"""Offline policy-and-receipt simulator, Python 3.10+, standard library only.
Not production IAM, a payment API, or a cryptographic audit implementation.
The caller is trusted test code. Untrusted model output supplies Action only.
"""
from dataclasses import dataclass, replace
from hashlib import sha256
import json

@dataclass(frozen=True)
class Action:
    tenant: str
    invoice: str
    revision: int
    payee: str
    cents: int
    operation: str = 'pay'

@dataclass(frozen=True)
class Grant:
    user: str
    tenant: str
    audience: str
    invoice: str
    max_cents: int
    expires: int
    revision: int


def digest(action):
    return sha256(json.dumps(action.__dict__, sort_keys=True,
                             separators=(',', ':')).encode()).hexdigest()

class Ledger:
    def __init__(self):
        self.commits = {}  # durable and transactional in a real implementation
        self.events = []
        self.revoked = set()
        self.audit_available = True
        self.invoice_revision = 4

    def execute(self, action, grant, approval, key, now):
        """approval is a trusted approval-service record, not model text.
        Single-threaded transaction simulation; no exactly-once network claim.
        """
        reason = None
        if grant.user in self.revoked: reason = 'revoked'
        elif now >= grant.expires: reason = 'expired'
        elif grant.audience != 'ledger': reason = 'audience'
        elif action.tenant != grant.tenant: reason = 'tenant'
        elif action.invoice != grant.invoice: reason = 'resource'
        elif action.operation != 'pay': reason = 'operation'
        elif type(action.cents) is not int or not 0 < action.cents <= grant.max_cents: reason = 'amount'
        elif action.revision != self.invoice_revision or grant.revision != self.invoice_revision: reason = 'stale_revision'
        elif approval != digest(action): reason = 'approval_binding'
        elif not self.audit_available: reason = 'audit_unavailable'
        elif key in self.commits:
            if self.commits[key]['digest'] != digest(action): reason = 'idempotency_conflict'
            else: return self.commits[key]  # same receipt, no second write
        if reason is None and any(r['invoice'] == action.invoice and r['tenant'] == action.tenant for r in self.commits.values()):
            reason = 'already_settled'
        if reason:
            self.events.append({'state': 'denied', 'reason': reason})
            return {'state': 'denied', 'reason': reason}
        receipt = {'state': 'committed', 'digest': digest(action), 'key': key,
                   'actor': grant.user, 'tenant': grant.tenant,
                   'invoice': action.invoice, 'revision': action.revision, 'policy': 'teaching-v1'}
        # In production: commit effect and durable audit/outbox atomically,
        # or use downstream idempotency + reconciliation; this is in-memory.
        self.commits[key] = receipt
        self.events.append(dict(receipt))
        return receipt


action = Action('tenant-a', 'inv-7', 4, 'supplier-42', 5000)
grant = Grant('user-1', 'tenant-a', 'ledger', 'inv-7', 10000, 100, 4)
approval = digest(action)
ledger = Ledger()
assert ledger.execute(action, grant, approval, 'op-1', 50)['state'] == 'committed'
assert ledger.execute(action, grant, approval, 'op-1', 51)['state'] == 'committed'
assert len(ledger.commits) == 1
assert ledger.execute(action, grant, approval, 'different-key', 51)['reason'] == 'already_settled'
assert ledger.execute(replace(action, tenant='tenant-b'), grant, approval, 'op-2', 50)['reason'] == 'tenant'
assert ledger.execute(replace(action, payee='attacker'), grant, approval, 'op-2', 50)['reason'] == 'approval_binding'
assert ledger.execute(replace(action, cents=10001), grant, approval, 'op-2', 50)['reason'] == 'amount'
assert ledger.execute(action, replace(grant, audience='search'), approval, 'op-2', 50)['reason'] == 'audience'
assert ledger.execute(action, grant, approval, 'op-2', 100)['reason'] == 'expired'
ledger.invoice_revision = 5
assert ledger.execute(action, grant, approval, 'op-2', 50)['reason'] == 'stale_revision'
ledger.invoice_revision = 4
ledger.audit_available = False
assert ledger.execute(action, grant, approval, 'op-2', 50)['reason'] == 'audit_unavailable'
ledger.audit_available = True
ledger.revoked.add('user-1')
assert ledger.execute(action, grant, approval, 'op-2', 50)['reason'] == 'revoked'
assert len(ledger.commits) == 1
print('13 checks passed: approval, tenant, audience, budget, revision, expiry, audit, revocation and replay.')
print('One committed effect. Model-proposed changes cannot expand the trusted grant.')
13 checks passed: approval, tenant, audience, budget, revision, expiry, audit, revocation and replay.
One committed effect. Model-proposed changes cannot expand the trusted grant.

Production gaps: a digest is not a signature. Real approval records need authenticated issuance, approver authority, expiry and replay controls. Validate tokens cryptographically; use a transactional database, concurrency controls, authoritative invoice state, downstream idempotency and reconciliation. This single-threaded example does not prove exactly-once delivery or tamper-proof audit storage.

6. Data that actually tells us something

AgentDojo provides a concrete research substrate: 97 tasks and 629 security test cases involving tool-using agents. The original arXiv submission was June 19, 2024; this survey pins v3, November 24, 2024, whose revision notes include an implementation bug fix and travel-suite changes. Paper abstracthttps://arxiv.org/abs/2406.13352v3 · PDFhttps://arxiv.org/pdf/2406.13352v3 · Author codehttps://github.com/ethz-spylab/agentdojo. Those counts describe a benchmark, not the prevalence of attacks in customer deployments.

Three distinct quantities are needed. Clean utility measures completion without attack. Attack success measures a defined attacker objective under a specified opportunity and budget. Defended utility measures legitimate completion with the defense present. Report all three. An agent that cannot complete a banking task in the clean condition may also fail to execute the injected action; that is not necessarily a successful defense.

Denominators must be explicit. Was success counted per attack attempt, per task, or after the adversary tried many candidates and selected the best? Did the attacker see the defense or query its feedback? Was tool execution inspected, or only the final natural-language answer? A refusal message can accompany an earlier unauthorized tool call. A screenshot of a safe-looking answer is weak evidence about the full trajectory.

Synthetic population of 100000 requests: 100 malicious, 95 detected, 5 missed; among 99900 benign requests, 999 false alarms occur at a 1 percent false-positive rate.
Original arithmetic illustration, not measured vendor performance or an estimate of real attack prevalence. The numerical assumptions are deliberately stated so the result can be reproduced.

Assume 100,000 requests, an attack prevalence of 0.1%, a true-positive rate of 95%, and a false-positive rate of 1%. Expected counts are 95 true alerts, 999 false alerts and five missed attacks. Only 8.68% of alerts correspond to malicious requests. A seemingly small false-positive rate can overwhelm operations when attacks are rare.

Here π is attack prevalence, r is true-positive rate and f is false-positive rate. These are probabilities, not percentages entered as whole numbers. The formula follows by dividing expected true positives by all expected positive predictions. It does not imply that deployment prevalence is known or that adaptive attacks remain distributed like test samples.

Local false-alarm experiment

95 true alerts; 999.0 false alerts; alert precision 8.68%.

Population, prevalence and recall stay fixed at the stated teaching assumptions. No inputs leave your browser.

Download security-base-rate.py. Python 3.10+, standard library only; deterministic arithmetic, no random seed, network requests or model API. Run python3 security-base-rate.py.

"""Expected counts in a synthetic population, not vendor performance."""
from math import isclose


def audit(n, prevalence, tpr, fpr):
    if not isinstance(n, int) or isinstance(n, bool) or n <= 0:
        raise ValueError('n must be a positive integer')
    if not all(0 <= x <= 1 for x in (prevalence, tpr, fpr)):
        raise ValueError('Rates must be in [0, 1]')
    tp = n * prevalence * tpr
    fn = n * prevalence * (1 - tpr)
    fp = n * (1 - prevalence) * fpr
    tn = n * (1 - prevalence) * (1 - fpr)
    precision = tp / (tp + fp) if tp + fp else None
    return tp, fp, fn, tn, precision


if __name__ == '__main__':
    tp, fp, fn, tn, precision = audit(100000, .001, .95, .01)
    assert isclose(tp, 95)
    assert isclose(fp, 999)
    assert isclose(fn, 5)
    assert isclose(tp + fp + fn + tn, 100000)
    assert isclose(precision, 95/1094)
    assert audit(100, 0, 1, 0)[4] is None
    assert audit(100, 1, 1, 0)[4] == 1
    try:
        audit(100, -.1, .95, .01)
    except ValueError:
        pass
    else:
        raise AssertionError('Invalid rate accepted')
    print(f'TP={tp:.0f}, FP={fp:.0f}, FN={fn:.0f}, TN={tn:.0f}')
    print(f'Alert precision={100*precision:.2f}%')
    print('8 checks passed')

Expected output:

TP=95, FP=999, FN=5, TN=98901
Alert precision=8.68%
8 checks passed

Before reducing f to 0.1%, predict the outcome: false alerts become 99.9 and precision approximately 48.74% if recall is unchanged. In a real detector, changing the threshold generally changes recall too. The calculator isolates the arithmetic; it is not a threshold-tuning algorithm. A missed low-impact test attack and an unauthorized high-value transaction also have different costs.

7. Build a fair security evaluation

Start with a workflow inventory and concrete forbidden outcomes: sending data to a non-owner, accessing another tenant's record, running an unapproved command, or changing an authorization policy. Define the adversary's entry point and budget. A user who directly owns the account and a third party controlling one retrieved page are different adversaries; do not mix their results.

Maintain a clean task set with validated expected outcomes. Create attack families on separate development fixtures and freeze a held-out set before tuning. Include benign text that resembles an attack, multiple languages, long contexts, interrupted tool sequences and malicious content that appears only after an initial harmless step. Track software and policy versions so that a changed score can be attributed to an intervention.

Use four baselines: the original agent; a detector-only variant; a deterministic tool-policy gate; and their combination. Keep model, tools, tasks, allowed scopes and attack budgets fixed. Measure forbidden actions, clean success, defended success, false blocks, latency, cost and coverage of action paths. An ablation must not give one system fewer privileges or easier tasks without acknowledging the change.

Inspect complete traces. Validate tool arguments and actual side effects, not the model's description of them. Verify that denied actions did not execute, then verify that permitted actions still complete. Include timeouts, retries, expired approvals and revocation during a multi-step run. An observer that alerts after a transfer is a detective control; it is not preventive enforcement.

Repeat stochastic model runs and report uncertainty. Group results by task family to expose correlated failures. Repeated attempts against one template do not supply the same evidence as independent families. For zero observed violations, use a confidence bound with explicit assumptions; see the alignment evidence tutorial. Never turn 0/n into a promise of zero future risk.

8. Original interview practice and engineering decisions

These are original simulations, not verified questions from the companies discussed. Answer with a threat model and an observable test, not a list of product names.

1. Why can 95% attack recall still create an unusable alert queue?

First hint: Use prevalence and the benign population as well as recall.

Answer and analysis

At 0.1% prevalence in 100000 requests, there are 100 malicious and 99900 benign cases. A 1% false-positive rate creates 999 false alerts, exceeding 95 true alerts. Discuss triage cost, threshold trade-offs and the uncertainty of the prevalence estimate.

Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.

2. Why does an ethical explanation not authorize a tool call?

First hint: Separate behavior prediction from authoritative permission.

Answer and analysis

An explanation is generated content. A policy service should resolve identity, resource scope, recipient and approval from authoritative state. Test a persuasive proposal with an out-of-scope recipient and prove that the executor denies it before any side effect.

Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.

3. How could a defense achieve zero attack success while being useless?

First hint: Inspect clean task completion.

Answer and analysis

Refusing everything, disabling every tool, or failing the workflow can prevent attacker objectives while also preventing authorized work. Report utility and attack success together, plus the exact permissions and denominators.

Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.

4. What changes when an agent can write memory or improve its own prompt?

First hint: Think across runs, not only within one request.

Answer and analysis

Poisoned state can persist after the original input disappears. Version memory and prompts, restrict writes and evaluate candidates on untouched tasks with an independent gate. Test rollback and ensure the candidate cannot alter evaluation evidence.

Score yourself: 1/5 names the concept; 2/5 defines it; 3/5 derives the relevant mechanism or calculation; 4/5 tests a counterexample; 5/5 states assumptions, trade-offs and an evaluation that could disprove the answer.

For an engineer, a strong first project is a small controlled executor with auditable permissions and adversarial fixtures. For a researcher, the next step is a task family that exposes a specific failure in existing defenses without redefining success after seeing results. Continue with bounded tools, authorization after learning and the infrastructure companion.

9. Frontier research agenda: five falsifiable projects

The following are Token Lab's proposed research directions, not claims that these methods have already succeeded or are novel. Search the literature before claiming novelty. Each proposal starts with an engineering failure and ends with evidence that would count against the idea. Select one narrow workload for a first PoC; broad claims require replication across tasks, implementations and operating conditions.

1. Semantic authorization beyond a tool allowlist

Why existing approaches are insufficient: An API can be allowed while a particular recipient, amount or purpose is not. Natural-language requests often underspecify the legitimate scope, and agents revise plans as they work.

Testable hypothesis: A typed action proposal bound to resource-level policy and explicit approvals reduces unauthorized effects without excessive false blocks.

Baselines and fair comparison: Compare prompt-only restrictions, tool-name allowlists and resource-scoped enforcement. Use tasks with the same tool but different ownership and destinations.

Measurements: Measure actual unauthorized effects, authorized completion, approval burden and policy coverage. Mutate parameters after approval and test revocation between planning and execution.

What would falsify the claim: Reject if the evaluator accepts self-reported authorization or if benign success collapses. A typed schema checks structure; it does not by itself prove that the values are permitted.

2. Adaptive prompt injection under a fixed attack budget

Why existing approaches are insufficient: A defense tuned to a static phrase collection can learn surface patterns. An attacker can observe failures, alter content and exploit a different point in the workflow.

Testable hypothesis: Training or detection informed by provenance and task boundaries generalizes better than phrase matching at matched clean utility.

Baselines and fair comparison: Compare a phrase baseline, a model detector and an execution gate, with separate adaptive attack runs against each. Match query and token budgets.

Measurements: Report success per attempt and per task after the full budget, clean utility, attack cost and failures on unseen task families. Preserve all attempted attacks, not only successful examples.

What would falsify the claim: Reject if improvement disappears when attackers adapt, or if one defense gets less attack effort. Do not report a fixed-template score as a guarantee against strategic adversaries.

3. Long-horizon memory poisoning and recovery

Why existing approaches are insufficient: A single harmless-looking note can influence later decisions after the original document is gone. Most short episodes miss persistence, delayed triggers and the cost of removing contaminated state.

Testable hypothesis: Provenance-aware memory writes with quarantine and selective rollback can reduce persistent influence while retaining useful learned preferences.

Baselines and fair comparison: Compare no persistent memory, unrestricted memory and the proposed write policy. Use multi-session legitimate tasks interleaved with controlled contamination fixtures.

Measurements: Measure delayed violations, useful retention, time to detect, rollback completeness and collateral forgetting. Inspect every downstream copy of the tainted state.

What would falsify the claim: Reject if the defense succeeds only by deleting all memory or assumes a known infection time. Recovery must address derived summaries and replicas, not only the original record.

4. Multimodal and multilingual boundary preservation

Why existing approaches are insufficient: A security decision made on extracted text may not cover instructions encoded in an image, an audio segment or layout. Translation and OCR change what the model sees.

Testable hypothesis: Tracking provenance through preprocessing and testing modality-specific transformations improves coverage at matched utility.

Baselines and fair comparison: Compare a text-only detector, independent modality filters and the proposed end-to-end pipeline on paired inputs expressing the same authorized task.

Measurements: Report results by language and modality, extraction errors, false blocks, task quality and latency. Include benign ambiguous layouts and low-quality scans.

What would falsify the claim: Reject if a global average hides failure in one language, or if the defense gets access to a clean transcription unavailable in production. Human labels need agreement checks too.

5. Security evaluation for self-improving agents

Why existing approaches are insufficient: When candidates alter prompts, tools or code, the evaluator and reward become attractive targets. Passing more tests may mean learning the tests rather than improving robust capability.

Testable hypothesis: An isolated promotion gate with held-out task families and immutable execution receipts can detect more invalid improvements than an in-loop score alone.

Baselines and fair comparison: Compare self-scoring, a shared-state evaluator and an independent evaluator. Introduce benign improvements, reward exploits and unauthorized tool-scope changes as distinct candidate types.

Measurements: Measure legitimate gain, exploit acceptance, retained abilities, scope changes and audit reproducibility. Keep a final test set untouched until selection finishes.

What would falsify the claim: Reject if the candidate can edit the score, if test contamination is ignored, or if safety comes only from forbidding all changes. A useful gate must accept genuine improvements as well as reject exploits.

From a two-week PoC to a defensible report

In the first stage, write the system boundary, assumptions and failure criterion before implementing the proposed method. Build the simplest baseline and a reproducible fixture. In the second stage, add one intervention and run controlled sweeps; retain failed runs and explain exclusions. In the final stage, test an unseen workload or attack family and write a limitations section. Separate what your experiment measured from what you hope will generalize.

A useful report contains a configuration manifest, data-generation procedure, code commit, exact commands, raw observations, aggregation script and a table of threats to validity. Use multiple seeds when randomness matters; repeated deterministic arithmetic is not independent evidence. Publish negative results: the boundary where a technique stops working is often the most transferable lesson.

Foundations explained

Review the shared reasoning behind this method, then use the worked checks to identify gaps in your understanding.

An agent is a stateful control loop: observe, propose an action, check permission, execute, and inspect the result. A tool schema only describes valid arguments; it does not prove the caller is authorized or the requested side effect occurred. Model-generated text is therefore a proposal, not trusted execution evidence. You should distinguish an attempted call, a successful response and a verified task outcome.

Work it out before reading the solution

A payment tool times out after receiving a request. Should an agent immediately retry because it has no success response?

Reasoning, step by step

No. A timeout leaves the outcome unknown: payment may have succeeded before the reply was lost. Use an idempotency key and a status check, or pause for reconciliation. Blind retries can duplicate an irreversible action. A step limit bounds attempts but cannot repair this correctness error.

Why this matters in practice

This is why tool-using assistants need application-level memory, authorization and stop conditions. State must record what is known, what is uncertain and what still requires confirmation. More autonomy is useful only when completion can be verified and action scope remains controlled.

Evidence of understanding

Trace success, rejection, timeout and malformed response through the same workflow. Assert that denied actions never execute and that retries preserve idempotency. Separately cap steps, time and spend. Test these properties with a mock tool locally before considering real side effects.

Run the complete Python example

Python 3.10+ · standard library only · deterministic, no random seed needed · no network. Save the file and run the command below. These are teaching checks, not model benchmarks.

Download Python · agent.py
python3 agent.py
# Python 3.10+; standard library; deterministic; no network.
# In-memory mock only: no real payment or network call.
ledger = {}
def pay(key, amount):
    if key in ledger:
        if ledger[key] != amount:
            raise ValueError("same key with different request")
        return "already applied"
    ledger[key] = amount
    return "applied"

assert pay("order-1", 10) == "applied"
# Pretend the reply was lost. Reconcile and retry with the same key.
assert ledger.get("order-1") == 10
assert pay("order-1", 10) == "already applied"
assert len(ledger) == 1
try:
    pay("order-1", 20)
    raise AssertionError("conflicting retry accepted")
except ValueError:
    pass
print("one effect; duplicate retry suppressed")

Expected output

one effect; duplicate retry suppressed

Review the prerequisite →

Foundation revised:

Your next learning step

Freeze the holdout; compare paired runs, uncertainty, failures and compute budgets.

Build / verify → · Continue your route →

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope