Token LabSearch

Understanding AI agents: from control loops to inference infrastructure

Follow a post-deployment latency incident to build an SRE agent that gathers evidence, constrains tools, handles failures and explains its conclusions, then derive its inference and resource costs.

Sources & version details
Connected stages of an agent system
Concept illustration — Connected stages of an agent system
In this article

1. 14:02: define the deliverable behind the alert

This is an original synthetic incident, not customer data or a measured deployment. At 14:00, tenant A rolls checkout revision r17 to 10% of instances. At 14:02, five-minute p95 rises from a 120 ms baseline to 420 ms while aggregate request volume remains roughly steady. The on-call engineer asks whether the release caused it and whether to roll back. That question combines anomaly confirmation, attribution and an operation with side effects. The agent promises an evidence report, not automatic repair. Every later decision serves that deliverable.

An AI agent chooses actions from observations under a goal and constraints, then uses the results to decide what to do next. The model proposes decisions. The harness is the runtime inside the agent boundary: context construction, tool adapters, constraints, verification and correction. The environment contains the files, services and business state being observed or changed. These responsibilities remain distinct even when they share a process. A predictable report often needs a deterministic workflow rather than autonomous iteration.

Prerequisites are concrete: an HTTP timeout does not prove the server did nothing; JSON schema validates structure rather than business authority; process memory differs from durable records and external state. Readers should understand averages, percentiles and basic resource budgets. The model can initially be treated as a probabilistic decision component; the cache section additionally needs the meaning of attention keys and values.

An evidence chain through the incident (synthetic scenario)

Incident questionRequired evidenceJustified conclusion
Is latency really worse?Comparable percentiles, window and volumeConfirm the anomaly before attribution
Is r17 associated?Revision-grouped metrics and deployment logForm a release-associated hypothesis
Is the pool the mechanism?Live configuration and acquisition waitsSupport a candidate; retain alternatives
May it roll back automatically?Session capabilities and change approvalRead-only agent recommends and hands off
Reference architecture: model proposals cross an authorization gate before reaching tools. Evidence returns to the controller; denied or timed-out calls are recorded, not treated as successful actions.
Reference architecture: model proposals cross an authorization gate before reaching tools. Evidence returns to the controller; denied or timed-out calls are recorded, not treated as successful actions. View original size

2. First decision: turn a vague request into an investigation

The controller opens incident-017 with tenant A, service checkout, window 13:55–14:02 and read-only authority. Overall p95 cannot show which instances are affected, so the first proposal is two independent reads: latency grouped by revision and deployment history. The input is a scoped question; the output is a proposal, not yet new evidence. Before acquiring data, the system must determine whether these proposed calls may execute.

The entry point derives tenant, service and time-window scope from authenticated identity and assigns a task_id. The controller restores completed steps from events, then combines policy, tool definitions, the question, current state and relevant evidence. The model may request get_metrics or get_changes, or answer when evidence is sufficient. It cannot grant itself permissions or promote a document’s instructions into system policy.

Every proposal crosses a separate execution gate: tool allowlist, argument types, resource scope, call budget, deadline and approval are checked before dispatch. The executor records an observation with status, call ID, data timestamp, source and truncation information. This observation becomes input to the next decision. A timeout means evidence was not obtained; it must not become a claim that metrics were normal. Exhausted budgets and dead ends produce explicit incomplete outcomes.

Workflows suit known steps; agent decisions suit branches that depend on newly observed information. Combine them: identity, approval and archival remain deterministic, while the model chooses which diagnostic source to inspect next. Define states, contracts, stopping and recovery before adopting a framework; otherwise orchestration merely conceals ambiguous behavior.

Choosing an orchestration pattern

PatternUse whenCost or boundary
Fixed workflowSteps and branches are knownPredictable; unusual cases need an explicit fallback
Single agentNew evidence determines the next actionBound rounds, tool scope and stopping conditions
Parallel agentsEvidence gathering can be separatedCoordinate shared state; duplicated evidence is not independence

3. Before execution: put authority outside the model

The model proposes read_metrics(, , group_ The gate checks authenticated tenant identity and bounds time, fields and result size. A request for tenant B or restart_service is rejected even if its explanation sounds plausible: this session has only tenant A read capabilities. The rejection is recorded so the model can choose a legal alternative; it is not a reason to expand authority. Only then does evidence gathering begin.

A tool description tells the model how to propose a call; execution policy decides whether this user may make it. Protocols such as MCP standardize discovery and invocation, but do not automatically deliver tenant isolation, least-privilege credentials, egress restrictions or approval binding. Retrieve a capability catalogue and load a small relevant set of schemas when tool volume is large; every dynamically loaded tool still crosses the same gate.

Separate read_metrics, read_changes and restart_service. Even read-only tools can leak sensitive logs, so constrain resource scope, fields, time ranges, response size and destinations. Approval for a write must bind the target, action, argument digest, expiry and operation ID. Changed arguments invalidate approval rather than inheriting a vague statement that the user agreed.

Prompt injection attempts to turn untrusted data into instructions. A malicious runbook may request credential export. A warning in the system prompt is not a security boundary: the executor rejects unavailable capabilities, the sandbox constrains files and network, and secrets stay outside model context. Detectors provide signals; independent code enforces policy and records evidence. Production writes additionally need human takeover and recovery semantics.

4. First evidence: move from relevant documents to justified claims

Synthetic observation E1 shows r17 p95 at 420 ms and r16 at 125 ms in the same window. E2 records a connection-pool configuration change. A retrieved older runbook says the limit is 50, but live configuration E3 says 10. The runbook supplies an investigative idea, live configuration supplies a fact, and the revision comparison supplies an association. None alone proves the root cause. The next useful observation is connection-acquisition wait time, rather than a confident repetition of the document.

Use different stores for task events, cross-session user or organizational memory, and citable documents. Their retention, access and update rules differ. A document saying the default timeout is 30 seconds may be stale; runtime configuration is direct evidence of the current setting. A memory that a user usually permits restarts cannot authorize this restart.

A RAG ingestion pipeline preserves document version, provenance, timestamps and access policy. Filter by the caller’s permissions before retrieval and reranking; retain evidence identifiers in the supplied passages and connect verifiable answer claims to supporting passages. Relevant retrieval is not sufficient evidence: a runbook lists possible causes, whereas current metrics establish whether this incident fits them. An unsupported connection remains a hypothesis.

Memory updates should record who supplied a fact, when it was observed, whom it concerns and whether it supersedes earlier information. Structured facts support partial updates; narrative records retain context but create redundancy. Vectorizing everything does not solve identity or temporal validity. Deletion must propagate to derived indexes and caches as well as original text.

5. Continue the investigation: remember what matters

By round three, metrics have produced many raw records. Keep E1’s window and grouping, E2’s change, E3’s active setting, and the explicit gap that connection-wait evidence is still missing. Separate observations from pending hypotheses and retain raw evidence IDs. This compact state tells the next model call what remains to be learned; it does not manufacture new evidence. We now introduce a failure on that missing observation to see how the investigation continues honestly.

Each call receives a finite view of the system. A stable prefix contains role, policy and core tool schemas; the changing suffix contains the request, observations, rejected hypotheses and remaining budget. Dumping all logs into the prompt increases computation and obscures relevant details. Deterministic filters should select the service and time window, returning aggregates, representative samples and references to raw records.

Compression preserves the goal, constraints, evidence IDs, time ranges, failed actions and the scope of any approvals. A summary is a lossy view, not the source of truth. Retain original events for retrieval, and recheck current state before consequential actions. A retrieved runbook establishes what documentation recommends, not what the user has authorized.

Stable prefixes can improve cache reuse, but correctness takes priority over hits: changed policy or tool contracts must be updated. Reuse depends on exact tokens and positions, model/adapter identity and serving isolation, not semantic similarity. Rewriting early history during compression may invalidate later cached state. Context management and infrastructure therefore cannot be optimized independently.

6. A tool times out: recover without inventing evidence

The connection-wait read op-w1 times out. Missing data is neither zero nor proof that the pool is healthy. A read-only request may be retried under the remaining deadline and budget, with separate logical-operation and attempt records. In the synthetic continuation, E4 shows increased waits on r17 but not r16, strengthening the pool-change hypothesis. If the retry also fails, report clear release association but insufficient mechanism evidence. The local reference code demonstrates the incomplete branch; it does not implement these real monitoring queries.

Persist distinct task_id, logical operation_id and attempt_id. Repeated model proposals, duplicate message delivery and execution retries are different events. Reuse recorded tool results during recovery. If a crash occurs after a remote action but before recording success, query the target system instead of assuming the action never happened.

A reference design restores the controller from durable events and uses target-supported idempotency keys for business operations. An ambiguous result becomes unknown until reconciled or reviewed. A local database transaction cannot atomically commit an external API action. Define late results, cancellation, lease expiry and timeouts explicitly; stopping a local wait does not guarantee the remote action stopped.

Filesystem snapshots, process snapshots, conversation trajectories and KV caches are different checkpoints. Restoring a repository does not undo a database update; replaying a conversation does not guarantee the model chooses the same next action. Distinguish wrong business answers, environment failures and authorization denials, or evaluation will mislabel infrastructure faults as model limitations.

7. After six rounds: account for inference work

Pause the incident to account for its growing cost. Simplify it to six model calls, each with 4,000 input and 300 output tokens: the short final report hides repeated input work. Preserving E1–E4 and stable tool definitions supports useful context and prefix reuse. Retaining a stale E3 merely to preserve a cache hit would invalidate the investigation. Correct evidence comes first; the calculations below explain the optimization trade-off.

A model call commonly includes queueing, prefill and token-by-token decoding. Prefill processes input and creates state; decoding extends it. Agents repeatedly call the model while appending observations, so final-answer length is a poor proxy for total work. During tool waits the GPU may serve other requests, while environment memory and retained KV still consume capacity.

For an ordinary full-attention Transformer, the formula below multiplies layers L, concurrent sequences B, sequence length T, KV heads H_kv, head width d_h and bytes per element b, with a factor of two for keys and values. With , , , H_, d_ and , KV alone occupies 1 GiB; 32 such sequences occupy 32 GiB. This excludes weights, activations, fragmentation and runtime allocations. Sliding windows, quantization, MLA and hybrid recurrent architectures require different accounting.

Consider six calls, each with 4,000 input and 300 output tokens: 24,000 input and 1,800 output tokens in total. If the last five calls each reuse an exact 3,000-token prefix, at most 15,000 prefix tokens avoid repeated computation. Logical input length is unchanged and billing discounts are provider-specific. This synthetic example does not establish a speedup factor, nor an equal proportional reduction in end-to-end latency.

I_r and O_r are input/output token counts for round r; R is the number of calls. I and O are logical totals before any provider-specific cache discount.

8. Many alerts arrive: locate the actual capacity limit

Now scale the same investigation to an on-call surge: two admitted tasks per second, with each environment resident for 30 seconds across model and tool waits. CPU may be mostly waiting, but retained environment state does not disappear. More cores may therefore fail to reduce the queue. Capacity planning must jointly consider active compute, resident memory, model concurrency and external-query quotas; model throughput is not platform throughput.

Two resource paths are coupled by the controller: model serving consumes accelerators and model state, while tools consume CPU, memory, filesystems and external quotas. A tool may wait for a model decision; the next model call may wait for a tool observation. Optimize successful end-to-end completion under deadlines, rather than GPU utilization in isolation.

Original synthetic example: two arriving tasks per second, a 30-second environment lifetime and 0.5 GiB per environment imply 60 concurrent environments and 30 GiB on average in a stable system. Three CPU-seconds per task imply only six busy cores. An eight-core, 16-GiB machine is nevertheless insufficient: ignoring host overhead, memory permits 32 environments, about 1.07 tasks per second. At an arrival rate of two, the queue cannot remain stable under these assumptions.

Options include lazy environment creation, releasing reconstructible resources during waits, warm pools and admission limits. Warm pools reduce startup delay but increase residency; KV offload frees GPU memory but adds transfer and restoration; batching improves throughput but may hurt interaction latency. Dependency-heavy batches also require critical-path and makespan analysis. Prioritizing long trajectories can help a batch, but fairness requires aging or quotas to prevent starvation.

λ is admitted arrivals per second; T is environment lifetime in seconds; N is average resident environments. M(t) is GiB at time t, so the integral is GiB·s. Little’s law assumes a stable long-run system; it cannot justify an overloaded queue.

9. Deliver and close the incident before improving the agent

The deliverable cites E1–E4, names the r17 pool change as a candidate, discloses unresolved traffic and downstream differences, and recommends that the on-call engineer assess rollback through the existing change process. The read-only agent stops here. Suppose a human approves and executes rollback and comparable p95 returns to 130 ms: this strengthens release association but is not a confounder-free causal experiment. Save evidence, human action records and recovery observations. Closing the incident and improving the model are separate processes; only identified single-agent failures justify extra agents or learning.

Multiple agents help when work decomposes, evidence can be acquired independently and integration cost is bounded. Metrics analysis and change-history analysis can run concurrently, followed by a coordinator checking timing and causal hypotheses. Extra roles on a simple task often add calls, duplicate evidence and disagreement. Two agents repeating the same erroneous document do not provide independent confirmation.

Separate knowledge updates, prompt/tool changes, program changes and parameter training. In-context adaptation is not parameter learning; recording successful traces does not prove general capability improvement. Version each candidate change, budget, task set and rollback point. Held-out answers must not enter training. Having one model generate and judge its own solutions creates a biased feedback loop; independent outcome checks and human review are still needed.

Start with a deterministic baseline, then bounded decisions, and only then consider multiple agents or post-training. Open problems include long-horizon credit assignment, safe recovery of irreversible actions, cross-environment transfer, accumulated memory errors and measuring useful autonomy under a budget. Claims of self-improvement must identify the changed layer, held-out gains, and any safety or resource trade-off.

10. Practice and verification

Build the deterministic harness below before connecting a model. It demonstrates scope checks, replay, call budgets and incomplete evidence with synthetic tool responses. Python 3.11+; standard library only, no dependencies or seed required because no randomness is used. Download the file and run the command locally.

agent-book-harness.py
python3 agent-book-harness.py
scope=passed replay=passed budget=passed timeout=incomplete
kv_gib=1.0 environments=60 memory_gib=30.0
Python · agent-book-harness.py
"""Python 3.11+, standard library only; deterministic harness, NOT an LLM.
No network, secrets, shell, production writes or random sampling.
Run: python3 agent-book-harness.py
"""
from dataclasses import dataclass, field

@dataclass
class Harness:
    tenant: str = 'A'
    service: str = 'checkout'
    remaining: int = 4
    events: list = field(default_factory=list)
    results: dict = field(default_factory=dict)

    def call(self, operation_id, tool, tenant, service, *, timeout=False):
        # Authorization precedes result lookup: cached data is not public.
        if tenant != self.tenant or service != self.service:
            raise PermissionError('scope')
        if tool not in {'metrics', 'changes'}:
            raise PermissionError('capability')
        identity = (tool, tenant, service)
        if operation_id in self.results:
            previous_identity, result = self.results[operation_id]
            if previous_identity != identity:
                raise ValueError('operation ID reused for different arguments')
            return result
        if self.remaining == 0:
            raise RuntimeError('budget')
        self.remaining -= 1
        if timeout:
            result = {'status': 'timeout', 'evidence': None}
        else:
            result = {'status': 'ok', 'evidence':
                      {'id': 'metric-1', 'p95_ms': 420} if tool == 'metrics'
                      else {'id': 'change-1', 'revision': 'r17'}}
        self.events.append((operation_id, identity, result))
        self.results[operation_id] = (identity, result)
        return result


def scripted_investigation(h, timeout=False):
    # A model adapter would propose these actions; the gate must stay outside it.
    metrics = h.call('op-m', 'metrics', 'A', 'checkout', timeout=timeout)
    changes = h.call('op-c', 'changes', 'A', 'checkout')
    if any(x['status'] != 'ok' for x in (metrics, changes)):
        return {'status': 'incomplete', 'evidence': []}
    return {'status': 'observed', 'evidence': [metrics['evidence'], changes['evidence']],
            'conclusion': 'Revision and latency observed; causation not established.'}


def raises(error, action):
    try:
        action()
    except error:
        return
    raise AssertionError(f'expected {error.__name__}')


def main():
    h = Harness()
    answer = scripted_investigation(h)
    assert answer['status'] == 'observed' and len(h.events) == 2
    # Replay returns recorded result without consuming another budget unit.
    h.call('op-m', 'metrics', 'A', 'checkout')
    assert h.remaining == 2 and len(h.events) == 2
    raises(PermissionError, lambda: h.call('op-m', 'metrics', 'B', 'checkout'))
    raises(PermissionError, lambda: h.call('op-write', 'restart', 'A', 'checkout'))
    raises(ValueError, lambda: h.call('op-m', 'changes', 'A', 'checkout'))
    raises(RuntimeError, lambda: Harness(remaining=0).call('x', 'metrics', 'A', 'checkout'))
    assert scripted_investigation(Harness(), timeout=True)['status'] == 'incomplete'
    kv = 2 * 32 * 1 * 8192 * 8 * 128 * 2
    assert kv == 1024**3
    environments = 2 * 30
    memory_gib = environments * 0.5
    assert environments == 60 and memory_gib == 30
    assert 6 * 4000 - 5 * 3000 == 9000
    print('scope=passed replay=passed budget=passed timeout=incomplete')
    print(f'kv_gib={kv / 1024**3:.1f} environments={environments} memory_gib={memory_gib:.1f}')
    print('Synthetic control checks only; no model-quality or sandbox benchmark.')

if __name__ == '__main__':
    main()

This is a control-path implementation, not a deployed SRE agent or an LLM simulation. Its in-memory ledger does not survive a crash; timeout is an injected outcome, not a process-killing deadline. It has no real sandbox or writes. Production requires durable storage, authenticated identity, real deadlines, concurrency control, target idempotency and a separately configured model adapter.

Fair comparison: keep model, tasks, evidence, tool permissions and total token/time budget fixed. Compare a fixed workflow with one bounded agent, then multiple agents only if the task decomposes. Report task success, unsupported claims, denied actions, p50/p95 latency and total cost per successful task; count failed attempts in cost. Repeat on held-out tasks and disclose sample size. Passing these local assertions establishes control behavior, not agent intelligence.

Original interview practice—not reported company questions

Why can a timeout cause a duplicate restart?

Hint

Separate remote execution from local acknowledgement.

Answer and analysis

The action may finish before the response is lost. Query operation status and use target-supported idempotency; unknown is not failure-to-execute.

Why can compressing context increase next-call latency?

Hint

Consider the first changed token.

Answer and analysis

Compression may invalidate a cached prefix and require prefill again. Measure saved input work against lost reuse and compression cost, while checking retained evidence.

Why does the example need 30 GiB but only six CPU cores on average?

Hint

Separate residency from active CPU time.

Answer and analysis

Little’s law gives 2×30=60 environments; 60×0.5=30 GiB. CPU demand is 2×3=6 core-seconds per second. Waiting state remains resident.

Do two agreeing agents establish correctness?

Hint

Ask whether their evidence is independent.

Answer and analysis

Shared models or sources can produce correlated errors. Check the actual outcome and independent evidence; compare against a budget-matched single-agent baseline.

Scoring: 1—names concepts; 2—explains the normal flow; 3—reproduces units and examples; 4—handles timeout, scope and replay failures; 5—designs a fair comparison, recovery boundary and measurable trade-off.

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope

11. Attribution, source versions and further reading

This article is Token Lab’s original synthesis, informed by Bojie Li’s Understanding AI Infra: Quantitative Analysis and System Design and its companion Understanding AI Agent: Design Principles and Engineering Practice. The supplied link points to the Infra book, not the Agent book. Both repositories declare Apache-2.0. We link to pinned source versions; our SRE scenario, diagram, synthetic numbers and Python control example are independently authored. This is not a translation of the complete books, an official edition or an author endorsement.

AI Infra §3https://github.com/bojieli/ai-infra-book/blob/3bdcb4fcab73010eeaccf31cc10a8242a895d82e/manuscripts/03-%E6%8E%A8%E7%90%86%E4%B8%8E%E8%AE%AD%E7%BB%83%E8%B4%9F%E8%BD%BD.md · AI Infra §8https://github.com/bojieli/ai-infra-book/blob/3bdcb4fcab73010eeaccf31cc10a8242a895d82e/manuscripts/08-%E6%8E%A8%E7%90%86%E4%BC%98%E5%8C%96.md · AI Infra §11https://github.com/bojieli/ai-infra-book/blob/3bdcb4fcab73010eeaccf31cc10a8242a895d82e/manuscripts/11-%E8%B5%84%E6%BA%90%E8%B0%83%E5%BA%A6%E4%B8%8E%E8%BF%90%E8%A1%8C%E7%8E%AF%E5%A2%83.md · AI Agent §1https://github.com/bojieli/ai-agent-book/blob/dbc046eb896ac4e39aa19c7774c8bf49583b89a6/book/chapter1.md · AI Agent §2https://github.com/bojieli/ai-agent-book/blob/dbc046eb896ac4e39aa19c7774c8bf49583b89a6/book/chapter2.md · AI Agent §3https://github.com/bojieli/ai-agent-book/blob/dbc046eb896ac4e39aa19c7774c8bf49583b89a6/book/chapter3.md · AI Agent §4https://github.com/bojieli/ai-agent-book/blob/dbc046eb896ac4e39aa19c7774c8bf49583b89a6/book/chapter4.md · AI Agent §7https://github.com/bojieli/ai-agent-book/blob/dbc046eb896ac4e39aa19c7774c8bf49583b89a6/book/chapter7.md · AI Agent §9https://github.com/bojieli/ai-agent-book/blob/dbc046eb896ac4e39aa19c7774c8bf49583b89a6/book/chapter9.md · AI Agent §10https://github.com/bojieli/ai-agent-book/blob/dbc046eb896ac4e39aa19c7774c8bf49583b89a6/book/chapter10.md

AI Infra · Apache-2.0https://github.com/bojieli/ai-infra-book/blob/3bdcb4fcab73010eeaccf31cc10a8242a895d82e/LICENSE · AI Agent · Apache-2.0https://github.com/bojieli/ai-agent-book/blob/dbc046eb896ac4e39aa19c7774c8bf49583b89a6/LICENSE

Continue with the implementation tutorials

Sources & version details

Videos with a purpose

Watch → explain → implement → test

These are external English-language resources. Captions and availability depend on YouTube. The player loads only after you choose Play; it contacts YouTube, and no experiment inputs are sent.

Your next learning step

Test invalid arguments, expired permission and duplicate retries; inspect final state.

Build / verify → · Continue your route →