Token LabSearch

STANFORD CS329Z · FALL 2026

Stanford CS329Z: a complete guide to Engineering AI Agents

Understand the curriculum, build a harness, design an evaluation task, and find every currently linked lecture, reading and assignment.

AI AgentsHarnessBM25HarborEvaluation

Sources & version details
In this article

What this course teaches—and what is public

An agent harness is the software that turns model output into controlled interaction with an environment. It owns the action parser, tool registry, execution boundaries, context, persistent state and stopping rules. A model generates a proposal; the surrounding program determines whether it is valid, authorized and executable. This distinction is the organizing idea of this guide. You will first understand a fixed workflow, then a model-directed loop, then how to collect evidence that either system works.

CS329Z is Stanford’s Fall 2026 Engineering AI Agents course, taught by Diyi Yang, Michael Ryan and John Yang. At our November 11 source check, the official timetable linked 13 slide PDFs, two public videos and the lecture-five code repository. Both homework repositories and their version 26.1.0 handouts were public. Later timetable entries are a plan, not proof that a recording exists. The logistics page still describes Canvas recordings; use the individually verified public links below rather than assuming access to Canvas.

This is Token Lab’s independent study companion, not a Stanford course, certificate, official solution set or transcript of every lecture. We inspected the timetable, logistics, assignment handouts and repository setup files. We have not watched every lecture, reproduced the paid model evaluations or completed the Stanford assignments. The explanations, diagrams and additional exercises are ours; the official materials remain at their original destinations.

Prerequisites that actually help you build

The official prerequisite is an NLP background equivalent to CS224N, CS224U, CS224V or CS336. For self-study, translate that into concrete skills. In Python, understand dictionaries, typed records, exceptions, file I/O and tests: a tool call is a data object, and a failed subprocess must become an explicit result rather than a crashed conversation. Know HTTP requests, environment variables and Git so that secrets, code and experimental outputs remain separate.

For LLMs, explain next-token prediction, sampling and the difference between a message and a token. A context window bounds what a model sees now; persistent storage can retain facts that are not currently in the prompt. Similarity retrieval is not truth verification. Basic probability helps you distinguish one successful run from a reliable system; elementary algebra is enough for BM25 and cost accounting. Docker adds a reproducible filesystem and process environment, but an image alone does not guarantee safe permissions or deterministic model output.

Start with a small local corpus and synthetic identities. Do not upload your employer’s mailbox merely because the assignment uses corporate email. The provided teaching archive has its own provenance. If you cannot explain a function that an AI assistant writes for you, reduce its scope and reconstruct the data flow before proceeding.

How the lectures build on each other

The opening block moves from agent systems to LLMs, retrieval and tools. A fixed workflow selects its next step in code; an agent delegates some step selection to a model. That delegation makes the message contract, evidence selection and tool schema part of your program semantics. Read RAG as a mechanism for selecting evidence under a token budget, then tool use as a mechanism for changing state. Confusing these two leads to treating a retrieved instruction as permission.

Frameworks and design patterns come next because abstraction is easier to judge after implementing the primitive loop. Compare a graph with explicit transitions against a free-form loop: the graph exposes state and failure paths, while the loop accommodates less predictable work at the cost of a larger action space. Memory changes what survives a session. Multi-agent design changes who owns each subproblem and shared state; it also introduces coordination cost and conflicting conclusions. More agents are not a substitute for a clear protocol.

The optimization and data block asks what should improve and what evidence drives improvement. Data selection must match the target capability; repeated easy traces can inflate volume without covering new failures. Separate training, development and held-out evaluation examples before tuning. Terminal-Bench provides the bridge from an agent implementation to a task/environment/verifier unit. The data-quality and benchmark lectures then explain why a result is only meaningful relative to the chosen task distribution and scoring procedure.

The remaining schedule covers LLM judges and evaluation infrastructure, SWE-bench, safety, coding agents and proactive agents, plus the Andrew Ng conversation and final demos. These are timetable topics, not summaries of unreleased recordings. A useful connection is authority over time: a proactive agent acts without a fresh user message, so its trigger, scope, spending limit and stopping condition must already be explicit. Safety therefore belongs in the system design, not just a final prompt. Use the public schedule to discover new materials rather than assuming that every planned session has a video.

HW1: from deterministic workflows to an agent runtime

Assignment 1 deliberately removes the convenience of agent frameworks. Its LM primitive accepts role/content messages and returns text; your code must describe tools, parse calls and dispatch execution. The starter exposes adapters as the contract used by the tests. Implement inside cs329z_hw1, wire adapters to your modules, and preserve supplied types and grading code. Do not add dependencies; NumPy is already available. This is an educational harness, not feature parity with Claude Code or Codex.

The first four problems create reusable components: email priority classification, daily digest, BM25 retrieval and multi-hop email question answering. BM25 combines term rarity, within-document frequency saturation and length normalization. A query term repeated twenty times should not receive twenty times the evidence weight. Multi-hop retrieval matters when one message identifies a project owner and another records a later decision: the answer requires joining evidence, not merely summarizing the first hit.

The agent half adds a registry, a text protocol, a bounded loop, company tools, document retrieval and a terminal. Parse model text as untrusted input. Validate tool name and arguments before requesting approval; record denied and failed calls as observations. Retain separate execution state and display text so that a friendly sentence cannot hide an unexecuted action. The course’s confirmation mode checks non-read-only calls; production systems also need resource scope and data-disclosure policies because reading can expose secrets.

Finally, add compaction, persistent memory, user steering and guardrails. Compaction is a lossy transformation: preserve the user’s constraints, outstanding actions and evidence pointers, not just a fluent story. Memory must be scoped by the configured directory; a separate user’s store must not leak into the session. Keep a ledger of actual actions independently of model-written summaries. The assignment’s terminal restrictions are not equivalent to a production security boundary: inspect sandbox.jail(), especially on Windows where the documented result is none.

HW2: design a task that measures the right capability

Harbor coordinates the evaluation; Docker supplies the task environment; mini-SWE-agent is the system being evaluated. Keep these responsibilities separate. The task package describes the request, initial filesystem and dependencies, time/resource limits, verification and an oracle solution. An oracle is a known solution used to validate the task; it must not be available to the evaluated agent. A model’s final claim of success is not the reward: the verifier examines the resulting state.

The handout moves from a local HumanEval task to SWE-bench Verified and Terminal-Bench 2.1, then asks you to author a task. The required model in version 26.1.0 is Gemini 3.8 Flash, with five independent attempts using the supplied pinned_agent.py wrapper. Preserve the repository’s dependency pins. Its Python constraint is >=3.12,<3.14 and Harbor is pinned to 0.18.0; these are course snapshot requirements, not a recommendation to use those versions for every new project.

A useful task has a clear capability claim. For example, “restore a service configuration without altering unrelated services” tests diagnosis and constrained repair. A verifier that checks only whether one file exists is too weak; one that requires your exact patch rejects valid alternatives. Test postconditions, invariants and prohibited side effects. Do not hide the answer in the starting image, leak test fixtures to the agent, depend on an unpinned remote service or make difficulty come from a broken installation.

The four parts are basics 5%, running examples 10%, authoring 70% and peer review 15%. Within authoring: reproducibility 10, capability 15, task–verifier fidelity 15, difficulty 20 and report 10. The handout asks for three peer-task reviews, while the general logistics page describes two peers per review activity. Treat the specific HW2 handout as the task-specific instruction, and ask course staff if enrolled; do not silently merge the two requirements.

The mechanisms connecting the assignments

Consider an employee asking whether a reimbursement is allowed and then requesting an action. Retrieval supplies candidate evidence, not permission. The harness reads the relevant policy, binds the answer to a document and version, constructs a proposed tool call, validates it and checks authority before execution. A denied action must not reappear as a differently named tool call. A timeout creates uncertainty about whether a side effect occurred; inspect the operation state before retrying. This is an original production extension of the course’s teaching design, not a claim about its starter implementation.

In the reference architecture, the model cannot directly reach credentials or execute a shell. All action proposals cross a host-controlled policy gate. Read-only document access still respects tenant scope. Writes use approval and an operation identifier; returned observations go into the event log. Context is a selected view over that log, while persistent memory is a separate scoped store. A summary may describe a completed action only when execution evidence supports it.

A context budget is additive: system instructions, conversation, selected tool observations and reserved output must fit the model limit. With a synthetic 16,000-token window, 2,000 system tokens, 4,000 dialogue tokens and 2,000 output reserve leave 8,000 for evidence. Four characters per token is the course wrapper’s approximation, not a tokenizer identity or a multilingual guarantee. A production budget should use the actual model tokenizer and include provider overhead.

Reliability compounds across required steps. Under the strong simplifying assumption of independent 95%-reliable steps, ten necessary steps succeed together only about 59.9% of the time. Real failures may be correlated, so this is not a benchmark prediction. It explains why shorter action paths, recovery and explicit termination matter. More tools increase possibility, not automatically reliability.

Reference design: the model proposes; the host authorizes and executes. A denial returns to the event log without execution.
Reference design: the model proposes; the host authorizes and executes. A denial returns to the event log without execution.

Environment, model versions and a realistic budget

HW1 needs Python 3.11+ and uv; Windows additionally uses Git for Windows for Bash, or WSL. HW2 requires Docker with a reachable daemon, uv and Python 3.12–3.13 according to its project file. On Apple Silicon the SWE-bench example requires the documented linux/amd64 platform setting. This may involve emulation and different runtime costs. Use separate project environments; do not upgrade a shared environment and assume the locked assignments remain reproducible.

The inspected HW1 README identifies openai/gpt-6-luna as the grading model and openai/gpt-6-sol as judge; HW2 names Gemini 3.8 Flash through Gemini or OpenRouter. These are identifiers in the inspected course files, not a promise that every external account has access or the same price. External learners supply their own provider credentials and credits. We did not provision a key or make paid calls. Token Lab itself does not collect assignment keys or send readers’ exercise inputs to model providers.

The HW1 handout reports $0.05 for one reference-solution pass over live tests excluding the final evaluation, then $0.18 for that evaluation. This is $0.23 for that reported run, not a completion guarantee. Debugging, prompt changes, model substitutions and retries create new paid requests. Cache identity includes model, sampling settings and exact messages. A cache hit is not an independent repeat for reliability measurement.

Set both a provider-side spending limit and CS329Z_BUDGET_USD. The local ledger guard stops new calls after its recorded total reaches the threshold; do not treat it as a transactionally exact cap across concurrent processes. Deleting .lm_cache removes cached replies and local accounting. Record run identifiers, model, configuration, token counts and cost before cleanup. A native terminal or Docker exercise is local execution, but live model tests still transmit prompts to an external API and incur charges.

Course deliverables, project and responsible reuse

The course-wide weights are project 40%, homework 15%, homework quizzes 15%, paper video 10% and peer review 20%. Project components are proposal 5%, midway report 5%, midpoint demo 5%, final submission 15% and final system demo 10%. These are Stanford enrollment requirements; an independent reader can use the same milestones without claiming enrollment or credit. We did not find a complete public final-project rubric in the inspected pages, so the project plan below is our suggested study structure, not an official substitute.

A useful independent project is an SRE incident assistant: ingest a synthetic incident snapshot, retrieve a runbook, return an evidence-linked diagnosis and propose a bounded action requiring approval. First deliver a read-only baseline. Then add persistence and failure handling. Finally compare the improved system with the baseline on a frozen set of incidents, including missing evidence and a denied action. Produce source, dependency lock, configuration, architecture, traces, cost report and a clear list of unsupported operations.

The official paper-video handout requests 10–15 minutes, a paper explanation, critique and an added contribution such as a code experiment. Its rubric is 20+20+20+10 points. Submitters supply an unlisted YouTube link and their materials. Its AI rules are stricter than general coding assistance: slides, script, analysis and narration must be the student’s own; AI may support understanding and feedback but must not make the slides. Course project reports also cannot be written by AI. Homework code assistance is permitted only with understanding and validation; oral quizzes check that understanding.

The two assignment repositories carry MIT licenses; preserve copyright and license notices when reusing code or substantial documentation. That does not automatically license Stanford lecture videos, slides or third-party readings. This guide links those resources rather than mirroring them. The public repositories contain starters, not an invitation to publish classmates’ solutions. Enrolled students should follow the private-submission and academic-integrity rules.

Curriculum and learning sequence

Slides / videos / code / readings are listed below in course order. A missing video link means no public video was linked in the checked timetable; it does not prove a recording does not exist. Reading titles retain their original published language.

  1. Agent systems Materials (5)
  2. LLMs for builders Materials (5)
  3. RAG Materials (4)
  4. Tool use Materials (4)
  5. Frameworks Materials (5)
  6. Design patterns Materials (6)
  7. Memory Materials (4)
  8. Multi-agent systems Materials (4)
  9. Optimization Materials (4)
  10. Terminal-Bench Materials (3)
  11. Agent data Materials (3)
  12. Data quality Materials (4)
  13. Evaluation and benchmarks Materials (4)
  14. Andrew Ng conversation Materials (0)
  15. LLM judges and evaluation infrastructure Materials (4)
  16. SWE-bench Materials (1)
  17. Safety and guardrails Materials (6)
  18. No class Materials (0)
  19. No class Materials (0)
  20. Coding agents Materials (4)
  21. Proactive agents Materials (3)
  22. Final demo Materials (0)

Timeline and deliverables

Date (2026, PT)Topic
10-09 23:59Project proposal
10-30Assignment 1
11-06Midpoint report + demo
11-13 23:59Paper video
11-23 23:59Assignment 2 task
11-30Paper peer reviews (2)
12-02Assignment 2 peer reviews (3)
12-09 15:30–18:30Final demo
12-11 23:59Final report

Practice, validation and self-study plan

Use four milestones rather than watching everything passively. Week 1: explain the model/message contract and implement deterministic workflow and BM25 tests. Week 2: implement the bounded harness, then document permissions, summaries and memory decisions. Week 3: package a task with an oracle and adversarial verifier checks. Week 4: run five controlled attempts, analyse failures and write a report. This is an original self-study schedule, not a Stanford deadline.

Assignment 1: grading map

ModuleDeterministicLive APITotal
priority123
daily_digest134
bm25808
email_qa268
tool_registry303
tool_protocol224
agent_loop808
cardinal202
search_docs235
terminal246
compaction538
memory358
interactive707
guardrails224
sim_eval01010

Autograded: 48 + 40 = 88. Field notes: 10. Design memo: 20. Assignment total: 118; the 88 is not the whole assignment.

Run deterministic tests first. The live and simulation commands below send data to external providers and may incur charges; they are optional coursework commands, not browser-local experiments. Put your own credentials in .env, never in source control. Token Lab has not run these paid tests or completed the assignments for you.

git clone https://github.com/cs329z/assignment1-harness.git
cd assignment1-harness
uv sync
uv run python data/download.py
uv run pytest
CS329Z_EVAL_SLICE=5 uv run pytest -m live -s
uv run python -m cs329z_hw1.simulation --slice 3
uv run python -m cs329z_hw1.chat --mode confirm
git clone https://github.com/cs329z/assignment2-evals.git
cd assignment2-evals
uv sync
uv run python check_submission.py

Assignment 2: setup 5%, example runs 10%, authored task 70%, peer review 15%. The 70% consists of reproducibility 10, capability 15, fidelity 15, difficulty 20 and report 10. Read the official handout for the pinned agent command and provider setup; do not substitute an arbitrary Harbor agent.

uv run harbor run --path tasks/YOUR_TASK_ID --agent-import-path pinned_agent:MiniSweAgent --model openrouter/google/gemini-3.8-flash --env-file .env --n-attempts 5 --n-concurrent 1 --job-name YOUR_TASK_ID-gemini --jobs-dir tasks/YOUR_TASK_ID/artifacts --yes

Original local exercise: five attempts are not certainty

The standard-library program below prints a 95% Wilson interval for three successes in five independent Bernoulli trials, then checks empty, all-failure and all-success cases. Expected interval: 0.2307–0.8824. This measures sampling uncertainty, not task validity, and assumes comparable independent trials. Cached repetitions do not supply independent evidence.

Local exercise download
"""Token Lab original supplement. Python 3.11+, stdlib, no network or randomness."""
from math import sqrt

def wilson(successes, trials, z=1.959963984540054):
    if not isinstance(trials, int) or not isinstance(successes, int) or trials <= 0 or not 0 <= successes <= trials:
        raise ValueError("Require 0 <= successes <= positive integer trials")
    p = successes / trials
    denominator = 1 + z*z/trials
    center = (p + z*z/(2*trials)) / denominator
    half = z * sqrt(p*(1-p)/trials + z*z/(4*trials*trials)) / denominator
    return max(0.0, center-half), min(1.0, center+half)

if __name__ == "__main__":
    lo, hi = wilson(3, 5)
    assert abs(lo - 0.2307242813) < 1e-9
    assert abs(hi - 0.8823792258) < 1e-9
    assert wilson(0, 5)[0] < 1e-12
    assert abs(wilson(5, 5)[1] - 1) < 1e-12
    for args in [(0, 0), (6, 5), (-1, 5)]:
        try:
            wilson(*args)
        except ValueError:
            pass
        else:
            raise AssertionError("invalid input accepted")
    print(f"3/5: 95% Wilson interval [{lo:.4f}, {hi:.4f}]")
    print("PASS: boundaries and invalid inputs")
python3 uncertainty.py
# 3/5: 95% Wilson interval [0.2307, 0.8824]
# PASS: boundaries and invalid inputs

Original interview practice—not company interview reports

1. Why is valid tool-call syntax not authorization?

Hint: separate parsing from policy. Answer: parse against a schema, then check identity, scope and approval in host code; a model-provided is not authority. Test a syntactically valid forbidden action.

2. Does five cached successes prove reliability?

Hint: count independent draws. Answer: cached replay tests plumbing, not fresh model variability. Use independent controlled runs, retain seed/settings and report counts and uncertainty.

3. How can a task pass while failing its purpose?

Hint: verifier coverage. Answer: a weak check may accept a hard-coded string or edited answer file. Test trivial bypasses, isolate hidden checks, verify resulting state, and ensure the oracle passes. Difficulty without validity is not useful.

Score each answer: 1 names a concept; 2 defines it; 3 explains a concrete mechanism; 4 handles failure and alternatives; 5 designs a fair test and states evidence limits.

Official materials, videos and further reading

Slides / videos / code / readings are listed below in course order. A missing video link means no public video was linked in the checked timetable; it does not prove a recording does not exist. Reading titles retain their original published language.

1. Agent systems

Wed Sep 23 · 2026

2. LLMs for builders

Mon Sep 28 · 2026

3. RAG

Wed Sep 30 · 2026

4. Tool use

Mon Oct 5 · 2026

5. Frameworks

Wed Oct 7 · 2026

6. Design patterns

Mon Oct 12 · 2026

7. Memory

Wed Oct 14 · 2026

8. Multi-agent systems

Mon Oct 19 · 2026

9. Optimization

Wed Oct 21 · 2026

10. Terminal-Bench

Mon Oct 26 · 2026

11. Agent data

Wed Oct 28 · 2026

12. Data quality

Mon Nov 2 · 2026

13. Evaluation and benchmarks

Wed Nov 4 · 2026

14. Andrew Ng conversation

Mon Nov 9 · 2026

    15. LLM judges and evaluation infrastructure

    Wed Nov 11 · 2026

    16. SWE-bench

    Mon Nov 16 · 2026

    17. Safety and guardrails

    Wed Nov 18 · 2026

    18. No class

    Mon Nov 23 · 2026

      19. No class

      Wed Nov 25 · 2026

        20. Coding agents

        Mon Nov 30 · 2026

        21. Proactive agents

        Wed Dec 2 · 2026

        22. Final demo

        Wed Dec 9 · 2026

          Study guide & lesson outlineLEARNING CONTRACT

          Understand → implement → investigate

          Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

          Predict, test and keep evidence

          Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

          Open notebook / export / clear →

          The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope