Stanford CS329Z: a complete guide to Engineering AI Agents
Understand the curriculum, build a harness, design an evaluation task, and find every currently linked lecture, reading and assignment.
Sources & version details
In this article
What this course teaches—and what is public
An agent harness is the software that turns model output into controlled interaction with an environment. It owns the action parser, tool registry, execution boundaries, context, persistent state and stopping rules. A model generates a proposal; the surrounding program determines whether it is valid, authorized and executable. This distinction is the organizing idea of this guide. You will first understand a fixed workflow, then a model-directed loop, then how to collect evidence that either system works.
CS329Z is Stanford’s Fall 2026 Engineering AI Agents course, taught by Diyi Yang, Michael Ryan and John Yang. At our November 11 source check, the official timetable linked 13 slide PDFs, two public videos and the lecture-five code repository. Both homework repositories and their version 26.1.0 handouts were public. Later timetable entries are a plan, not proof that a recording exists. The logistics page still describes Canvas recordings; use the individually verified public links below rather than assuming access to Canvas.
This is Token Lab’s independent study companion, not a Stanford course, certificate, official solution set or transcript of every lecture. We inspected the timetable, logistics, assignment handouts and repository setup files. We have not watched every lecture, reproduced the paid model evaluations or completed the Stanford assignments. The explanations, diagrams and additional exercises are ours; the official materials remain at their original destinations.
Prerequisites that actually help you build
The official prerequisite is an NLP background equivalent to CS224N, CS224U, CS224V or CS336. For self-study, translate that into concrete skills. In Python, understand dictionaries, typed records, exceptions, file I/O and tests: a tool call is a data object, and a failed subprocess must become an explicit result rather than a crashed conversation. Know HTTP requests, environment variables and Git so that secrets, code and experimental outputs remain separate.
For LLMs, explain next-token prediction, sampling and the difference between a message and a token. A context window bounds what a model sees now; persistent storage can retain facts that are not currently in the prompt. Similarity retrieval is not truth verification. Basic probability helps you distinguish one successful run from a reliable system; elementary algebra is enough for BM25 and cost accounting. Docker adds a reproducible filesystem and process environment, but an image alone does not guarantee safe permissions or deterministic model output.
Start with a small local corpus and synthetic identities. Do not upload your employer’s mailbox merely because the assignment uses corporate email. The provided teaching archive has its own provenance. If you cannot explain a function that an AI assistant writes for you, reduce its scope and reconstruct the data flow before proceeding.
How the lectures build on each other
The opening block moves from agent systems to LLMs, retrieval and tools. A fixed workflow selects its next step in code; an agent delegates some step selection to a model. That delegation makes the message contract, evidence selection and tool schema part of your program semantics. Read RAG as a mechanism for selecting evidence under a token budget, then tool use as a mechanism for changing state. Confusing these two leads to treating a retrieved instruction as permission.
Frameworks and design patterns come next because abstraction is easier to judge after implementing the primitive loop. Compare a graph with explicit transitions against a free-form loop: the graph exposes state and failure paths, while the loop accommodates less predictable work at the cost of a larger action space. Memory changes what survives a session. Multi-agent design changes who owns each subproblem and shared state; it also introduces coordination cost and conflicting conclusions. More agents are not a substitute for a clear protocol.
The optimization and data block asks what should improve and what evidence drives improvement. Data selection must match the target capability; repeated easy traces can inflate volume without covering new failures. Separate training, development and held-out evaluation examples before tuning. Terminal-Bench provides the bridge from an agent implementation to a task/environment/verifier unit. The data-quality and benchmark lectures then explain why a result is only meaningful relative to the chosen task distribution and scoring procedure.
The remaining schedule covers LLM judges and evaluation infrastructure, SWE-bench, safety, coding agents and proactive agents, plus the Andrew Ng conversation and final demos. These are timetable topics, not summaries of unreleased recordings. A useful connection is authority over time: a proactive agent acts without a fresh user message, so its trigger, scope, spending limit and stopping condition must already be explicit. Safety therefore belongs in the system design, not just a final prompt. Use the public schedule to discover new materials rather than assuming that every planned session has a video.
HW1: from deterministic workflows to an agent runtime
Assignment 1 deliberately removes the convenience of agent frameworks. Its LM primitive accepts role/content messages and returns text; your code must describe tools, parse calls and dispatch execution. The starter exposes adapters as the contract used by the tests. Implement inside cs329z_hw1, wire adapters to your modules, and preserve supplied types and grading code. Do not add dependencies; NumPy is already available. This is an educational harness, not feature parity with Claude Code or Codex.
The first four problems create reusable components: email priority classification, daily digest, BM25 retrieval and multi-hop email question answering. BM25 combines term rarity, within-document frequency saturation and length normalization. A query term repeated twenty times should not receive twenty times the evidence weight. Multi-hop retrieval matters when one message identifies a project owner and another records a later decision: the answer requires joining evidence, not merely summarizing the first hit.
The agent half adds a registry, a text protocol, a bounded loop, company tools, document retrieval and a terminal. Parse model text as untrusted input. Validate tool name and arguments before requesting approval; record denied and failed calls as observations. Retain separate execution state and display text so that a friendly sentence cannot hide an unexecuted action. The course’s confirmation mode checks non-read-only calls; production systems also need resource scope and data-disclosure policies because reading can expose secrets.
Finally, add compaction, persistent memory, user steering and guardrails. Compaction is a lossy transformation: preserve the user’s constraints, outstanding actions and evidence pointers, not just a fluent story. Memory must be scoped by the configured directory; a separate user’s store must not leak into the session. Keep a ledger of actual actions independently of model-written summaries. The assignment’s terminal restrictions are not equivalent to a production security boundary: inspect sandbox.jail(), especially on Windows where the documented result is none.
HW2: design a task that measures the right capability
Harbor coordinates the evaluation; Docker supplies the task environment; mini-SWE-agent is the system being evaluated. Keep these responsibilities separate. The task package describes the request, initial filesystem and dependencies, time/resource limits, verification and an oracle solution. An oracle is a known solution used to validate the task; it must not be available to the evaluated agent. A model’s final claim of success is not the reward: the verifier examines the resulting state.
The handout moves from a local HumanEval task to SWE-bench Verified and Terminal-Bench 2.1, then asks you to author a task. The required model in version 26.1.0 is Gemini 3.8 Flash, with five independent attempts using the supplied pinned_agent.py wrapper. Preserve the repository’s dependency pins. Its Python constraint is >=3.12,<3.14 and Harbor is pinned to 0.18.0; these are course snapshot requirements, not a recommendation to use those versions for every new project.
A useful task has a clear capability claim. For example, “restore a service configuration without altering unrelated services” tests diagnosis and constrained repair. A verifier that checks only whether one file exists is too weak; one that requires your exact patch rejects valid alternatives. Test postconditions, invariants and prohibited side effects. Do not hide the answer in the starting image, leak test fixtures to the agent, depend on an unpinned remote service or make difficulty come from a broken installation.
The four parts are basics 5%, running examples 10%, authoring 70% and peer review 15%. Within authoring: reproducibility 10, capability 15, task–verifier fidelity 15, difficulty 20 and report 10. The handout asks for three peer-task reviews, while the general logistics page describes two peers per review activity. Treat the specific HW2 handout as the task-specific instruction, and ask course staff if enrolled; do not silently merge the two requirements.
The mechanisms connecting the assignments
Consider an employee asking whether a reimbursement is allowed and then requesting an action. Retrieval supplies candidate evidence, not permission. The harness reads the relevant policy, binds the answer to a document and version, constructs a proposed tool call, validates it and checks authority before execution. A denied action must not reappear as a differently named tool call. A timeout creates uncertainty about whether a side effect occurred; inspect the operation state before retrying. This is an original production extension of the course’s teaching design, not a claim about its starter implementation.
In the reference architecture, the model cannot directly reach credentials or execute a shell. All action proposals cross a host-controlled policy gate. Read-only document access still respects tenant scope. Writes use approval and an operation identifier; returned observations go into the event log. Context is a selected view over that log, while persistent memory is a separate scoped store. A summary may describe a completed action only when execution evidence supports it.
A context budget is additive: system instructions, conversation, selected tool observations and reserved output must fit the model limit. With a synthetic 16,000-token window, 2,000 system tokens, 4,000 dialogue tokens and 2,000 output reserve leave 8,000 for evidence. Four characters per token is the course wrapper’s approximation, not a tokenizer identity or a multilingual guarantee. A production budget should use the actual model tokenizer and include provider overhead.
Reliability compounds across required steps. Under the strong simplifying assumption of independent 95%-reliable steps, ten necessary steps succeed together only about 59.9% of the time. Real failures may be correlated, so this is not a benchmark prediction. It explains why shorter action paths, recovery and explicit termination matter. More tools increase possibility, not automatically reliability.
Environment, model versions and a realistic budget
HW1 needs Python 3.11+ and uv; Windows additionally uses Git for Windows for Bash, or WSL. HW2 requires Docker with a reachable daemon, uv and Python 3.12–3.13 according to its project file. On Apple Silicon the SWE-bench example requires the documented linux/amd64 platform setting. This may involve emulation and different runtime costs. Use separate project environments; do not upgrade a shared environment and assume the locked assignments remain reproducible.
The inspected HW1 README identifies openai/gpt-6-luna as the grading model and openai/gpt-6-sol as judge; HW2 names Gemini 3.8 Flash through Gemini or OpenRouter. These are identifiers in the inspected course files, not a promise that every external account has access or the same price. External learners supply their own provider credentials and credits. We did not provision a key or make paid calls. Token Lab itself does not collect assignment keys or send readers’ exercise inputs to model providers.
The HW1 handout reports $0.05 for one reference-solution pass over live tests excluding the final evaluation, then $0.18 for that evaluation. This is $0.23 for that reported run, not a completion guarantee. Debugging, prompt changes, model substitutions and retries create new paid requests. Cache identity includes model, sampling settings and exact messages. A cache hit is not an independent repeat for reliability measurement.
Set both a provider-side spending limit and CS329Z_BUDGET_USD. The local ledger guard stops new calls after its recorded total reaches the threshold; do not treat it as a transactionally exact cap across concurrent processes. Deleting .lm_cache removes cached replies and local accounting. Record run identifiers, model, configuration, token counts and cost before cleanup. A native terminal or Docker exercise is local execution, but live model tests still transmit prompts to an external API and incur charges.
Course deliverables, project and responsible reuse
The course-wide weights are project 40%, homework 15%, homework quizzes 15%, paper video 10% and peer review 20%. Project components are proposal 5%, midway report 5%, midpoint demo 5%, final submission 15% and final system demo 10%. These are Stanford enrollment requirements; an independent reader can use the same milestones without claiming enrollment or credit. We did not find a complete public final-project rubric in the inspected pages, so the project plan below is our suggested study structure, not an official substitute.
A useful independent project is an SRE incident assistant: ingest a synthetic incident snapshot, retrieve a runbook, return an evidence-linked diagnosis and propose a bounded action requiring approval. First deliver a read-only baseline. Then add persistence and failure handling. Finally compare the improved system with the baseline on a frozen set of incidents, including missing evidence and a denied action. Produce source, dependency lock, configuration, architecture, traces, cost report and a clear list of unsupported operations.
The official paper-video handout requests 10–15 minutes, a paper explanation, critique and an added contribution such as a code experiment. Its rubric is 20+20+20+10 points. Submitters supply an unlisted YouTube link and their materials. Its AI rules are stricter than general coding assistance: slides, script, analysis and narration must be the student’s own; AI may support understanding and feedback but must not make the slides. Course project reports also cannot be written by AI. Homework code assistance is permitted only with understanding and validation; oral quizzes check that understanding.
The two assignment repositories carry MIT licenses; preserve copyright and license notices when reusing code or substantial documentation. That does not automatically license Stanford lecture videos, slides or third-party readings. This guide links those resources rather than mirroring them. The public repositories contain starters, not an invitation to publish classmates’ solutions. Enrolled students should follow the private-submission and academic-integrity rules.
Curriculum and learning sequence
Slides / videos / code / readings are listed below in course order. A missing video link means no public video was linked in the checked timetable; it does not prove a recording does not exist. Reading titles retain their original published language.
- Agent systems Materials (5)
- LLMs for builders Materials (5)
- RAG Materials (4)
- Tool use Materials (4)
- Frameworks Materials (5)
- Design patterns Materials (6)
- Memory Materials (4)
- Multi-agent systems Materials (4)
- Optimization Materials (4)
- Terminal-Bench Materials (3)
- Agent data Materials (3)
- Data quality Materials (4)
- Evaluation and benchmarks Materials (4)
- Andrew Ng conversation Materials (0)
- LLM judges and evaluation infrastructure Materials (4)
- SWE-bench Materials (1)
- Safety and guardrails Materials (6)
- No class Materials (0)
- No class Materials (0)
- Coding agents Materials (4)
- Proactive agents Materials (3)
- Final demo Materials (0)
Timeline and deliverables
| Date (2026, PT) | Topic |
|---|---|
| 10-09 23:59 | Project proposal |
| 10-30 | Assignment 1 |
| 11-06 | Midpoint report + demo |
| 11-13 23:59 | Paper video |
| 11-23 23:59 | Assignment 2 task |
| 11-30 | Paper peer reviews (2) |
| 12-02 | Assignment 2 peer reviews (3) |
| 12-09 15:30–18:30 | Final demo |
| 12-11 23:59 | Final report |
Practice, validation and self-study plan
Use four milestones rather than watching everything passively. Week 1: explain the model/message contract and implement deterministic workflow and BM25 tests. Week 2: implement the bounded harness, then document permissions, summaries and memory decisions. Week 3: package a task with an oracle and adversarial verifier checks. Week 4: run five controlled attempts, analyse failures and write a report. This is an original self-study schedule, not a Stanford deadline.
Assignment 1: grading map
| Module | Deterministic | Live API | Total |
|---|---|---|---|
| priority | 1 | 2 | 3 |
| daily_digest | 1 | 3 | 4 |
| bm25 | 8 | 0 | 8 |
| email_qa | 2 | 6 | 8 |
| tool_registry | 3 | 0 | 3 |
| tool_protocol | 2 | 2 | 4 |
| agent_loop | 8 | 0 | 8 |
| cardinal | 2 | 0 | 2 |
| search_docs | 2 | 3 | 5 |
| terminal | 2 | 4 | 6 |
| compaction | 5 | 3 | 8 |
| memory | 3 | 5 | 8 |
| interactive | 7 | 0 | 7 |
| guardrails | 2 | 2 | 4 |
| sim_eval | 0 | 10 | 10 |
Autograded: 48 + 40 = 88. Field notes: 10. Design memo: 20. Assignment total: 118; the 88 is not the whole assignment.
Run deterministic tests first. The live and simulation commands below send data to external providers and may incur charges; they are optional coursework commands, not browser-local experiments. Put your own credentials in .env, never in source control. Token Lab has not run these paid tests or completed the assignments for you.
git clone https://github.com/cs329z/assignment1-harness.git
cd assignment1-harness
uv sync
uv run python data/download.py
uv run pytestCS329Z_EVAL_SLICE=5 uv run pytest -m live -s
uv run python -m cs329z_hw1.simulation --slice 3
uv run python -m cs329z_hw1.chat --mode confirmgit clone https://github.com/cs329z/assignment2-evals.git
cd assignment2-evals
uv sync
uv run python check_submission.py
Assignment 2: setup 5%, example runs 10%, authored task 70%, peer review 15%. The 70% consists of reproducibility 10, capability 15, fidelity 15, difficulty 20 and report 10. Read the official handout for the pinned agent command and provider setup; do not substitute an arbitrary Harbor agent.
uv run harbor run --path tasks/YOUR_TASK_ID --agent-import-path pinned_agent:MiniSweAgent --model openrouter/google/gemini-3.8-flash --env-file .env --n-attempts 5 --n-concurrent 1 --job-name YOUR_TASK_ID-gemini --jobs-dir tasks/YOUR_TASK_ID/artifacts --yesOriginal local exercise: five attempts are not certainty
The standard-library program below prints a 95% Wilson interval for three successes in five independent Bernoulli trials, then checks empty, all-failure and all-success cases. Expected interval: 0.2307–0.8824. This measures sampling uncertainty, not task validity, and assumes comparable independent trials. Cached repetitions do not supply independent evidence.
Local exercise download"""Token Lab original supplement. Python 3.11+, stdlib, no network or randomness."""
from math import sqrt
def wilson(successes, trials, z=1.959963984540054):
if not isinstance(trials, int) or not isinstance(successes, int) or trials <= 0 or not 0 <= successes <= trials:
raise ValueError("Require 0 <= successes <= positive integer trials")
p = successes / trials
denominator = 1 + z*z/trials
center = (p + z*z/(2*trials)) / denominator
half = z * sqrt(p*(1-p)/trials + z*z/(4*trials*trials)) / denominator
return max(0.0, center-half), min(1.0, center+half)
if __name__ == "__main__":
lo, hi = wilson(3, 5)
assert abs(lo - 0.2307242813) < 1e-9
assert abs(hi - 0.8823792258) < 1e-9
assert wilson(0, 5)[0] < 1e-12
assert abs(wilson(5, 5)[1] - 1) < 1e-12
for args in [(0, 0), (6, 5), (-1, 5)]:
try:
wilson(*args)
except ValueError:
pass
else:
raise AssertionError("invalid input accepted")
print(f"3/5: 95% Wilson interval [{lo:.4f}, {hi:.4f}]")
print("PASS: boundaries and invalid inputs")
python3 uncertainty.py
# 3/5: 95% Wilson interval [0.2307, 0.8824]
# PASS: boundaries and invalid inputsOriginal interview practice—not company interview reports
1. Why is valid tool-call syntax not authorization?
Hint: separate parsing from policy. Answer: parse against a schema, then check identity, scope and approval in host code; a model-provided
2. Does five cached successes prove reliability?
Hint: count independent draws. Answer: cached replay tests plumbing, not fresh model variability. Use independent controlled runs, retain seed/settings and report counts and uncertainty.
3. How can a task pass while failing its purpose?
Hint: verifier coverage. Answer: a weak check may accept a hard-coded string or edited answer file. Test trivial bypasses, isolate hidden checks, verify resulting state, and ensure the oracle passes. Difficulty without validity is not useful.
Score each answer: 1 names a concept; 2 defines it; 3 explains a concrete mechanism; 4 handles failure and alternatives; 5 designs a fair test and states evidence limits.
Official materials, videos and further reading
- Official coursehttps://cs329z.stanford.edu/
- Course logisticshttps://cs329z.stanford.edu/logistics.html
- Assignment 1 · GitHubhttps://github.com/cs329z/assignment1-harness · PDF v26.1.0https://github.com/cs329z/assignment1-harness/blob/main/cs329z_assignment1_harness.pdf
- Assignment 2 · GitHubhttps://github.com/cs329z/assignment2-evals · PDF v26.1.0https://github.com/cs329z/assignment2-evals/blob/main/cs329z_assignment2_evaluation.pdf
- Paper video assignment · PDFhttps://web.stanford.edu/class/cs329z/assignments/paper_video.pdf
Slides / videos / code / readings are listed below in course order. A missing video link means no public video was linked in the checked timetable; it does not prove a recording does not exist. Reading titles retain their original published language.
1. Agent systems
Wed Sep 23 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture01.pdf
- Videohttps://youtu.be/DGI4vucR71w
- Zaharia et al. "The Shift from Models to Compound AI Systems." BAIR Blog (2024).https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/
- Weng. "LLM Powered Autonomous Agents." Lil'Log (2023).https://lilianweng.github.io/posts/2023-06-23-agent/
- Kapoor et al. "AI Agents That Matter." TMLR (2025).https://arxiv.org/abs/2407.01502
2. LLMs for builders
Mon Sep 28 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture02.pdf
- Videohttps://youtu.be/Vr1NzH8bT44
- Vaswani et al. "Attention Is All You Need." NeurIPS (2017).https://arxiv.org/abs/1706.03762
- Katharopoulos et al. "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention." ICML (2020).https://arxiv.org/abs/2006.16236
- Rajasekaran et al. "Effective Context Engineering for AI Agents." Anthropic (2025).https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
3. RAG
Wed Sep 30 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture03.pdf
- Lewis et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS (2020).https://arxiv.org/abs/2005.11401
- Karpukhin et al. "Dense Passage Retrieval for Open-Domain Question Answering." EMNLP (2020).https://arxiv.org/abs/2004.04906
- Khattab & Zaharia. "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT." SIGIR (2020).https://arxiv.org/abs/2004.12832
4. Tool use
Mon Oct 5 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture04.pdf
- Schick et al. "Toolformer: Language Models Can Teach Themselves to Use Tools." NeurIPS (2023).https://arxiv.org/abs/2302.04761
- Patil et al. "Gorilla: Large Language Model Connected with Massive APIs." NeurIPS (2024).https://arxiv.org/abs/2305.15334
- Model Context Protocol Specification, revision 2025-06-18. modelcontextprotocol.io (2025).https://modelcontextprotocol.io/specification/2025-06-18
5. Frameworks
Wed Oct 7 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture05.pdf
- Codehttps://github.com/cs329z/lecture5-frameworks
- Khattab et al. "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines." ICLR (2024).https://arxiv.org/abs/2310.03714
- LangChain. "LangGraph Overview." LangChain Docs.https://docs.langchain.com/oss/python/langgraph/overview
- Arena.ai. "HarnessTax: How Much Does the Harness Matter for Coding Agents?" Arena Blog (2026).https://arena.ai/blog/coding-agents-harness-tax
6. Design patterns
Mon Oct 12 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture06.pdf
- Yao et al. "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR (2023).https://arxiv.org/abs/2210.03629
- Schluntz & Zhang. "Building Effective AI Agents." Anthropic (2024).https://www.anthropic.com/engineering/building-effective-agents
- LangChain. "Context Engineering." LangChain Blog (2025).https://www.langchain.com/blog/context-engineering-for-agents
- Shinn et al. "Reflexion: Language Agents with Verbal Reinforcement Learning." NeurIPS (2023).https://arxiv.org/abs/2303.11366
- Wang et al. "Executable Code Actions Elicit Better LLM Agents." ICML (2024).https://arxiv.org/abs/2402.01030
7. Memory
Wed Oct 14 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture07.pdf
- Sumers et al. "Cognitive Architectures for Language Agents." TMLR (2024).https://arxiv.org/abs/2309.02427
- Packer et al. "MemGPT: Towards LLMs as Operating Systems." arXiv (2023).https://arxiv.org/abs/2310.08560
- Park et al. "Generative Agents: Interactive Simulacra of Human Behavior." UIST (2023).https://arxiv.org/abs/2304.03442
8. Multi-agent systems
Mon Oct 19 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture08.pdf
- Wu et al. "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation." COLM (2024).https://arxiv.org/abs/2308.08155
- Cemri et al. "Why Do Multi-Agent LLM Systems Fail?" NeurIPS Datasets & Benchmarks (2025).https://arxiv.org/abs/2503.13657
- Kim et al. "Towards a Science of Scaling Agent Systems." arXiv (2025).https://arxiv.org/abs/2512.08296
9. Optimization
Wed Oct 21 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture09.pdf
- Agrawal et al. "GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning." ICLR (2026).https://arxiv.org/abs/2507.19457
- Opsahl-Ong et al. "Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs." EMNLP (2024).https://arxiv.org/abs/2406.11695
- Shao et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv (2024).https://arxiv.org/abs/2402.03300
10. Terminal-Bench
Mon Oct 26 · 2026
11. Agent data
Wed Oct 28 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture11.pdf
- Shankar. "Data Flywheels for LLM Applications." (2024).https://www.sh-reya.com/blog/ai-engineering-flywheel/
- Tan et al. "Large Language Models for Data Annotation and Synthesis: A Survey." EMNLP (2024).https://aclanthology.org/2024.emnlp-main.54/
12. Data quality
Mon Nov 2 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture12.pdf
- Yang et al. "SWE-smith: Scaling Data for Software Engineering Agents." NeurIPS D&B (2025).https://arxiv.org/abs/2504.21798
- Shankar et al. "Who Validates the Validators? Aligning LLM-Assisted Evaluation with Human Preferences." UIST (2024).https://arxiv.org/abs/2404.12272
- Zhou et al. "LIMA: Less Is More for Alignment." NeurIPS (2023).https://arxiv.org/abs/2305.11206
13. Evaluation and benchmarks
Wed Nov 4 · 2026
- Slideshttps://web.stanford.edu/class/cs329z/slides/lecture13.pdf
- Zhu et al. "Establishing Best Practices for Building Rigorous Agentic Benchmarks." (2025).https://arxiv.org/abs/2507.02825
- Press. "How to Build Good Language Modeling Benchmarks." (2024).https://ofir.io/How-to-Build-Good-Language-Modeling-Benchmarks/
- Polo et al. "tinyBenchmarks: Evaluating LLMs with Fewer Examples." ICML (2024).https://arxiv.org/abs/2402.14992
14. Andrew Ng conversation
Mon Nov 9 · 2026
15. LLM judges and evaluation infrastructure
Wed Nov 11 · 2026
- Grace et al. "Demystifying Evals for AI Agents." Anthropic (2026).https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- Zheng et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS (2023).https://arxiv.org/abs/2306.05685
- Ryan et al. "AutoMetrics: Approximate Human Judgements with Automatically Generated Evaluators." ICLR (2026).https://arxiv.org/abs/2512.17267
- Zhu et al. "AutoLibra: Agent Metric Induction from Open-Ended Human Feedback." ICLR (2026).https://openreview.net/forum?id=4BjGVZ7Bxn
16. SWE-bench
Mon Nov 16 · 2026
17. Safety and guardrails
Wed Nov 18 · 2026
- Shao et al. "PrivacyLens: Evaluating Privacy Norm Awareness of LMs in Action." NeurIPS D&B (2024).https://arxiv.org/abs/2409.00138
- Zhang & Yang. "Searching for Privacy Risks in LLM Agents via Simulation." arXiv (2025).https://arxiv.org/abs/2508.10880
- Li. "Agentic LLMs as Powerful Deanonymizers." arXiv (2026).https://arxiv.org/abs/2601.05918
- Wen et al. "Contextualized Privacy Defense for LLM Agents." arXiv (2026).https://arxiv.org/abs/2603.02983
- OpenAI. "Understanding Prompt Injections: A Frontier Security Challenge." (2025).https://openai.com/index/prompt-injections/
- Anthropic. "Anthropic's Responsible Scaling Policy." (2023).https://www.anthropic.com/news/anthropics-responsible-scaling-policy
18. No class
Mon Nov 23 · 2026
19. No class
Wed Nov 25 · 2026
20. Coding agents
Mon Nov 30 · 2026
- Yang et al. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." NeurIPS (2024).https://arxiv.org/abs/2405.15793
- Wang et al. "OpenHands: An Open Platform for AI Software Developers." ICLR (2025).https://arxiv.org/abs/2407.16741
- Anthropic. "Claude Code: Best Practices for Agentic Coding." (2025).https://www.anthropic.com/engineering/claude-code-best-practices
- Young. "Effective Harnesses for Long-Running Agents." Anthropic (2025).https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
21. Proactive agents
Wed Dec 2 · 2026
- Shaikh et al. "Creating General User Models from Computer Use." UIST (2025).https://arxiv.org/abs/2505.10831
- Shaikh et al. "Learning Next Action Predictors from Human-Computer Interaction." arXiv (2026).https://arxiv.org/abs/2603.05923
- Steinberger. "OpenClaw: Open-source proactive AI agent." (2025).https://github.com/openclaw/openclaw
22. Final demo
Wed Dec 9 · 2026