Token Lab검색

← 모든 실험
500 / ORIGINAL PRACTICE PROMPTS

면접 연습

시나리오 기반의 독창적인 연습 문제 500개입니다. 기업 면접 기출이 아닙니다. 방법론 자료와 재현 가능한 로컬 계산을 사용하며 실제 모델 벤치마크를 주장하지 않습니다.

시나리오당 10문항입니다. 문제 형식을 재사용하며 500개의 독립적인 면접 기록이 아닙니다.

기술 설명, 매개변수 이름과 참고 답변은 현재 영어입니다.

연구직 면접과 연습

여섯 가지 심화 실험으로 목적 함수를 유도하고 구현을 검증하며 실패와 결론을 분석합니다.

PhD 연구 실험실 →
PUBLIC REPORTS / 06 STUDY CARDS

OpenAI & Anthropic · reported interviews

Six study cards based on public reports: three specific OpenAI question topics and three Anthropic assessment formats. Candidate accounts are not company-confirmed questions. Exercises, test cases and solution outlines below are original Token Lab material, not verbatim interview prompts or official answers.

Sources checked: 2026-10-25. OpenAI source: anonymous account hosted by Aced; exact publication date unavailable (page shows relative dates). Anthropic source: Andrey Goncharov, 2025-02-12, updated September 2025. These are individual experiences, not representative hiring statistics.

Technical study cards are currently available in English and Chinese. This locale displays English.

OpenAI

OpenAI / 1 / CANDIDATE REPORT

GPU credit ledger

A backend candidate reports a coding round on managing GPU credits.

Read source report ↗
Original exercise & acceptance cases

Original variant: implement grant(id, amount, start, end), consume(amount, time), and balance(time). Use [start,end), consume earliest-expiring credits first, reject insufficient funds atomically, and accept events in chronological order.

Acceptance cases: Grant A=10 on [0,10), B=5 on [3,6). Consume 7 at t=4: B becomes 0, A becomes 8. Balance at t=6 is 8; at t=10 is 0. Test duplicate IDs and rejected withdrawals.

Solution outline & self-assessment

Start with a sorted grant list and a reference implementation. State interval semantics before choosing a heap. A rejected debit must leave every grant unchanged. Extension: out-of-order events require explicit replay or versioning semantics.

Score 0–2 on each: clear assumptions, correct reasoning, edge-case coverage, trade-off discussion. Total /8 is a personal study rubric, not a company hiring score.

OpenAI / 2 / CANDIDATE REPORT

Multidimensional and async iterators

The same candidate describes an iterator task extended to 2D, 3D and asynchronous traversal.

Read source report ↗
Original exercise & acceptance cases

Original variant: traverse [[],[1,2],[],[3]] lazily, then add a serializable checkpoint. Restore after yielding 2. Extend to async sources with cancellation.

Acceptance cases: Expected sequence: 1,2,3. Restored suffix: 3. Cover empty inputs, repeated exhaustion, stale checkpoints, and cancellation during an awaited read.

Solution outline & self-assessment

Keep explicit cursor state; normalize across empty rows without flattening the entire input. Bind checkpoints to an input version. Explain whether an async read advances the cursor before or after success.

Score 0–2 on each: clear assumptions, correct reasoning, edge-case coverage, trade-off discussion. Total /8 is a personal study rubric, not a company hiring score.

OpenAI / 3 / CANDIDATE REPORT

Distributed job scheduler

The candidate reports a system-design round about a job scheduler, including scaling and failures.

Read source report ↗
Original exercise & acceptance cases

Original variant: design a scheduler for 10,000 queued experiment jobs and 100 workers. Add tenant quotas, retries, cancellation, and experiment artifacts. Define delivery guarantees before drawing components.

Acceptance cases: Inject a worker crash after writing output but before acknowledgement. Show how a duplicate attempt avoids publishing duplicate artifacts. Measure queue wait p95 and starvation by tenant.

Solution outline & self-assessment

Use durable job state, leases with fencing tokens, idempotent output commits, and bounded retries. Separate scheduling fairness from execution. Exactly-once effects require cooperation from the output store.

Score 0–2 on each: clear assumptions, correct reasoning, edge-case coverage, trade-off discussion. Total /8 is a personal study rubric, not a company hiring score.

Anthropic

Anthropic / 1 / CANDIDATE REPORT

Progressive API implementation

A Research Fellowship applicant describes a four-stage, test-unlocked coding assessment. The concrete API specification is not disclosed.

Read source report ↗
Original exercise & acceptance cases

Original practice, not the undisclosed question: build a key-value store in four stages: get/set; prefix scan; TTL; snapshot restore. Keep earlier behavior working at every stage.

Acceptance cases: Define expiry at t >= expiresAt. Test missing keys, overwrite, sorted scans, and restoring a snapshot at a later time. Decide whether restore preserves absolute expiry or remaining TTL.

Solution outline & self-assessment

Separate the clock from storage; inject a fake clock for deterministic tests. Specify restore semantics explicitly and preserve a regression suite while changing the internal representation.

Score 0–2 on each: clear assumptions, correct reasoning, edge-case coverage, trade-off discussion. Total /8 is a personal study rubric, not a company hiring score.

Anthropic / 2 / CANDIDATE REPORT

Research brainstorm

The applicant reports a short brainstorm with two open-ended questions about black-box LLM behavior; exact questions were not published.

Read source report ↗
Original exercise & acceptance cases

Original practice: a model gives different answers when irrelevant context is added. Propose three competing explanations and a 200-query experiment that distinguishes them.

Acceptance cases: Specify a falsifiable prediction per hypothesis, randomized order, repeated items, a held-out set, and a stopping rule. Explain what would change your mind.

Solution outline & self-assessment

Distinguish position effects, semantic interference, and sampling noise. Hold one factor fixed while varying another. Report effect sizes and uncertainty; a null result is informative only relative to power.

Score 0–2 on each: clear assumptions, correct reasoning, edge-case coverage, trade-off discussion. Total /8 is a personal study rubric, not a company hiring score.

Anthropic / 3 / CANDIDATE REPORT

Black-box evaluation proposal

The applicant describes receiving a notebook for API-based black-box investigation after the take-home round was cancelled. They did not complete that round.

Read source report ↗
Original exercise & acceptance cases

Original practice: test whether a judge favors verbose answers. Prepare 30 matched answer pairs with the same factual content, randomize presentation order, and preregister the analysis. Use synthetic labels first.

Acceptance cases: Keep correctness and verbosity separate. Include swapped-order trials, disagreement review, confidence intervals, and a cost ceiling. Preserve prompts, model version, raw outputs and exclusions.

Solution outline & self-assessment

Estimate paired preference differences and cluster uncertainty by item. Distinguish model preference from human quality. A synthetic dry run validates plumbing; it supplies no evidence about a real model.

Score 0–2 on each: clear assumptions, correct reasoning, edge-case coverage, trade-off discussion. Total /8 is a personal study rubric, not a company hiring score.

Official preparation guidance

These official pages describe process and tool-use expectations, not a published question bank. Confirm assessment-specific rules with your recruiter.

OpenAI Interview Guide ↗Anthropic Candidate AI Guidance ↗Practice experiment design →

attention-1 / 01What does “Causal baseline” test?

Future positions must receive zero probability.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-1 / 02Which computation produces the result in “Causal baseline”?

softmax(scale × QKᵀ / √2 + mask)V

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-1 / 03What is the baseline configuration for “Causal baseline”?

{"scale":1,"causal":1}

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-1 / 04What numerical result should you check for “Causal baseline”?

{"Mean entropy (bits)":1.0510152779349418,"Last output":2.980121288333647}

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-1 / 05How would you run a controlled comparison for “Causal baseline”?

Keep the supplied fixture and other settings fixed; change scale. Inspect per-row normalization, entropy and the weighted value output.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-1 / 06What failure could invalidate a conclusion from “Causal baseline”?

A high attention weight does not establish causal feature importance.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-1 / 07Is “Causal baseline” a benchmark of a deployed language model?

No. One synthetic head, four positions, Q, K and V are fixed and printed in exported method metadata. This visualizes attention weights, not an explanation of model reasoning.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-1 / 08Which evidence should you report for “Causal baseline”?

Inspect per-row normalization, entropy and the weighted value output. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-1 / 09How can another researcher reproduce “Causal baseline”?

Open attention-1, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-1 / 10What is the next validation step after “Causal baseline”?

A high attention weight does not establish causal feature importance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-2 / 01What does “Bidirectional context” test?

Removing the mask exposes future keys.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-2 / 02Which computation produces the result in “Bidirectional context”?

softmax(scale × QKᵀ / √2 + mask)V

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-2 / 03What is the baseline configuration for “Bidirectional context”?

{"scale":1,"causal":0}

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-2 / 04What numerical result should you check for “Bidirectional context”?

{"Mean entropy (bits)":1.8471329825256673,"Last output":2.980121288333647}

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-2 / 05How would you run a controlled comparison for “Bidirectional context”?

Keep the supplied fixture and other settings fixed; change scale. Inspect per-row normalization, entropy and the weighted value output.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-2 / 06What failure could invalidate a conclusion from “Bidirectional context”?

A high attention weight does not establish causal feature importance.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-2 / 07Is “Bidirectional context” a benchmark of a deployed language model?

No. One synthetic head, four positions, Q, K and V are fixed and printed in exported method metadata. This visualizes attention weights, not an explanation of model reasoning.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-2 / 08Which evidence should you report for “Bidirectional context”?

Inspect per-row normalization, entropy and the weighted value output. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-2 / 09How can another researcher reproduce “Bidirectional context”?

Open attention-2, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-2 / 10What is the next validation step after “Bidirectional context”?

A high attention weight does not establish causal feature importance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-3 / 01What does “Sharp attention” test?

Larger logits concentrate probability on the largest visible score.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-3 / 02Which computation produces the result in “Sharp attention”?

softmax(scale × QKᵀ / √2 + mask)V

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-3 / 03What is the baseline configuration for “Sharp attention”?

{"scale":5,"causal":1}

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-3 / 04What numerical result should you check for “Sharp attention”?

{"Mean entropy (bits)":0.19672661245634565,"Last output":1.1271721043873806}

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-3 / 05How would you run a controlled comparison for “Sharp attention”?

Keep the supplied fixture and other settings fixed; change scale. Inspect per-row normalization, entropy and the weighted value output.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-3 / 06What failure could invalidate a conclusion from “Sharp attention”?

A high attention weight does not establish causal feature importance.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-3 / 07Is “Sharp attention” a benchmark of a deployed language model?

No. One synthetic head, four positions, Q, K and V are fixed and printed in exported method metadata. This visualizes attention weights, not an explanation of model reasoning.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-3 / 08Which evidence should you report for “Sharp attention”?

Inspect per-row normalization, entropy and the weighted value output. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-3 / 09How can another researcher reproduce “Sharp attention”?

Open attention-3, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-3 / 10What is the next validation step after “Sharp attention”?

A high attention weight does not establish causal feature importance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-4 / 01What does “Flat attention” test?

Equal visible logits yield uniform weights.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-4 / 02Which computation produces the result in “Flat attention”?

softmax(scale × QKᵀ / √2 + mask)V

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-4 / 03What is the baseline configuration for “Flat attention”?

{"scale":0,"causal":1}

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-4 / 04What numerical result should you check for “Flat attention”?

{"Mean entropy (bits)":1.146240625180289,"Last output":4}

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-4 / 05How would you run a controlled comparison for “Flat attention”?

Keep the supplied fixture and other settings fixed; change scale. Inspect per-row normalization, entropy and the weighted value output.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-4 / 06What failure could invalidate a conclusion from “Flat attention”?

A high attention weight does not establish causal feature importance.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-4 / 07Is “Flat attention” a benchmark of a deployed language model?

No. One synthetic head, four positions, Q, K and V are fixed and printed in exported method metadata. This visualizes attention weights, not an explanation of model reasoning.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-4 / 08Which evidence should you report for “Flat attention”?

Inspect per-row normalization, entropy and the weighted value output. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-4 / 09How can another researcher reproduce “Flat attention”?

Open attention-4, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-4 / 10What is the next validation step after “Flat attention”?

A high attention weight does not establish causal feature importance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-5 / 01What does “Soft attention” test?

A smaller scale moves weights toward uniform without changing visibility.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-5 / 02Which computation produces the result in “Soft attention”?

softmax(scale × QKᵀ / √2 + mask)V

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-5 / 03What is the baseline configuration for “Soft attention”?

{"scale":0.3,"causal":1}

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-5 / 04What numerical result should you check for “Soft attention”?

{"Mean entropy (bits)":1.1377371118107673,"Last output":3.700414589154465}

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-5 / 05How would you run a controlled comparison for “Soft attention”?

Keep the supplied fixture and other settings fixed; change scale. Inspect per-row normalization, entropy and the weighted value output.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-5 / 06What failure could invalidate a conclusion from “Soft attention”?

A high attention weight does not establish causal feature importance.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-5 / 07Is “Soft attention” a benchmark of a deployed language model?

No. One synthetic head, four positions, Q, K and V are fixed and printed in exported method metadata. This visualizes attention weights, not an explanation of model reasoning.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-5 / 08Which evidence should you report for “Soft attention”?

Inspect per-row normalization, entropy and the weighted value output. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-5 / 09How can another researcher reproduce “Soft attention”?

Open attention-5, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
attention-5 / 10What is the next validation step after “Soft attention”?

A high attention weight does not establish causal feature importance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Attention Is All You Need · §3.2 ↗

실험 열기 →
embedding-1 / 01What does “Cosine nearest neighbors” test?

A large document norm should not dominate cosine ranking.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-1 / 02Which computation produces the result in “Cosine nearest neighbors”?

·d/(||q|| ||d||);

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-1 / 03What is the baseline configuration for “Cosine nearest neighbors”?

{"x":1,"y":0.2,"metric":0}

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-1 / 04What numerical result should you check for “Cosine nearest neighbors”?

{"Top score":0.9999999999999998,"Query norm":1.019803902718557}

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-1 / 05How would you run a controlled comparison for “Cosine nearest neighbors”?

Keep the supplied fixture and other settings fixed; change x. Compare ranking order, vector norms and scores on identical documents.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-1 / 06What failure could invalidate a conclusion from “Cosine nearest neighbors”?

Geometry alone cannot establish semantic relevance; these vectors are hand-authored.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-1 / 07Is “Cosine nearest neighbors” a benchmark of a deployed language model?

No. Five manually chosen 2D vectors, not embeddings generated by a language model. Zero-vector cosine is defined as 0 in this lab. Ties preserve fixture order.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-1 / 08Which evidence should you report for “Cosine nearest neighbors”?

Compare ranking order, vector norms and scores on identical documents. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-1 / 09How can another researcher reproduce “Cosine nearest neighbors”?

Open embedding-1, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-1 / 10What is the next validation step after “Cosine nearest neighbors”?

Geometry alone cannot establish semantic relevance; these vectors are hand-authored. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-2 / 01What does “Dot-product norm bias” test?

The long vector may outrank a better-aligned short vector.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-2 / 02Which computation produces the result in “Dot-product norm bias”?

·d/(||q|| ||d||);

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-2 / 03What is the baseline configuration for “Dot-product norm bias”?

{"x":1,"y":0.2,"metric":1}

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-2 / 04What numerical result should you check for “Dot-product norm bias”?

{"Top score":5.2,"Query norm":1.019803902718557}

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-2 / 05How would you run a controlled comparison for “Dot-product norm bias”?

Keep the supplied fixture and other settings fixed; change x. Compare ranking order, vector norms and scores on identical documents.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-2 / 06What failure could invalidate a conclusion from “Dot-product norm bias”?

Geometry alone cannot establish semantic relevance; these vectors are hand-authored.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-2 / 07Is “Dot-product norm bias” a benchmark of a deployed language model?

No. Five manually chosen 2D vectors, not embeddings generated by a language model. Zero-vector cosine is defined as 0 in this lab. Ties preserve fixture order.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-2 / 08Which evidence should you report for “Dot-product norm bias”?

Compare ranking order, vector norms and scores on identical documents. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-2 / 09How can another researcher reproduce “Dot-product norm bias”?

Open embedding-2, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-2 / 10What is the next validation step after “Dot-product norm bias”?

Geometry alone cannot establish semantic relevance; these vectors are hand-authored. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-3 / 01What does “Query rotation” test?

Rotating the query changes which document aligns best.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-3 / 02Which computation produces the result in “Query rotation”?

·d/(||q|| ||d||);

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-3 / 03What is the baseline configuration for “Query rotation”?

{"x":0,"y":1,"metric":0}

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-3 / 04What numerical result should you check for “Query rotation”?

{"Top score":1,"Query norm":1}

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-3 / 05How would you run a controlled comparison for “Query rotation”?

Keep the supplied fixture and other settings fixed; change x. Compare ranking order, vector norms and scores on identical documents.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-3 / 06What failure could invalidate a conclusion from “Query rotation”?

Geometry alone cannot establish semantic relevance; these vectors are hand-authored.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-3 / 07Is “Query rotation” a benchmark of a deployed language model?

No. Five manually chosen 2D vectors, not embeddings generated by a language model. Zero-vector cosine is defined as 0 in this lab. Ties preserve fixture order.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-3 / 08Which evidence should you report for “Query rotation”?

Compare ranking order, vector norms and scores on identical documents. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-3 / 09How can another researcher reproduce “Query rotation”?

Open embedding-3, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-3 / 10What is the next validation step after “Query rotation”?

Geometry alone cannot establish semantic relevance; these vectors are hand-authored. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-4 / 01What does “Opposite direction” test?

Opposite vectors receive negative cosine scores.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-4 / 02Which computation produces the result in “Opposite direction”?

·d/(||q|| ||d||);

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-4 / 03What is the baseline configuration for “Opposite direction”?

{"x":-1,"y":0,"metric":0}

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-4 / 04What numerical result should you check for “Opposite direction”?

{"Top score":1,"Query norm":1}

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-4 / 05How would you run a controlled comparison for “Opposite direction”?

Keep the supplied fixture and other settings fixed; change x. Compare ranking order, vector norms and scores on identical documents.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-4 / 06What failure could invalidate a conclusion from “Opposite direction”?

Geometry alone cannot establish semantic relevance; these vectors are hand-authored.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-4 / 07Is “Opposite direction” a benchmark of a deployed language model?

No. Five manually chosen 2D vectors, not embeddings generated by a language model. Zero-vector cosine is defined as 0 in this lab. Ties preserve fixture order.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-4 / 08Which evidence should you report for “Opposite direction”?

Compare ranking order, vector norms and scores on identical documents. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-4 / 09How can another researcher reproduce “Opposite direction”?

Open embedding-4, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-4 / 10What is the next validation step after “Opposite direction”?

Geometry alone cannot establish semantic relevance; these vectors are hand-authored. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-5 / 01What does “Zero-vector edge case” test?

The lab returns zero scores for an undefined cosine denominator.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-5 / 02Which computation produces the result in “Zero-vector edge case”?

·d/(||q|| ||d||);

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-5 / 03What is the baseline configuration for “Zero-vector edge case”?

{"x":0,"y":0,"metric":0}

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-5 / 04What numerical result should you check for “Zero-vector edge case”?

{"Top score":0,"Query norm":0}

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-5 / 05How would you run a controlled comparison for “Zero-vector edge case”?

Keep the supplied fixture and other settings fixed; change x. Compare ranking order, vector norms and scores on identical documents.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-5 / 06What failure could invalidate a conclusion from “Zero-vector edge case”?

Geometry alone cannot establish semantic relevance; these vectors are hand-authored.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-5 / 07Is “Zero-vector edge case” a benchmark of a deployed language model?

No. Five manually chosen 2D vectors, not embeddings generated by a language model. Zero-vector cosine is defined as 0 in this lab. Ties preserve fixture order.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-5 / 08Which evidence should you report for “Zero-vector edge case”?

Compare ranking order, vector norms and scores on identical documents. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-5 / 09How can another researcher reproduce “Zero-vector edge case”?

Open embedding-5, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
embedding-5 / 10What is the next validation step after “Zero-vector edge case”?

Geometry alone cannot establish semantic relevance; these vectors are hand-authored. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Introduction to Information Retrieval · vector space scoring ↗

실험 열기 →
kv-1 / 01What does “GQA baseline” test?

Eight KV heads use one quarter the cache of 32 heads.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-1 / 02Which computation produces the result in “GQA baseline”?

KV ×layers×KV heads×head dimension×sequence×batch×bytes/value

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-1 / 03What is the baseline configuration for “GQA baseline”?

{"sequence":4096,"batch":1,"heads":8,"bytes":2,"block":16}

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-1 / 04What numerical result should you check for “GQA baseline”?

{"KV cache (GiB)":0.5,"Allocated KV (GiB)":0.5,"Padding tokens":0}

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-1 / 05How would you run a controlled comparison for “GQA baseline”?

Keep the supplied fixture and other settings fixed; change sequence. Compare logical KV GiB, allocated GiB and padding tokens.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-1 / 06What failure could invalidate a conclusion from “GQA baseline”?

An estimated cache footprint is not a measured GPU throughput or total memory requirement.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-1 / 07Is “GQA baseline” a benchmark of a deployed language model?

No. 32 layers, head dimension 128. Dense cache only; excludes model weights, activations, block tables and sharing. Per-sequence allocation rounds up to the next page.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-1 / 08Which evidence should you report for “GQA baseline”?

Compare logical KV GiB, allocated GiB and padding tokens. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-1 / 09How can another researcher reproduce “GQA baseline”?

Open kv-1, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-1 / 10What is the next validation step after “GQA baseline”?

An estimated cache footprint is not a measured GPU throughput or total memory requirement. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-2 / 01What does “Multi-head cache” test?

More KV heads scale cache bytes linearly.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-2 / 02Which computation produces the result in “Multi-head cache”?

KV ×layers×KV heads×head dimension×sequence×batch×bytes/value

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-2 / 03What is the baseline configuration for “Multi-head cache”?

{"sequence":4096,"batch":1,"heads":32,"bytes":2,"block":16}

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-2 / 04What numerical result should you check for “Multi-head cache”?

{"KV cache (GiB)":2,"Allocated KV (GiB)":2,"Padding tokens":0}

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-2 / 05How would you run a controlled comparison for “Multi-head cache”?

Keep the supplied fixture and other settings fixed; change sequence. Compare logical KV GiB, allocated GiB and padding tokens.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-2 / 06What failure could invalidate a conclusion from “Multi-head cache”?

An estimated cache footprint is not a measured GPU throughput or total memory requirement.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-2 / 07Is “Multi-head cache” a benchmark of a deployed language model?

No. 32 layers, head dimension 128. Dense cache only; excludes model weights, activations, block tables and sharing. Per-sequence allocation rounds up to the next page.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-2 / 08Which evidence should you report for “Multi-head cache”?

Compare logical KV GiB, allocated GiB and padding tokens. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-2 / 09How can another researcher reproduce “Multi-head cache”?

Open kv-2, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-2 / 10What is the next validation step after “Multi-head cache”?

An estimated cache footprint is not a measured GPU throughput or total memory requirement. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-3 / 01What does “Multi-query cache” test?

One KV head sharply reduces logical cache size.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-3 / 02Which computation produces the result in “Multi-query cache”?

KV ×layers×KV heads×head dimension×sequence×batch×bytes/value

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-3 / 03What is the baseline configuration for “Multi-query cache”?

{"sequence":4096,"batch":1,"heads":1,"bytes":2,"block":16}

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-3 / 04What numerical result should you check for “Multi-query cache”?

{"KV cache (GiB)":0.0625,"Allocated KV (GiB)":0.0625,"Padding tokens":0}

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-3 / 05How would you run a controlled comparison for “Multi-query cache”?

Keep the supplied fixture and other settings fixed; change sequence. Compare logical KV GiB, allocated GiB and padding tokens.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-3 / 06What failure could invalidate a conclusion from “Multi-query cache”?

An estimated cache footprint is not a measured GPU throughput or total memory requirement.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-3 / 07Is “Multi-query cache” a benchmark of a deployed language model?

No. 32 layers, head dimension 128. Dense cache only; excludes model weights, activations, block tables and sharing. Per-sequence allocation rounds up to the next page.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-3 / 08Which evidence should you report for “Multi-query cache”?

Compare logical KV GiB, allocated GiB and padding tokens. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-3 / 09How can another researcher reproduce “Multi-query cache”?

Open kv-3, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-3 / 10What is the next validation step after “Multi-query cache”?

An estimated cache footprint is not a measured GPU throughput or total memory requirement. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-4 / 01What does “Long-context batch” test?

Sequence and batch size multiply memory demand.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-4 / 02Which computation produces the result in “Long-context batch”?

KV ×layers×KV heads×head dimension×sequence×batch×bytes/value

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-4 / 03What is the baseline configuration for “Long-context batch”?

{"sequence":32768,"batch":8,"heads":8,"bytes":2,"block":16}

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-4 / 04What numerical result should you check for “Long-context batch”?

{"KV cache (GiB)":32,"Allocated KV (GiB)":32,"Padding tokens":0}

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-4 / 05How would you run a controlled comparison for “Long-context batch”?

Keep the supplied fixture and other settings fixed; change sequence. Compare logical KV GiB, allocated GiB and padding tokens.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-4 / 06What failure could invalidate a conclusion from “Long-context batch”?

An estimated cache footprint is not a measured GPU throughput or total memory requirement.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-4 / 07Is “Long-context batch” a benchmark of a deployed language model?

No. 32 layers, head dimension 128. Dense cache only; excludes model weights, activations, block tables and sharing. Per-sequence allocation rounds up to the next page.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-4 / 08Which evidence should you report for “Long-context batch”?

Compare logical KV GiB, allocated GiB and padding tokens. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-4 / 09How can another researcher reproduce “Long-context batch”?

Open kv-4, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-4 / 10What is the next validation step after “Long-context batch”?

An estimated cache footprint is not a measured GPU throughput or total memory requirement. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-5 / 01What does “Page fragmentation” test?

One extra token allocates an additional page per sequence.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-5 / 02Which computation produces the result in “Page fragmentation”?

KV ×layers×KV heads×head dimension×sequence×batch×bytes/value

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-5 / 03What is the baseline configuration for “Page fragmentation”?

{"sequence":4097,"batch":1,"heads":8,"bytes":2,"block":256}

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-5 / 04What numerical result should you check for “Page fragmentation”?

{"KV cache (GiB)":0.5001220703125,"Allocated KV (GiB)":0.53125,"Padding tokens":255}

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-5 / 05How would you run a controlled comparison for “Page fragmentation”?

Keep the supplied fixture and other settings fixed; change sequence. Compare logical KV GiB, allocated GiB and padding tokens.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-5 / 06What failure could invalidate a conclusion from “Page fragmentation”?

An estimated cache footprint is not a measured GPU throughput or total memory requirement.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-5 / 07Is “Page fragmentation” a benchmark of a deployed language model?

No. 32 layers, head dimension 128. Dense cache only; excludes model weights, activations, block tables and sharing. Per-sequence allocation rounds up to the next page.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-5 / 08Which evidence should you report for “Page fragmentation”?

Compare logical KV GiB, allocated GiB and padding tokens. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-5 / 09How can another researcher reproduce “Page fragmentation”?

Open kv-5, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
kv-5 / 10What is the next validation step after “Page fragmentation”?

An estimated cache footprint is not a measured GPU throughput or total memory requirement. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: GQA (2023); PagedAttention (2023) ↗

실험 열기 →
neural-1 / 01What does “Learn XOR quadrants” test?

A nonlinear hidden layer can fit this non-linearly separable task.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-1 / 02Which computation produces the result in “Learn XOR quadrants”?

; σ; log p+(1−y)log(1−p)]

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-1 / 03What is the baseline configuration for “Learn XOR quadrants”?

{"rate":0.3,"hidden":8,"epochs":1000,"task":0,"seed":42}

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-1 / 04What numerical result should you check for “Learn XOR quadrants”?

Run the seeded training lab and inspect the actual loss and accuracy; convergence is not assumed.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-1 / 05How would you run a controlled comparison for “Learn XOR quadrants”?

Keep the supplied fixture and other settings fixed; change rate. Watch train loss, held-out accuracy and decision boundary over epochs.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-1 / 06What failure could invalidate a conclusion from “Learn XOR quadrants”?

Training convergence on tiny synthetic data does not imply an LLM can train in the browser.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-1 / 07Is “Learn XOR quadrants” a benchmark of a deployed language model?

No. Actual full-batch gradient descent in JavaScript. 2 inputs → tanh hidden layer → sigmoid. Binary cross-entropy; 100 synthetic training and 100 held-out samples from the same generator. Held-out accuracy is not real-world generalization.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-1 / 08Which evidence should you report for “Learn XOR quadrants”?

Watch train loss, held-out accuracy and decision boundary over epochs. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-1 / 09How can another researcher reproduce “Learn XOR quadrants”?

Open neural-1, reset the preset, retain the exact inputs and run. Retain the seed, training epochs and all weights.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-1 / 10What is the next validation step after “Learn XOR quadrants”?

Training convergence on tiny synthetic data does not imply an LLM can train in the browser. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-2 / 01What does “Underpowered hidden layer” test?

One hidden unit limits the available decision boundaries.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-2 / 02Which computation produces the result in “Underpowered hidden layer”?

; σ; log p+(1−y)log(1−p)]

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-2 / 03What is the baseline configuration for “Underpowered hidden layer”?

{"rate":0.3,"hidden":1,"epochs":1000,"task":0,"seed":42}

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-2 / 04What numerical result should you check for “Underpowered hidden layer”?

Run the seeded training lab and inspect the actual loss and accuracy; convergence is not assumed.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-2 / 05How would you run a controlled comparison for “Underpowered hidden layer”?

Keep the supplied fixture and other settings fixed; change rate. Watch train loss, held-out accuracy and decision boundary over epochs.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-2 / 06What failure could invalidate a conclusion from “Underpowered hidden layer”?

Training convergence on tiny synthetic data does not imply an LLM can train in the browser.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-2 / 07Is “Underpowered hidden layer” a benchmark of a deployed language model?

No. Actual full-batch gradient descent in JavaScript. 2 inputs → tanh hidden layer → sigmoid. Binary cross-entropy; 100 synthetic training and 100 held-out samples from the same generator. Held-out accuracy is not real-world generalization.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-2 / 08Which evidence should you report for “Underpowered hidden layer”?

Watch train loss, held-out accuracy and decision boundary over epochs. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-2 / 09How can another researcher reproduce “Underpowered hidden layer”?

Open neural-2, reset the preset, retain the exact inputs and run. Retain the seed, training epochs and all weights.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-2 / 10What is the next validation step after “Underpowered hidden layer”?

Training convergence on tiny synthetic data does not imply an LLM can train in the browser. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-3 / 01What does “Slow learning” test?

A lower learning rate may need more epochs.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-3 / 02Which computation produces the result in “Slow learning”?

; σ; log p+(1−y)log(1−p)]

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-3 / 03What is the baseline configuration for “Slow learning”?

{"rate":0.01,"hidden":8,"epochs":1000,"task":0,"seed":42}

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-3 / 04What numerical result should you check for “Slow learning”?

Run the seeded training lab and inspect the actual loss and accuracy; convergence is not assumed.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-3 / 05How would you run a controlled comparison for “Slow learning”?

Keep the supplied fixture and other settings fixed; change rate. Watch train loss, held-out accuracy and decision boundary over epochs.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-3 / 06What failure could invalidate a conclusion from “Slow learning”?

Training convergence on tiny synthetic data does not imply an LLM can train in the browser.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-3 / 07Is “Slow learning” a benchmark of a deployed language model?

No. Actual full-batch gradient descent in JavaScript. 2 inputs → tanh hidden layer → sigmoid. Binary cross-entropy; 100 synthetic training and 100 held-out samples from the same generator. Held-out accuracy is not real-world generalization.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-3 / 08Which evidence should you report for “Slow learning”?

Watch train loss, held-out accuracy and decision boundary over epochs. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-3 / 09How can another researcher reproduce “Slow learning”?

Open neural-3, reset the preset, retain the exact inputs and run. Retain the seed, training epochs and all weights.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-3 / 10What is the next validation step after “Slow learning”?

Training convergence on tiny synthetic data does not imply an LLM can train in the browser. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-4 / 01What does “Circle classification” test?

The hidden layer can approximate a curved boundary.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-4 / 02Which computation produces the result in “Circle classification”?

; σ; log p+(1−y)log(1−p)]

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-4 / 03What is the baseline configuration for “Circle classification”?

{"rate":0.3,"hidden":16,"epochs":1000,"task":1,"seed":42}

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-4 / 04What numerical result should you check for “Circle classification”?

Run the seeded training lab and inspect the actual loss and accuracy; convergence is not assumed.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-4 / 05How would you run a controlled comparison for “Circle classification”?

Keep the supplied fixture and other settings fixed; change rate. Watch train loss, held-out accuracy and decision boundary over epochs.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-4 / 06What failure could invalidate a conclusion from “Circle classification”?

Training convergence on tiny synthetic data does not imply an LLM can train in the browser.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-4 / 07Is “Circle classification” a benchmark of a deployed language model?

No. Actual full-batch gradient descent in JavaScript. 2 inputs → tanh hidden layer → sigmoid. Binary cross-entropy; 100 synthetic training and 100 held-out samples from the same generator. Held-out accuracy is not real-world generalization.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-4 / 08Which evidence should you report for “Circle classification”?

Watch train loss, held-out accuracy and decision boundary over epochs. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-4 / 09How can another researcher reproduce “Circle classification”?

Open neural-4, reset the preset, retain the exact inputs and run. Retain the seed, training epochs and all weights.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-4 / 10What is the next validation step after “Circle classification”?

Training convergence on tiny synthetic data does not imply an LLM can train in the browser. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-5 / 01What does “Seed sensitivity” test?

Initialization and sampled data affect the learning trajectory.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-5 / 02Which computation produces the result in “Seed sensitivity”?

; σ; log p+(1−y)log(1−p)]

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-5 / 03What is the baseline configuration for “Seed sensitivity”?

{"rate":0.3,"hidden":8,"epochs":1000,"task":0,"seed":7}

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-5 / 04What numerical result should you check for “Seed sensitivity”?

Run the seeded training lab and inspect the actual loss and accuracy; convergence is not assumed.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-5 / 05How would you run a controlled comparison for “Seed sensitivity”?

Keep the supplied fixture and other settings fixed; change rate. Watch train loss, held-out accuracy and decision boundary over epochs.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-5 / 06What failure could invalidate a conclusion from “Seed sensitivity”?

Training convergence on tiny synthetic data does not imply an LLM can train in the browser.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-5 / 07Is “Seed sensitivity” a benchmark of a deployed language model?

No. Actual full-batch gradient descent in JavaScript. 2 inputs → tanh hidden layer → sigmoid. Binary cross-entropy; 100 synthetic training and 100 held-out samples from the same generator. Held-out accuracy is not real-world generalization.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-5 / 08Which evidence should you report for “Seed sensitivity”?

Watch train loss, held-out accuracy and decision boundary over epochs. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-5 / 09How can another researcher reproduce “Seed sensitivity”?

Open neural-5, reset the preset, retain the exact inputs and run. Retain the seed, training epochs and all weights.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
neural-5 / 10What is the next validation step after “Seed sensitivity”?

Training convergence on tiny synthetic data does not imply an LLM can train in the browser. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Dive into Deep Learning · multilayer perceptrons ↗

실험 열기 →
agent-1 / 01What does “Human approval gate” test?

A write pauses without the mock approval toggle.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-1 / 02Which computation produces the result in “Human approval gate”?

Plan → allowlist → approval → budget → mock tool → FIFO memory → next step

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-1 / 03What is the baseline configuration for “Human approval gate”?

{"plan":"search,calculator,recall,write","allowed":"search,calculator,recall,write","steps":6,"budget":0.1,"memory":3,"approval":0}

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-1 / 04What numerical result should you check for “Human approval gate”?

{"Executed / observed steps":4,"Simulated cost (USD)":0.03,"Memory entries":3}

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-1 / 05How would you run a controlled comparison for “Human approval gate”?

Keep the supplied fixture and other settings fixed; change steps. Inspect the trace, stopping reason, simulated spend and retained memory.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-1 / 06What failure could invalidate a conclusion from “Human approval gate”?

A tool allowlist alone cannot prevent data exfiltration or prompt injection in a real agent.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-1 / 07Is “Human approval gate” a benchmark of a deployed language model?

No. A deterministic state machine executes your plan against mock tools, not an autonomous language model. Each executed tool costs a synthetic $0.01. The write action never changes external data. Each mock result is truncated to 1,000 characters to bound memory. Guards are illustrative policy rules, not a complete security system.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-1 / 08Which evidence should you report for “Human approval gate”?

Inspect the trace, stopping reason, simulated spend and retained memory. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-1 / 09How can another researcher reproduce “Human approval gate”?

Open agent-1, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-1 / 10What is the next validation step after “Human approval gate”?

A tool allowlist alone cannot prevent data exfiltration or prompt injection in a real agent. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-2 / 01What does “Authorized mock write” test?

Approval permits only the local mock write.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-2 / 02Which computation produces the result in “Authorized mock write”?

Plan → allowlist → approval → budget → mock tool → FIFO memory → next step

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-2 / 03What is the baseline configuration for “Authorized mock write”?

{"plan":"search,calculator,recall,write","allowed":"search,calculator,recall,write","steps":6,"budget":0.1,"memory":3,"approval":1}

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-2 / 04What numerical result should you check for “Authorized mock write”?

{"Executed / observed steps":4,"Simulated cost (USD)":0.04,"Memory entries":3}

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-2 / 05How would you run a controlled comparison for “Authorized mock write”?

Keep the supplied fixture and other settings fixed; change steps. Inspect the trace, stopping reason, simulated spend and retained memory.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-2 / 06What failure could invalidate a conclusion from “Authorized mock write”?

A tool allowlist alone cannot prevent data exfiltration or prompt injection in a real agent.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-2 / 07Is “Authorized mock write” a benchmark of a deployed language model?

No. A deterministic state machine executes your plan against mock tools, not an autonomous language model. Each executed tool costs a synthetic $0.01. The write action never changes external data. Each mock result is truncated to 1,000 characters to bound memory. Guards are illustrative policy rules, not a complete security system.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-2 / 08Which evidence should you report for “Authorized mock write”?

Inspect the trace, stopping reason, simulated spend and retained memory. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-2 / 09How can another researcher reproduce “Authorized mock write”?

Open agent-2, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-2 / 10What is the next validation step after “Authorized mock write”?

A tool allowlist alone cannot prevent data exfiltration or prompt injection in a real agent. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-3 / 01What does “Tool allowlist” test?

A disallowed calculator call stops the workflow.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-3 / 02Which computation produces the result in “Tool allowlist”?

Plan → allowlist → approval → budget → mock tool → FIFO memory → next step

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-3 / 03What is the baseline configuration for “Tool allowlist”?

{"plan":"search,calculator,recall,write","allowed":"search,recall","steps":6,"budget":0.1,"memory":3,"approval":0}

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-3 / 04What numerical result should you check for “Tool allowlist”?

{"Executed / observed steps":2,"Simulated cost (USD)":0.01,"Memory entries":1}

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-3 / 05How would you run a controlled comparison for “Tool allowlist”?

Keep the supplied fixture and other settings fixed; change steps. Inspect the trace, stopping reason, simulated spend and retained memory.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-3 / 06What failure could invalidate a conclusion from “Tool allowlist”?

A tool allowlist alone cannot prevent data exfiltration or prompt injection in a real agent.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-3 / 07Is “Tool allowlist” a benchmark of a deployed language model?

No. A deterministic state machine executes your plan against mock tools, not an autonomous language model. Each executed tool costs a synthetic $0.01. The write action never changes external data. Each mock result is truncated to 1,000 characters to bound memory. Guards are illustrative policy rules, not a complete security system.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-3 / 08Which evidence should you report for “Tool allowlist”?

Inspect the trace, stopping reason, simulated spend and retained memory. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-3 / 09How can another researcher reproduce “Tool allowlist”?

Open agent-3, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-3 / 10What is the next validation step after “Tool allowlist”?

A tool allowlist alone cannot prevent data exfiltration or prompt injection in a real agent. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-4 / 01What does “Budget exhaustion” test?

The second paid action is blocked before exceeding budget.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-4 / 02Which computation produces the result in “Budget exhaustion”?

Plan → allowlist → approval → budget → mock tool → FIFO memory → next step

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-4 / 03What is the baseline configuration for “Budget exhaustion”?

{"plan":"search,calculator,recall,write","allowed":"search,calculator,recall,write","steps":6,"budget":0.01,"memory":3,"approval":0}

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-4 / 04What numerical result should you check for “Budget exhaustion”?

{"Executed / observed steps":1,"Simulated cost (USD)":0.01,"Memory entries":1}

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-4 / 05How would you run a controlled comparison for “Budget exhaustion”?

Keep the supplied fixture and other settings fixed; change steps. Inspect the trace, stopping reason, simulated spend and retained memory.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-4 / 06What failure could invalidate a conclusion from “Budget exhaustion”?

A tool allowlist alone cannot prevent data exfiltration or prompt injection in a real agent.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-4 / 07Is “Budget exhaustion” a benchmark of a deployed language model?

No. A deterministic state machine executes your plan against mock tools, not an autonomous language model. Each executed tool costs a synthetic $0.01. The write action never changes external data. Each mock result is truncated to 1,000 characters to bound memory. Guards are illustrative policy rules, not a complete security system.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-4 / 08Which evidence should you report for “Budget exhaustion”?

Inspect the trace, stopping reason, simulated spend and retained memory. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-4 / 09How can another researcher reproduce “Budget exhaustion”?

Open agent-4, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-4 / 10What is the next validation step after “Budget exhaustion”?

A tool allowlist alone cannot prevent data exfiltration or prompt injection in a real agent. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-5 / 01What does “Memory truncation” test?

Only the latest result survives FIFO truncation.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-5 / 02Which computation produces the result in “Memory truncation”?

Plan → allowlist → approval → budget → mock tool → FIFO memory → next step

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-5 / 03What is the baseline configuration for “Memory truncation”?

{"plan":"search,calculator,recall","allowed":"search,calculator,recall,write","steps":6,"budget":0.1,"memory":1,"approval":0}

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-5 / 04What numerical result should you check for “Memory truncation”?

{"Executed / observed steps":3,"Simulated cost (USD)":0.03,"Memory entries":1}

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-5 / 05How would you run a controlled comparison for “Memory truncation”?

Keep the supplied fixture and other settings fixed; change steps. Inspect the trace, stopping reason, simulated spend and retained memory.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-5 / 06What failure could invalidate a conclusion from “Memory truncation”?

A tool allowlist alone cannot prevent data exfiltration or prompt injection in a real agent.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-5 / 07Is “Memory truncation” a benchmark of a deployed language model?

No. A deterministic state machine executes your plan against mock tools, not an autonomous language model. Each executed tool costs a synthetic $0.01. The write action never changes external data. Each mock result is truncated to 1,000 characters to bound memory. Guards are illustrative policy rules, not a complete security system.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-5 / 08Which evidence should you report for “Memory truncation”?

Inspect the trace, stopping reason, simulated spend and retained memory. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-5 / 09How can another researcher reproduce “Memory truncation”?

Open agent-5, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
agent-5 / 10What is the next validation step after “Memory truncation”?

A tool allowlist alone cannot prevent data exfiltration or prompt injection in a real agent. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: ReAct · reasoning and tool actions ↗

실험 열기 →
retrieval-1 / 01What does “Hybrid fusion baseline” test?

Combining independent ranks can promote shared candidates.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-1 / 02Which computation produces the result in “Hybrid fusion baseline”?

;

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-1 / 03What is the baseline configuration for “Hybrid fusion baseline”?

{"query":"KV cache memory","documents":"[\n {\n \"id\": \"D1\",\n \"text\": \"KV cache stores keys and values for generation\",\n \"vector\": [\n 1,\n 0.1\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D2\",\n \"text\": \"Cache restaurant menu and opening hours\",\n \"vector\": [\n 0.1,\n 1\n ],\n \"relevance\": 0\n },\n {\n \"id\": \"D3\",\n \"text\": \"Grouped query attention reduces memory\",\n \"vector\": [\n 0.95,\n 0.2\n ],\n \"relevance\": 2\n },\n {\n \"id\": \"D4\",\n \"text\": \"KV cache memory footprint depends on sequence length\",\n \"vector\": [\n 0.8,\n 0.3\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D5\",\n \"text\": \"Tokenization splits input text\",\n \"vector\": [\n 0.4,\n 0.8\n ],\n \"relevance\": 0\n }\n]","x":1,"y":0.1,"weight":0.5,"k":60,"retrieve":5,"top":3,"rerank":0}

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-1 / 04What numerical result should you check for “Hybrid fusion baseline”?

{"Recall@k":1,"nDCG@k":1}

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-1 / 05How would you run a controlled comparison for “Hybrid fusion baseline”?

Keep the supplied fixture and other settings fixed; change weight. Compare Recall@k and nDCG@k while varying candidate pool and fusion weight.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-1 / 06What failure could invalidate a conclusion from “Hybrid fusion baseline”?

An oracle reranker leaks labels intentionally; never report it as learned model performance.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-1 / 07Is “Hybrid fusion baseline” a benchmark of a deployed language model?

No. Lexical score counts matching query terms (not BM25). Dense score is cosine on supplied 2D vectors, not a live embedding model. RRF combines ranks. Optional reranking uses known relevance labels as an oracle upper-bound, not a trained reranker. Empty query ties preserve document order.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-1 / 08Which evidence should you report for “Hybrid fusion baseline”?

Compare Recall@k and nDCG@k while varying candidate pool and fusion weight. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-1 / 09How can another researcher reproduce “Hybrid fusion baseline”?

Open retrieval-1, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-1 / 10What is the next validation step after “Hybrid fusion baseline”?

An oracle reranker leaks labels intentionally; never report it as learned model performance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-2 / 01What does “Lexical only” test?

Exact query words alone set the ranking.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-2 / 02Which computation produces the result in “Lexical only”?

;

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-2 / 03What is the baseline configuration for “Lexical only”?

{"query":"KV cache memory","documents":"[\n {\n \"id\": \"D1\",\n \"text\": \"KV cache stores keys and values for generation\",\n \"vector\": [\n 1,\n 0.1\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D2\",\n \"text\": \"Cache restaurant menu and opening hours\",\n \"vector\": [\n 0.1,\n 1\n ],\n \"relevance\": 0\n },\n {\n \"id\": \"D3\",\n \"text\": \"Grouped query attention reduces memory\",\n \"vector\": [\n 0.95,\n 0.2\n ],\n \"relevance\": 2\n },\n {\n \"id\": \"D4\",\n \"text\": \"KV cache memory footprint depends on sequence length\",\n \"vector\": [\n 0.8,\n 0.3\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D5\",\n \"text\": \"Tokenization splits input text\",\n \"vector\": [\n 0.4,\n 0.8\n ],\n \"relevance\": 0\n }\n]","x":1,"y":0.1,"weight":1,"k":60,"retrieve":5,"top":3,"rerank":0}

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-2 / 04What numerical result should you check for “Lexical only”?

{"Recall@k":0.6666666666666666,"nDCG@k":0.8838695436828513}

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-2 / 05How would you run a controlled comparison for “Lexical only”?

Keep the supplied fixture and other settings fixed; change weight. Compare Recall@k and nDCG@k while varying candidate pool and fusion weight.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-2 / 06What failure could invalidate a conclusion from “Lexical only”?

An oracle reranker leaks labels intentionally; never report it as learned model performance.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-2 / 07Is “Lexical only” a benchmark of a deployed language model?

No. Lexical score counts matching query terms (not BM25). Dense score is cosine on supplied 2D vectors, not a live embedding model. RRF combines ranks. Optional reranking uses known relevance labels as an oracle upper-bound, not a trained reranker. Empty query ties preserve document order.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-2 / 08Which evidence should you report for “Lexical only”?

Compare Recall@k and nDCG@k while varying candidate pool and fusion weight. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-2 / 09How can another researcher reproduce “Lexical only”?

Open retrieval-2, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-2 / 10What is the next validation step after “Lexical only”?

An oracle reranker leaks labels intentionally; never report it as learned model performance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-3 / 01What does “Dense only” test?

Only vector direction determines the ranking.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-3 / 02Which computation produces the result in “Dense only”?

;

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-3 / 03What is the baseline configuration for “Dense only”?

{"query":"KV cache memory","documents":"[\n {\n \"id\": \"D1\",\n \"text\": \"KV cache stores keys and values for generation\",\n \"vector\": [\n 1,\n 0.1\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D2\",\n \"text\": \"Cache restaurant menu and opening hours\",\n \"vector\": [\n 0.1,\n 1\n ],\n \"relevance\": 0\n },\n {\n \"id\": \"D3\",\n \"text\": \"Grouped query attention reduces memory\",\n \"vector\": [\n 0.95,\n 0.2\n ],\n \"relevance\": 2\n },\n {\n \"id\": \"D4\",\n \"text\": \"KV cache memory footprint depends on sequence length\",\n \"vector\": [\n 0.8,\n 0.3\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D5\",\n \"text\": \"Tokenization splits input text\",\n \"vector\": [\n 0.4,\n 0.8\n ],\n \"relevance\": 0\n }\n]","x":1,"y":0.1,"weight":0,"k":60,"retrieve":5,"top":3,"rerank":0}

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-3 / 04What numerical result should you check for “Dense only”?

{"Recall@k":1,"nDCG@k":0.9594535145926796}

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-3 / 05How would you run a controlled comparison for “Dense only”?

Keep the supplied fixture and other settings fixed; change weight. Compare Recall@k and nDCG@k while varying candidate pool and fusion weight.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-3 / 06What failure could invalidate a conclusion from “Dense only”?

An oracle reranker leaks labels intentionally; never report it as learned model performance.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-3 / 07Is “Dense only” a benchmark of a deployed language model?

No. Lexical score counts matching query terms (not BM25). Dense score is cosine on supplied 2D vectors, not a live embedding model. RRF combines ranks. Optional reranking uses known relevance labels as an oracle upper-bound, not a trained reranker. Empty query ties preserve document order.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-3 / 08Which evidence should you report for “Dense only”?

Compare Recall@k and nDCG@k while varying candidate pool and fusion weight. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-3 / 09How can another researcher reproduce “Dense only”?

Open retrieval-3, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-3 / 10What is the next validation step after “Dense only”?

An oracle reranker leaks labels intentionally; never report it as learned model performance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-4 / 01What does “Oracle reranking ceiling” test?

Perfect labels can improve ordering only inside the candidate pool.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-4 / 02Which computation produces the result in “Oracle reranking ceiling”?

;

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-4 / 03What is the baseline configuration for “Oracle reranking ceiling”?

{"query":"KV cache memory","documents":"[\n {\n \"id\": \"D1\",\n \"text\": \"KV cache stores keys and values for generation\",\n \"vector\": [\n 1,\n 0.1\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D2\",\n \"text\": \"Cache restaurant menu and opening hours\",\n \"vector\": [\n 0.1,\n 1\n ],\n \"relevance\": 0\n },\n {\n \"id\": \"D3\",\n \"text\": \"Grouped query attention reduces memory\",\n \"vector\": [\n 0.95,\n 0.2\n ],\n \"relevance\": 2\n },\n {\n \"id\": \"D4\",\n \"text\": \"KV cache memory footprint depends on sequence length\",\n \"vector\": [\n 0.8,\n 0.3\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D5\",\n \"text\": \"Tokenization splits input text\",\n \"vector\": [\n 0.4,\n 0.8\n ],\n \"relevance\": 0\n }\n]","x":1,"y":0.1,"weight":0.5,"k":60,"retrieve":5,"top":3,"rerank":1}

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-4 / 04What numerical result should you check for “Oracle reranking ceiling”?

{"Recall@k":1,"nDCG@k":1}

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-4 / 05How would you run a controlled comparison for “Oracle reranking ceiling”?

Keep the supplied fixture and other settings fixed; change weight. Compare Recall@k and nDCG@k while varying candidate pool and fusion weight.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-4 / 06What failure could invalidate a conclusion from “Oracle reranking ceiling”?

An oracle reranker leaks labels intentionally; never report it as learned model performance.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-4 / 07Is “Oracle reranking ceiling” a benchmark of a deployed language model?

No. Lexical score counts matching query terms (not BM25). Dense score is cosine on supplied 2D vectors, not a live embedding model. RRF combines ranks. Optional reranking uses known relevance labels as an oracle upper-bound, not a trained reranker. Empty query ties preserve document order.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-4 / 08Which evidence should you report for “Oracle reranking ceiling”?

Compare Recall@k and nDCG@k while varying candidate pool and fusion weight. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-4 / 09How can another researcher reproduce “Oracle reranking ceiling”?

Open retrieval-4, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-4 / 10What is the next validation step after “Oracle reranking ceiling”?

An oracle reranker leaks labels intentionally; never report it as learned model performance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-5 / 01What does “Candidate starvation” test?

Even an oracle cannot recover documents excluded before reranking.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-5 / 02Which computation produces the result in “Candidate starvation”?

;

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-5 / 03What is the baseline configuration for “Candidate starvation”?

{"query":"KV cache memory","documents":"[\n {\n \"id\": \"D1\",\n \"text\": \"KV cache stores keys and values for generation\",\n \"vector\": [\n 1,\n 0.1\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D2\",\n \"text\": \"Cache restaurant menu and opening hours\",\n \"vector\": [\n 0.1,\n 1\n ],\n \"relevance\": 0\n },\n {\n \"id\": \"D3\",\n \"text\": \"Grouped query attention reduces memory\",\n \"vector\": [\n 0.95,\n 0.2\n ],\n \"relevance\": 2\n },\n {\n \"id\": \"D4\",\n \"text\": \"KV cache memory footprint depends on sequence length\",\n \"vector\": [\n 0.8,\n 0.3\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D5\",\n \"text\": \"Tokenization splits input text\",\n \"vector\": [\n 0.4,\n 0.8\n ],\n \"relevance\": 0\n }\n]","x":1,"y":0.1,"weight":0.5,"k":60,"retrieve":1,"top":3,"rerank":1}

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-5 / 04What numerical result should you check for “Candidate starvation”?

{"Recall@k":0.3333333333333333,"nDCG@k":0.541942129480027}

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-5 / 05How would you run a controlled comparison for “Candidate starvation”?

Keep the supplied fixture and other settings fixed; change weight. Compare Recall@k and nDCG@k while varying candidate pool and fusion weight.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-5 / 06What failure could invalidate a conclusion from “Candidate starvation”?

An oracle reranker leaks labels intentionally; never report it as learned model performance.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-5 / 07Is “Candidate starvation” a benchmark of a deployed language model?

No. Lexical score counts matching query terms (not BM25). Dense score is cosine on supplied 2D vectors, not a live embedding model. RRF combines ranks. Optional reranking uses known relevance labels as an oracle upper-bound, not a trained reranker. Empty query ties preserve document order.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-5 / 08Which evidence should you report for “Candidate starvation”?

Compare Recall@k and nDCG@k while varying candidate pool and fusion weight. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-5 / 09How can another researcher reproduce “Candidate starvation”?

Open retrieval-5, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
retrieval-5 / 10What is the next validation step after “Candidate starvation”?

An oracle reranker leaks labels intentionally; never report it as learned model performance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗

실험 열기 →
judge-1 / 01What does “Mixed judge agreement” test?

Agreement and order consistency measure different properties.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-1 / 02Which computation produces the result in “Mixed judge agreement”?

; , remapped swapped)/n

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-1 / 03What is the baseline configuration for “Mixed judge agreement”?

{"task":"Compare two answers for factual correctness using supplied evidence.","ratings":"[\n {\n \"human\": \"A\",\n \"judge\": \"A\",\n \"swapped\": \"B\"\n },\n {\n \"human\": \"B\",\n \"judge\": \"A\",\n \"swapped\": \"A\"\n },\n {\n \"human\": \"tie\",\n \"judge\": \"tie\",\n \"swapped\": \"tie\"\n },\n {\n \"human\": \"B\",\n \"judge\": \"B\",\n \"swapped\": \"A\"\n }\n]"}

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-1 / 04What numerical result should you check for “Mixed judge agreement”?

{"Human agreement":0.75,"Order consistency":0.75,"A wins (original)":0.5}

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-1 / 05How would you run a controlled comparison for “Mixed judge agreement”?

Keep the supplied fixture and other settings fixed; change ratings. Compare human agreement and order consistency; retain evidence spans in real evaluations.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-1 / 06What failure could invalidate a conclusion from “Mixed judge agreement”?

High agreement on a small biased sample does not validate a judge for a new domain.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-1 / 07Is “Mixed judge agreement” a benchmark of a deployed language model?

No. Build an exportable rubric and analyze supplied judge labels. No LLM is called. In swapped trials A means original B and vice versa; normalize before computing consistency. Fixture labels are synthetic. Rubric wording is an original starting point, not a validated scale.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-1 / 08Which evidence should you report for “Mixed judge agreement”?

Compare human agreement and order consistency; retain evidence spans in real evaluations. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-1 / 09How can another researcher reproduce “Mixed judge agreement”?

Open judge-1, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-1 / 10What is the next validation step after “Mixed judge agreement”?

High agreement on a small biased sample does not validate a judge for a new domain. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-2 / 01What does “Perfect agreement” test?

Perfect labels yield agreement and consistency of one.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-2 / 02Which computation produces the result in “Perfect agreement”?

; , remapped swapped)/n

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-2 / 03What is the baseline configuration for “Perfect agreement”?

{"task":"Compare two answers for factual correctness using supplied evidence.","ratings":"[{\"human\":\"A\",\"judge\":\"A\",\"swapped\":\"B\"},{\"human\":\"B\",\"judge\":\"B\",\"swapped\":\"A\"}]"}

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-2 / 04What numerical result should you check for “Perfect agreement”?

{"Human agreement":1,"Order consistency":1,"A wins (original)":0.5}

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-2 / 05How would you run a controlled comparison for “Perfect agreement”?

Keep the supplied fixture and other settings fixed; change ratings. Compare human agreement and order consistency; retain evidence spans in real evaluations.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-2 / 06What failure could invalidate a conclusion from “Perfect agreement”?

High agreement on a small biased sample does not validate a judge for a new domain.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-2 / 07Is “Perfect agreement” a benchmark of a deployed language model?

No. Build an exportable rubric and analyze supplied judge labels. No LLM is called. In swapped trials A means original B and vice versa; normalize before computing consistency. Fixture labels are synthetic. Rubric wording is an original starting point, not a validated scale.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-2 / 08Which evidence should you report for “Perfect agreement”?

Compare human agreement and order consistency; retain evidence spans in real evaluations. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-2 / 09How can another researcher reproduce “Perfect agreement”?

Open judge-2, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-2 / 10What is the next validation step after “Perfect agreement”?

High agreement on a small biased sample does not validate a judge for a new domain. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-3 / 01What does “Always choose first” test?

Selecting the first position fails swap consistency.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-3 / 02Which computation produces the result in “Always choose first”?

; , remapped swapped)/n

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-3 / 03What is the baseline configuration for “Always choose first”?

{"task":"Compare two answers for factual correctness using supplied evidence.","ratings":"[{\"human\":\"A\",\"judge\":\"A\",\"swapped\":\"A\"},{\"human\":\"B\",\"judge\":\"A\",\"swapped\":\"A\"}]"}

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-3 / 04What numerical result should you check for “Always choose first”?

{"Human agreement":0.5,"Order consistency":0,"A wins (original)":1}

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-3 / 05How would you run a controlled comparison for “Always choose first”?

Keep the supplied fixture and other settings fixed; change ratings. Compare human agreement and order consistency; retain evidence spans in real evaluations.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-3 / 06What failure could invalidate a conclusion from “Always choose first”?

High agreement on a small biased sample does not validate a judge for a new domain.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-3 / 07Is “Always choose first” a benchmark of a deployed language model?

No. Build an exportable rubric and analyze supplied judge labels. No LLM is called. In swapped trials A means original B and vice versa; normalize before computing consistency. Fixture labels are synthetic. Rubric wording is an original starting point, not a validated scale.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-3 / 08Which evidence should you report for “Always choose first”?

Compare human agreement and order consistency; retain evidence spans in real evaluations. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-3 / 09How can another researcher reproduce “Always choose first”?

Open judge-3, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-3 / 10What is the next validation step after “Always choose first”?

High agreement on a small biased sample does not validate a judge for a new domain. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-4 / 01What does “Tie-heavy dataset” test?

A large tie class can make aggregate agreement misleading.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-4 / 02Which computation produces the result in “Tie-heavy dataset”?

; , remapped swapped)/n

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-4 / 03What is the baseline configuration for “Tie-heavy dataset”?

{"task":"Compare two answers for factual correctness using supplied evidence.","ratings":"[{\"human\":\"tie\",\"judge\":\"tie\",\"swapped\":\"tie\"},{\"human\":\"tie\",\"judge\":\"A\",\"swapped\":\"B\"},{\"human\":\"tie\",\"judge\":\"tie\",\"swapped\":\"tie\"}]"}

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-4 / 04What numerical result should you check for “Tie-heavy dataset”?

{"Human agreement":0.6666666666666666,"Order consistency":1,"A wins (original)":0.3333333333333333}

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-4 / 05How would you run a controlled comparison for “Tie-heavy dataset”?

Keep the supplied fixture and other settings fixed; change ratings. Compare human agreement and order consistency; retain evidence spans in real evaluations.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-4 / 06What failure could invalidate a conclusion from “Tie-heavy dataset”?

High agreement on a small biased sample does not validate a judge for a new domain.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-4 / 07Is “Tie-heavy dataset” a benchmark of a deployed language model?

No. Build an exportable rubric and analyze supplied judge labels. No LLM is called. In swapped trials A means original B and vice versa; normalize before computing consistency. Fixture labels are synthetic. Rubric wording is an original starting point, not a validated scale.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-4 / 08Which evidence should you report for “Tie-heavy dataset”?

Compare human agreement and order consistency; retain evidence spans in real evaluations. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-4 / 09How can another researcher reproduce “Tie-heavy dataset”?

Open judge-4, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-4 / 10What is the next validation step after “Tie-heavy dataset”?

High agreement on a small biased sample does not validate a judge for a new domain. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-5 / 01What does “Consistently wrong” test?

A judge can be order-consistent and disagree with every human label.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-5 / 02Which computation produces the result in “Consistently wrong”?

; , remapped swapped)/n

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-5 / 03What is the baseline configuration for “Consistently wrong”?

{"task":"Compare two answers for factual correctness using supplied evidence.","ratings":"[{\"human\":\"A\",\"judge\":\"B\",\"swapped\":\"A\"},{\"human\":\"B\",\"judge\":\"A\",\"swapped\":\"B\"}]"}

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-5 / 04What numerical result should you check for “Consistently wrong”?

{"Human agreement":0,"Order consistency":1,"A wins (original)":0.5}

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-5 / 05How would you run a controlled comparison for “Consistently wrong”?

Keep the supplied fixture and other settings fixed; change ratings. Compare human agreement and order consistency; retain evidence spans in real evaluations.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-5 / 06What failure could invalidate a conclusion from “Consistently wrong”?

High agreement on a small biased sample does not validate a judge for a new domain.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-5 / 07Is “Consistently wrong” a benchmark of a deployed language model?

No. Build an exportable rubric and analyze supplied judge labels. No LLM is called. In swapped trials A means original B and vice versa; normalize before computing consistency. Fixture labels are synthetic. Rubric wording is an original starting point, not a validated scale.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-5 / 08Which evidence should you report for “Consistently wrong”?

Compare human agreement and order consistency; retain evidence spans in real evaluations. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-5 / 09How can another researcher reproduce “Consistently wrong”?

Open judge-5, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
judge-5 / 10What is the next validation step after “Consistently wrong”?

High agreement on a small biased sample does not validate a judge for a new domain. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Judging LLM-as-a-Judge · MT-Bench ↗

실험 열기 →
routing-1 / 01What does “Two-tier routing” test?

Most traffic uses the cheap path; failures add strong-model cost.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-1 / 02Which computation produces the result in “Two-tier routing”?

E[C]=s(Ccheap+f×Cstrong)+(1−s)Cstrong; E[T]=s(Tcheap+f×Tstrong)+(1−s)Tstrong

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-1 / 03What is the baseline configuration for “Two-tier routing”?

{"requests":1000,"cheap":0.8,"failure":0.1,"cheapCost":0.001,"strongCost":0.02,"cheapMs":200,"strongMs":1000}

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-1 / 04What numerical result should you check for “Two-tier routing”?

{"Expected USD/request":0.0063999999999999994,"Expected total USD":6.3999999999999995,"Expected latency (ms)":439.99999999999994,"Fallback calls":80}

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-1 / 05How would you run a controlled comparison for “Two-tier routing”?

Keep the supplied fixture and other settings fixed; change failure. Compare expected cost, mean latency and fallback volume with observed production traces.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-1 / 06What failure could invalidate a conclusion from “Two-tier routing”?

A cheaper route may violate task-quality requirements; this estimator does not measure quality.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-1 / 07Is “Two-tier routing” a benchmark of a deployed language model?

No. Analytical expectation with user-supplied synthetic per-request prices and latencies, not current provider rates. Sequential one-time fallback; failed primary attempts are charged; fallback always succeeds. No quality prediction, queueing, correlated failures or tail-latency model.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-1 / 08Which evidence should you report for “Two-tier routing”?

Compare expected cost, mean latency and fallback volume with observed production traces. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-1 / 09How can another researcher reproduce “Two-tier routing”?

Open routing-1, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-1 / 10What is the next validation step after “Two-tier routing”?

A cheaper route may violate task-quality requirements; this estimator does not measure quality. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-2 / 01What does “All strong model” test?

Strong-only routing provides a simple cost baseline.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-2 / 02Which computation produces the result in “All strong model”?

E[C]=s(Ccheap+f×Cstrong)+(1−s)Cstrong; E[T]=s(Tcheap+f×Tstrong)+(1−s)Tstrong

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-2 / 03What is the baseline configuration for “All strong model”?

{"requests":1000,"cheap":0,"failure":0.1,"cheapCost":0.001,"strongCost":0.02,"cheapMs":200,"strongMs":1000}

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-2 / 04What numerical result should you check for “All strong model”?

{"Expected USD/request":0.02,"Expected total USD":20,"Expected latency (ms)":1000,"Fallback calls":0}

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-2 / 05How would you run a controlled comparison for “All strong model”?

Keep the supplied fixture and other settings fixed; change failure. Compare expected cost, mean latency and fallback volume with observed production traces.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-2 / 06What failure could invalidate a conclusion from “All strong model”?

A cheaper route may violate task-quality requirements; this estimator does not measure quality.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-2 / 07Is “All strong model” a benchmark of a deployed language model?

No. Analytical expectation with user-supplied synthetic per-request prices and latencies, not current provider rates. Sequential one-time fallback; failed primary attempts are charged; fallback always succeeds. No quality prediction, queueing, correlated failures or tail-latency model.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-2 / 08Which evidence should you report for “All strong model”?

Compare expected cost, mean latency and fallback volume with observed production traces. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-2 / 09How can another researcher reproduce “All strong model”?

Open routing-2, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-2 / 10What is the next validation step after “All strong model”?

A cheaper route may violate task-quality requirements; this estimator does not measure quality. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-3 / 01What does “Frequent fallback” test?

High fallback frequency can erase routing savings.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-3 / 02Which computation produces the result in “Frequent fallback”?

E[C]=s(Ccheap+f×Cstrong)+(1−s)Cstrong; E[T]=s(Tcheap+f×Tstrong)+(1−s)Tstrong

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-3 / 03What is the baseline configuration for “Frequent fallback”?

{"requests":1000,"cheap":0.8,"failure":0.8,"cheapCost":0.001,"strongCost":0.02,"cheapMs":200,"strongMs":1000}

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-3 / 04What numerical result should you check for “Frequent fallback”?

{"Expected USD/request":0.0176,"Expected total USD":17.6,"Expected latency (ms)":1000,"Fallback calls":640}

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-3 / 05How would you run a controlled comparison for “Frequent fallback”?

Keep the supplied fixture and other settings fixed; change failure. Compare expected cost, mean latency and fallback volume with observed production traces.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-3 / 06What failure could invalidate a conclusion from “Frequent fallback”?

A cheaper route may violate task-quality requirements; this estimator does not measure quality.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-3 / 07Is “Frequent fallback” a benchmark of a deployed language model?

No. Analytical expectation with user-supplied synthetic per-request prices and latencies, not current provider rates. Sequential one-time fallback; failed primary attempts are charged; fallback always succeeds. No quality prediction, queueing, correlated failures or tail-latency model.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-3 / 08Which evidence should you report for “Frequent fallback”?

Compare expected cost, mean latency and fallback volume with observed production traces. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-3 / 09How can another researcher reproduce “Frequent fallback”?

Open routing-3, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-3 / 10What is the next validation step after “Frequent fallback”?

A cheaper route may violate task-quality requirements; this estimator does not measure quality. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-4 / 01What does “Cheap model outage” test?

Every request pays both cost and latency before successful fallback.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-4 / 02Which computation produces the result in “Cheap model outage”?

E[C]=s(Ccheap+f×Cstrong)+(1−s)Cstrong; E[T]=s(Tcheap+f×Tstrong)+(1−s)Tstrong

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-4 / 03What is the baseline configuration for “Cheap model outage”?

{"requests":1000,"cheap":1,"failure":1,"cheapCost":0.001,"strongCost":0.02,"cheapMs":200,"strongMs":1000}

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-4 / 04What numerical result should you check for “Cheap model outage”?

{"Expected USD/request":0.021,"Expected total USD":21,"Expected latency (ms)":1200,"Fallback calls":1000}

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-4 / 05How would you run a controlled comparison for “Cheap model outage”?

Keep the supplied fixture and other settings fixed; change failure. Compare expected cost, mean latency and fallback volume with observed production traces.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-4 / 06What failure could invalidate a conclusion from “Cheap model outage”?

A cheaper route may violate task-quality requirements; this estimator does not measure quality.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-4 / 07Is “Cheap model outage” a benchmark of a deployed language model?

No. Analytical expectation with user-supplied synthetic per-request prices and latencies, not current provider rates. Sequential one-time fallback; failed primary attempts are charged; fallback always succeeds. No quality prediction, queueing, correlated failures or tail-latency model.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-4 / 08Which evidence should you report for “Cheap model outage”?

Compare expected cost, mean latency and fallback volume with observed production traces. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-4 / 09How can another researcher reproduce “Cheap model outage”?

Open routing-4, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-4 / 10What is the next validation step after “Cheap model outage”?

A cheaper route may violate task-quality requirements; this estimator does not measure quality. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-5 / 01What does “No primary failures” test?

Expected cost is a weighted average without retries.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-5 / 02Which computation produces the result in “No primary failures”?

E[C]=s(Ccheap+f×Cstrong)+(1−s)Cstrong; E[T]=s(Tcheap+f×Tstrong)+(1−s)Tstrong

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-5 / 03What is the baseline configuration for “No primary failures”?

{"requests":1000,"cheap":0.8,"failure":0,"cheapCost":0.001,"strongCost":0.02,"cheapMs":200,"strongMs":1000}

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-5 / 04What numerical result should you check for “No primary failures”?

{"Expected USD/request":0.0048,"Expected total USD":4.8,"Expected latency (ms)":359.99999999999994,"Fallback calls":0}

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-5 / 05How would you run a controlled comparison for “No primary failures”?

Keep the supplied fixture and other settings fixed; change failure. Compare expected cost, mean latency and fallback volume with observed production traces.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-5 / 06What failure could invalidate a conclusion from “No primary failures”?

A cheaper route may violate task-quality requirements; this estimator does not measure quality.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-5 / 07Is “No primary failures” a benchmark of a deployed language model?

No. Analytical expectation with user-supplied synthetic per-request prices and latencies, not current provider rates. Sequential one-time fallback; failed primary attempts are charged; fallback always succeeds. No quality prediction, queueing, correlated failures or tail-latency model.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-5 / 08Which evidence should you report for “No primary failures”?

Compare expected cost, mean latency and fallback volume with observed production traces. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-5 / 09How can another researcher reproduce “No primary failures”?

Open routing-5, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
routing-5 / 10What is the next validation step after “No primary failures”?

A cheaper route may violate task-quality requirements; this estimator does not measure quality. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: AWS Builders Library · timeouts, retries and backoff ↗

실험 열기 →
quantization-1 / 01What does “Four-bit baseline” test?

Fewer representable values introduce rounding error.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-1 / 02Which computation produces the result in “Four-bit baseline”?

; ; ×scale

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-1 / 03What is the baseline configuration for “Four-bit baseline”?

{"values":"-1,-0.7,-0.2,0,0.15,0.4,0.8,1","bits":4}

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-1 / 04What numerical result should you check for “Four-bit baseline”?

{"MSE":0.0009502551020408147,"Max absolute error":0.05714285714285716,"Ideal compression vs FP32":8,"Scale":0.14285714285714285}

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-1 / 05How would you run a controlled comparison for “Four-bit baseline”?

Keep the supplied fixture and other settings fixed; change bits. Compare reconstruction error, MSE, scale and ideal compression.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-1 / 06What failure could invalidate a conclusion from “Four-bit baseline”?

Low scalar MSE does not guarantee stable downstream model quality.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-1 / 07Is “Four-bit baseline” a benchmark of a deployed language model?

No. Symmetric per-tensor scalar quantizer with range ±(2^(bits−1)−1). This is not LLM.int8(), GPTQ or AWQ. Ideal compression ignores scale metadata and runtime packing. Scalar error does not measure language-model accuracy.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-1 / 08Which evidence should you report for “Four-bit baseline”?

Compare reconstruction error, MSE, scale and ideal compression. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-1 / 09How can another researcher reproduce “Four-bit baseline”?

Open quantization-1, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-1 / 10What is the next validation step after “Four-bit baseline”?

Low scalar MSE does not guarantee stable downstream model quality. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-2 / 01What does “Eight-bit reconstruction” test?

More quantization levels usually reduce reconstruction error.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-2 / 02Which computation produces the result in “Eight-bit reconstruction”?

; ; ×scale

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-2 / 03What is the baseline configuration for “Eight-bit reconstruction”?

{"values":"-1,-0.7,-0.2,0,0.15,0.4,0.8,1","bits":8}

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-2 / 04What numerical result should you check for “Eight-bit reconstruction”?

{"MSE":0.0000028868807737615443,"Max absolute error":0.003149606299212626,"Ideal compression vs FP32":4,"Scale":0.007874015748031496}

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-2 / 05How would you run a controlled comparison for “Eight-bit reconstruction”?

Keep the supplied fixture and other settings fixed; change bits. Compare reconstruction error, MSE, scale and ideal compression.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-2 / 06What failure could invalidate a conclusion from “Eight-bit reconstruction”?

Low scalar MSE does not guarantee stable downstream model quality.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-2 / 07Is “Eight-bit reconstruction” a benchmark of a deployed language model?

No. Symmetric per-tensor scalar quantizer with range ±(2^(bits−1)−1). This is not LLM.int8(), GPTQ or AWQ. Ideal compression ignores scale metadata and runtime packing. Scalar error does not measure language-model accuracy.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-2 / 08Which evidence should you report for “Eight-bit reconstruction”?

Compare reconstruction error, MSE, scale and ideal compression. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-2 / 09How can another researcher reproduce “Eight-bit reconstruction”?

Open quantization-2, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-2 / 10What is the next validation step after “Eight-bit reconstruction”?

Low scalar MSE does not guarantee stable downstream model quality. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-3 / 01What does “Two-bit stress test” test?

Only three signed levels preserve little detail.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-3 / 02Which computation produces the result in “Two-bit stress test”?

; ; ×scale

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-3 / 03What is the baseline configuration for “Two-bit stress test”?

{"values":"-1,-0.7,-0.2,0,0.15,0.4,0.8,1","bits":2}

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-3 / 04What numerical result should you check for “Two-bit stress test”?

{"MSE":0.044062500000000004,"Max absolute error":0.4,"Ideal compression vs FP32":16,"Scale":1}

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-3 / 05How would you run a controlled comparison for “Two-bit stress test”?

Keep the supplied fixture and other settings fixed; change bits. Compare reconstruction error, MSE, scale and ideal compression.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-3 / 06What failure could invalidate a conclusion from “Two-bit stress test”?

Low scalar MSE does not guarantee stable downstream model quality.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-3 / 07Is “Two-bit stress test” a benchmark of a deployed language model?

No. Symmetric per-tensor scalar quantizer with range ±(2^(bits−1)−1). This is not LLM.int8(), GPTQ or AWQ. Ideal compression ignores scale metadata and runtime packing. Scalar error does not measure language-model accuracy.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-3 / 08Which evidence should you report for “Two-bit stress test”?

Compare reconstruction error, MSE, scale and ideal compression. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-3 / 09How can another researcher reproduce “Two-bit stress test”?

Open quantization-3, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-3 / 10What is the next validation step after “Two-bit stress test”?

Low scalar MSE does not guarantee stable downstream model quality. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-4 / 01What does “Outlier dominates scale” test?

An outlier enlarges the global step and collapses small values.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-4 / 02Which computation produces the result in “Outlier dominates scale”?

; ; ×scale

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-4 / 03What is the baseline configuration for “Outlier dominates scale”?

{"values":"-1,-0.7,-0.2,0,0.15,0.4,0.8,30","bits":4}

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-4 / 04What numerical result should you check for “Outlier dominates scale”?

{"MSE":0.2940625,"Max absolute error":1,"Ideal compression vs FP32":8,"Scale":4.285714285714286}

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-4 / 05How would you run a controlled comparison for “Outlier dominates scale”?

Keep the supplied fixture and other settings fixed; change bits. Compare reconstruction error, MSE, scale and ideal compression.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-4 / 06What failure could invalidate a conclusion from “Outlier dominates scale”?

Low scalar MSE does not guarantee stable downstream model quality.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-4 / 07Is “Outlier dominates scale” a benchmark of a deployed language model?

No. Symmetric per-tensor scalar quantizer with range ±(2^(bits−1)−1). This is not LLM.int8(), GPTQ or AWQ. Ideal compression ignores scale metadata and runtime packing. Scalar error does not measure language-model accuracy.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-4 / 08Which evidence should you report for “Outlier dominates scale”?

Compare reconstruction error, MSE, scale and ideal compression. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-4 / 09How can another researcher reproduce “Outlier dominates scale”?

Open quantization-4, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-4 / 10What is the next validation step after “Outlier dominates scale”?

Low scalar MSE does not guarantee stable downstream model quality. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-5 / 01What does “All-zero tensor” test?

The zero-range safeguard reconstructs all zeros exactly.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-5 / 02Which computation produces the result in “All-zero tensor”?

; ; ×scale

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-5 / 03What is the baseline configuration for “All-zero tensor”?

{"values":"0,0,0,0","bits":4}

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-5 / 04What numerical result should you check for “All-zero tensor”?

{"MSE":0,"Max absolute error":0,"Ideal compression vs FP32":8,"Scale":1}

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-5 / 05How would you run a controlled comparison for “All-zero tensor”?

Keep the supplied fixture and other settings fixed; change bits. Compare reconstruction error, MSE, scale and ideal compression.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-5 / 06What failure could invalidate a conclusion from “All-zero tensor”?

Low scalar MSE does not guarantee stable downstream model quality.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-5 / 07Is “All-zero tensor” a benchmark of a deployed language model?

No. Symmetric per-tensor scalar quantizer with range ±(2^(bits−1)−1). This is not LLM.int8(), GPTQ or AWQ. Ideal compression ignores scale metadata and runtime packing. Scalar error does not measure language-model accuracy.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-5 / 08Which evidence should you report for “All-zero tensor”?

Compare reconstruction error, MSE, scale and ideal compression. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-5 / 09How can another researcher reproduce “All-zero tensor”?

Open quantization-5, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
quantization-5 / 10What is the next validation step after “All-zero tensor”?

Low scalar MSE does not guarantee stable downstream model quality. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: LLM.int8() · outliers and mixed precision ↗

실험 열기 →
cache-1 / 01What does “Locality baseline” test?

Repeated keys can avoid repeated model work.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-1 / 02Which computation produces the result in “Locality baseline”?

hit ; ×costPerMiss; mean ×hitMs+misses×missMs)/requests

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-1 / 03What is the baseline configuration for “Locality baseline”?

{"keys":"A,B,A,C,A,B,D,A","capacity":3,"ttl":60,"interval":1,"cost":0.01,"missMs":1000,"hitMs":10}

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-1 / 04What numerical result should you check for “Locality baseline”?

{"Hit rate":0.5,"Misses":4,"Cost USD":0.04,"Mean latency (ms)":505}

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-1 / 05How would you run a controlled comparison for “Locality baseline”?

Keep the supplied fixture and other settings fixed; change capacity. Inspect hit/miss trace, eviction order, cost and mean latency.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-1 / 06What failure could invalidate a conclusion from “Locality baseline”?

Cross-tenant key reuse can expose private data; stale cached answers may be incorrect.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-1 / 07Is “Locality baseline” a benchmark of a deployed language model?

No. Exact-key LRU simulation. TTL starts on insertion and does not refresh on hits. Expired entries are removed before each request. Assumes stable outputs within TTL. A real cache key should include tenant, model/version, prompt and generation settings; this lab uses letters.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-1 / 08Which evidence should you report for “Locality baseline”?

Inspect hit/miss trace, eviction order, cost and mean latency. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-1 / 09How can another researcher reproduce “Locality baseline”?

Open cache-1, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-1 / 10What is the next validation step after “Locality baseline”?

Cross-tenant key reuse can expose private data; stale cached answers may be incorrect. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-2 / 01What does “Capacity thrashing” test?

A tiny cache evicts useful entries before reuse.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-2 / 02Which computation produces the result in “Capacity thrashing”?

hit ; ×costPerMiss; mean ×hitMs+misses×missMs)/requests

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-2 / 03What is the baseline configuration for “Capacity thrashing”?

{"keys":"A,B,A,C,A,B,D,A","capacity":1,"ttl":60,"interval":1,"cost":0.01,"missMs":1000,"hitMs":10}

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-2 / 04What numerical result should you check for “Capacity thrashing”?

{"Hit rate":0,"Misses":8,"Cost USD":0.08,"Mean latency (ms)":1000}

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-2 / 05How would you run a controlled comparison for “Capacity thrashing”?

Keep the supplied fixture and other settings fixed; change capacity. Inspect hit/miss trace, eviction order, cost and mean latency.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-2 / 06What failure could invalidate a conclusion from “Capacity thrashing”?

Cross-tenant key reuse can expose private data; stale cached answers may be incorrect.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-2 / 07Is “Capacity thrashing” a benchmark of a deployed language model?

No. Exact-key LRU simulation. TTL starts on insertion and does not refresh on hits. Expired entries are removed before each request. Assumes stable outputs within TTL. A real cache key should include tenant, model/version, prompt and generation settings; this lab uses letters.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-2 / 08Which evidence should you report for “Capacity thrashing”?

Inspect hit/miss trace, eviction order, cost and mean latency. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-2 / 09How can another researcher reproduce “Capacity thrashing”?

Open cache-2, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-2 / 10What is the next validation step after “Capacity thrashing”?

Cross-tenant key reuse can expose private data; stale cached answers may be incorrect. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-3 / 01What does “TTL expiry” test?

Entries expire at the next request with this interval.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-3 / 02Which computation produces the result in “TTL expiry”?

hit ; ×costPerMiss; mean ×hitMs+misses×missMs)/requests

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-3 / 03What is the baseline configuration for “TTL expiry”?

{"keys":"A,B,A,C,A,B,D,A","capacity":3,"ttl":1,"interval":1,"cost":0.01,"missMs":1000,"hitMs":10}

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-3 / 04What numerical result should you check for “TTL expiry”?

{"Hit rate":0,"Misses":8,"Cost USD":0.08,"Mean latency (ms)":1000}

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-3 / 05How would you run a controlled comparison for “TTL expiry”?

Keep the supplied fixture and other settings fixed; change capacity. Inspect hit/miss trace, eviction order, cost and mean latency.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-3 / 06What failure could invalidate a conclusion from “TTL expiry”?

Cross-tenant key reuse can expose private data; stale cached answers may be incorrect.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-3 / 07Is “TTL expiry” a benchmark of a deployed language model?

No. Exact-key LRU simulation. TTL starts on insertion and does not refresh on hits. Expired entries are removed before each request. Assumes stable outputs within TTL. A real cache key should include tenant, model/version, prompt and generation settings; this lab uses letters.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-3 / 08Which evidence should you report for “TTL expiry”?

Inspect hit/miss trace, eviction order, cost and mean latency. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-3 / 09How can another researcher reproduce “TTL expiry”?

Open cache-3, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-3 / 10What is the next validation step after “TTL expiry”?

Cross-tenant key reuse can expose private data; stale cached answers may be incorrect. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-4 / 01What does “High reuse workload” test?

Temporal locality improves the hit rate.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-4 / 02Which computation produces the result in “High reuse workload”?

hit ; ×costPerMiss; mean ×hitMs+misses×missMs)/requests

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-4 / 03What is the baseline configuration for “High reuse workload”?

{"keys":"A,A,A,A,B,B,A,A","capacity":3,"ttl":60,"interval":1,"cost":0.01,"missMs":1000,"hitMs":10}

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-4 / 04What numerical result should you check for “High reuse workload”?

{"Hit rate":0.75,"Misses":2,"Cost USD":0.02,"Mean latency (ms)":257.5}

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-4 / 05How would you run a controlled comparison for “High reuse workload”?

Keep the supplied fixture and other settings fixed; change capacity. Inspect hit/miss trace, eviction order, cost and mean latency.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-4 / 06What failure could invalidate a conclusion from “High reuse workload”?

Cross-tenant key reuse can expose private data; stale cached answers may be incorrect.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-4 / 07Is “High reuse workload” a benchmark of a deployed language model?

No. Exact-key LRU simulation. TTL starts on insertion and does not refresh on hits. Expired entries are removed before each request. Assumes stable outputs within TTL. A real cache key should include tenant, model/version, prompt and generation settings; this lab uses letters.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-4 / 08Which evidence should you report for “High reuse workload”?

Inspect hit/miss trace, eviction order, cost and mean latency. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-4 / 09How can another researcher reproduce “High reuse workload”?

Open cache-4, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-4 / 10What is the next validation step after “High reuse workload”?

Cross-tenant key reuse can expose private data; stale cached answers may be incorrect. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-5 / 01What does “All unique requests” test?

Unique keys provide no exact-match cache savings.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-5 / 02Which computation produces the result in “All unique requests”?

hit ; ×costPerMiss; mean ×hitMs+misses×missMs)/requests

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-5 / 03What is the baseline configuration for “All unique requests”?

{"keys":"A,B,C,D,E,F,G,H","capacity":3,"ttl":60,"interval":1,"cost":0.01,"missMs":1000,"hitMs":10}

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-5 / 04What numerical result should you check for “All unique requests”?

{"Hit rate":0,"Misses":8,"Cost USD":0.08,"Mean latency (ms)":1000}

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-5 / 05How would you run a controlled comparison for “All unique requests”?

Keep the supplied fixture and other settings fixed; change capacity. Inspect hit/miss trace, eviction order, cost and mean latency.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-5 / 06What failure could invalidate a conclusion from “All unique requests”?

Cross-tenant key reuse can expose private data; stale cached answers may be incorrect.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-5 / 07Is “All unique requests” a benchmark of a deployed language model?

No. Exact-key LRU simulation. TTL starts on insertion and does not refresh on hits. Expired entries are removed before each request. Assumes stable outputs within TTL. A real cache key should include tenant, model/version, prompt and generation settings; this lab uses letters.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-5 / 08Which evidence should you report for “All unique requests”?

Inspect hit/miss trace, eviction order, cost and mean latency. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-5 / 09How can another researcher reproduce “All unique requests”?

Open cache-5, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →
cache-5 / 10What is the next validation step after “All unique requests”?

Cross-tenant key reuse can expose private data; stale cached answers may be incorrect. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.

Original teaching answer / local derivation. Method reference: Redis · eviction and TTL policies ↗

실험 열기 →