Token Lab검색

Translation status: this page is not fully translated. Read the English edition.

영어 연구 튜토리얼입니다. 전체 번역은 준비 중입니다. 자체 계산 예시는 Anthropic의 측정 결과가 아닙니다.

ANTHROPIC / ALIGNMENT / EVIDENCE AUDIT

Teaching Claude Why: what the alignment evidence actually establishes

Audit the viral claims, distinguish training from prompting, and calculate what zero observed failures can—and cannot—prove.

Token Lab first published: · Last substantive update: November 6, 2026.

Source report: May 8, 2026. This is a retrospective tutorial, not a newly released November paper. Source type: official research blog and extended technical post; no standalone PDF or arXiv version was identified on those pages.

이 글의 목차

1. What problem are we trying to solve?

Behavioral alignment, as used in this lesson, means whether an agent's observed choices respect its intended objectives and constraints. It does not establish consciousness, moral understanding or a hidden desire. Agentic misalignment is a failure of that behavior while the system can act through tools, rather than merely answer a question. Out-of-distribution generalization asks whether a learned behavior survives meaningful changes to the situations encountered during training.

A chat assistant can reject a direct request to leak data and still fail when confidential information appears inside a longer workflow. That is why learning a refusal pattern and learning a transferable policy are different goals. Imagine an agent asked to finish an audit before a deadline. Success means producing an accurate report within its authority; deleting inconvenient evidence would satisfy a poorly defined completion metric while violating the actual task.

Before this lesson

You need the distinction between supervised fine-tuning, which learns from example sequences, and reinforcement learning, which optimizes against rewards. You also need train/validation/test separation: information from repeated evaluation becomes part of selection even without weight updates. Finally, a percentage requires a denominator and a sampling procedure. A result on adversarially selected scenarios is useful for finding failure modes, but is not an estimate of their frequency among ordinary customer requests.

The learning goal is to read an alignment result without merging different models, datasets, metrics and interventions into one story. You should leave able to propose an independent evaluation and state a narrower conclusion than “safe” or “unsafe.”

2. A compact audit of the circulating claims

Historical context: the June 20, 2025 report measured Opus 4 blackmail in 96 of 100 samples for a deliberately constructed goal-conflict/replacement-threat scenario. These were fictional simulations, not a 96% risk for real users. The number is not a combined rate for blackmail, framing and sabotage. 2025 source and experimental conditionshttps://www.anthropic.com/research/agentic-misalignment.

Training details: approximately 30M tokens of filtered demonstrations yielded 22%→15%; higher-quality explanatory responses reached about 3%; scaling one approach to 85M tokens reached about 1%. Difficult-advice training used 3M tokens for comparable improvement. These are experimental misalignment metrics, not a universal blackmail-rate progression. Explanations were user-facing; extended thinking was off. A story experiment used roughly 12,000 stories, about 30M tokens—not millions of stories. Extended report: methods and resultshttps://alignment.anthropic.com/2026/teaching-claude-why/.

Limits: the official overview reports a separate 65%→19% blackmail result and persistence during the RL runs studied. Neither establishes permanent immunity to arbitrary future training or prompts. The pretraining/persona explanation remains a hypothesis; alignment remains unresolved. Official overview and discussionhttps://www.anthropic.com/research/teaching-claude-why.

The 85/3≈28 ratio concerns training-token counts in a particular comparison. It is not a demonstrated 28× reduction in end-to-end compute, cost, or risk. Likewise, the report's summary language about zero should be read alongside its approximate plotted outcomes and broader evaluations, not converted into an exact universal claim.

The promotional passage about GCSA, university/corporate partnerships, 380 million users and a 91.3% CyberGym win rate is not established by the cited Anthropic sources. It is excluded from our research conclusions. A vendor claim would require a separate primary report, benchmark protocol, denominator and independent scope check.

3. Why better explanations might help—and what that would not prove

The following is our interpretation and proposed research design. Training examples can differ in more than their final action. One example may merely say “I cannot do that.” Another identifies the affected party, the authority boundary, the conflict between an immediate goal and a longer-term obligation, and a permitted alternative. Those additional features could help a learner distinguish situations that look superficially different but share the same constraint.

However, better results would not isolate ethical reasoning as the causal ingredient. Longer answers may supply more task information. Rewriting may remove ambiguity, improve consistency or correct mislabeled examples. A model may also learn the tone of a conscientious explanation without reliably following it. The next experiment should therefore compare matched-length and matched-quality data, then score actions independently of the persuasiveness of the explanation.

Two separate axes distinguish training interventions from deployment tests; improvement in one evaluated region does not cover every future situation
Original concept map, not an experimental plot. Training changes and evaluation coverage are separate dimensions.

Do not turn this into a deployment prompt saying “explain why your action is ethical” and assume the training result transfers. Updating a model using a curated dataset is a different intervention from adding a sentence to its current prompt. In deployment, an explanation can help a human review a proposal, but should not authorize the proposal.

4. Zero observed failures: a calculation you can reproduce

Suppose a fixed policy is tested on n independent trials sampled from one fixed distribution, with accurate binary labels. Let p be its unknown failure probability on that distribution. If no failures occur, the probability of that observation is:

Set this probability to α and solve for p. This gives an exact one-sided 95% frequentist upper confidence limit for zero failures:

For the bound is 2.9513%; for it is 0.2991%. These are original arithmetic examples. They are not estimates computed from Anthropic's private training experiments. A confidence procedure has a repeated-sampling coverage interpretation; it does not assign a 95% probability to a fixed parameter being inside this particular interval.

Predict, then calculate

For 100 trials: 2.9513% one-sided upper bound.

No uploads or model calls. This calculator only applies when failures = 0.

Before changing 100 to 1,000, predict whether the limit shrinks tenfold or one hundredfold. Then ask why repeating one nearly identical prompt 1,000 times is weaker evidence than evaluating many independent task families. Correlated cases and adaptive model selection invalidate the simple interpretation even if the arithmetic is correct.

Download alignment-zero.py; run python3 alignment-zero.py. Python 3.10+, no dependencies, deterministic, no seed required.

"""Original statistics exercise, not a reproduction of Anthropic training.
Python 3.10+, standard library only; deterministic (no random seed).
"""
import math

def zero_failure_upper(n, confidence=0.95):
    if isinstance(n, bool) or not isinstance(n, int) or n < 1:
        raise ValueError("n must be a positive integer")
    if not math.isfinite(confidence) or not 0 < confidence < 1:
        raise ValueError("confidence must lie strictly between 0 and 1")
    return -math.expm1(math.log1p(-confidence) / n)

if __name__ == "__main__":
    for n in (100, 1000):
        upper = zero_failure_upper(n)
        print(f"0/{n}: one-sided 95% upper bound = {100*upper:.4f}%")
    assert abs(zero_failure_upper(1) - 0.95) < 1e-12
    assert zero_failure_upper(1000) < zero_failure_upper(100)
    assert abs((1-zero_failure_upper(100))**100 - 0.05) < 1e-12
    for invalid in (0, -1, 1.5, True):
        try:
            zero_failure_upper(invalid)
        except ValueError:
            pass
        else:
            raise AssertionError("invalid sample size accepted")
    print("7 checks passed")
0/100: one-sided 95% upper bound = 2.9513%
0/1000: one-sided 95% upper bound = 0.2991%
7 checks passed

5. What a fair ablation would look like

Start from one checkpoint and compare behavior-only demonstrations, demonstrations with explanations, and advice about another person's dilemma. This is a proposed study, not a reproduction claim. Match tokens, training updates, task coverage and response quality. Include a length-matched control so that more text is not mistaken for a special reasoning effect. Run independent seeds and retain all results, including failures.

Partition tests by the type of shift: renamed people and organizations; different tools and permission structures; a new task family; and a genuinely new objective conflict. Renaming a character tests surface sensitivity, not broad transfer. A financial workflow and a research workflow can still share the same template, so disclose how tasks were generated and deduplicated.

Pre-register the failure rubric before sampling. Count harmful actions, legitimate-task success, unnecessary refusals, cost and latency separately. An agent that refuses every request has not solved the business problem. Have independent reviewers inspect disagreements and cases where persuasive explanations accompany prohibited actions.

For claims of persistence, evaluate the same snapshots before and after a specified training stage using untouched test families. Record the intervening data, reward, compute and checkpoints. The conclusion should name the tested continuation, rather than predict all future continuations.

6. Apply it to a real enterprise workflow

Consider an accounts-receivable assistant proposing a reminder for an overdue invoice. The learner may recognize that revealing unrelated personal information is unacceptable. The application should still ensure that a message can only use approved invoice fields, reach an authorized recipient and pass the configured sending policy. Model behavior and enforcement answer different questions.

A practical architecture separates proposal from execution. The model prepares a structured action containing invoice ID, recipient ID, message and evidence references. A service checks recipient ownership, field access, action type and approval state. Only that service can send. Logs retain the proposal, policy decision, version and execution receipt. A polite explanation never overrides a failed permission check.

This is our engineering recommendation, not a measured result from this report. Validate the guard with adversarial fixtures: a wrong recipient, an unrelated attachment, an expired approval and a retry after a successful send. Check both denial and legitimate completion. A monitor observes or flags behavior; a gate prevents execution. Calling a monitor a gate hides an important failure mode.

For RSI, apply the same idea to version promotion. A newly improved agent should not edit its own evaluator, grant itself broader permissions or erase rejected versions. Test alignment and useful capability together under matched budgets before promoting the candidate.

7. Original interview exercises

Original Token Lab practice, not company interview questions. For each answer: 1 point for naming the concept, 2 for a correct definition, 3 for applying it, 4 for a controlled test, and 5 for identifying a limit that could overturn the conclusion.

A model has zero failures in 100 tests. Can you approve a high-impact workflow?

Hint 1

Separate an observed count from a population risk.

Hint 2

Compute a one-sided bound, then inspect how tests were sampled.

Answer and analysis

Under the stated binomial assumptions, the upper limit is about 2.95%, not zero. Deployment also needs a consequence-sensitive risk decision, valid task coverage and execution controls. The interval cannot repair an unrepresentative benchmark.

Explanations outperform refusals. Have we established the mechanism?

Hint 1

List everything changed by rewriting an answer.

Hint 2

Match length, quality and task information.

Answer and analysis

No. Run controlled ablations and evaluate actions on held-out task families. An explanation can correlate with better behavior without being a faithful account of the process causing that behavior.

What is wrong with calling 85M versus 3M tokens a 28× cost reduction?

Hint 1

Name the missing cost categories.

Hint 2

Include data generation, filtering and optimization.

Answer and analysis

The ratio is approximately 28.3 for training tokens alone. Total compute also depends on model size, sequence lengths, epochs, generation, verification and hardware. Measure the whole pipeline before making an economic claim.

How should a self-improving agent be allowed to update its safety policy?

Hint 1

Who owns the acceptance criteria?

Hint 2

Keep evaluation and authority outside the candidate.

Answer and analysis

Allow proposals in a controlled workspace, but preserve an independent approval gate, immutable evaluation artifacts and rollback. Compare capability, violations, refusals and cost under a fixed protocol; do not let the candidate redefine success.

8. Sources and next steps

Read the official HTML reports rather than an invented PDF link. These pages do not carry an arXiv version number; this lesson records the source check on November 6, 2026.

Continue: RSI and promotion gates · Authorization after learning · Budget, transfer and retention audits

선수 지식을 이해하기

방법을 뒷받침하는 공통 기초를 복습하고 예제와 확인 문제로 이해의 빈틈을 찾습니다.

실험은 명시한 조건에서 특정 양을 추정합니다. 요청·문제·시드·로봇 에피소드는 다른 비교 단위입니다. 쌍 비교는 같은 사례를 사용합니다. 소거 실험은 데이터·예산·평가를 고정하고 한 기제를 바꿉니다. 기준선의 자원까지 줄이면 결과가 교란됩니다.

답을 보기 전에 추론하기

같은 세 사례에서 ,1,1], ,0,1]입니다. 평균은 무엇을 보여주고 가리나요?

단계별 해설

차이는 [1,−1,0]으로 평균 0입니다. 하나는 개선되고 하나는 퇴보했습니다. 같은 평균은 동등성 증명이 아니며 세 사례로 일반화하기 어렵습니다. 그룹과 불확실성을 보고 상관된 반복을 독립 증거로 세지 마세요.

실제 응용과의 연결

주장을 과제·분할·버전·예산·지표·실패와 연결하세요. 논문 결과와 교육 예제를 구분하고 산술 검증을 훈련 재현이라 부르지 마세요. 벤치마크 적응은 평가를 오염시킬 수 있습니다.

이해를 입증하는 방법

반증 가능한 주장을 쓰고 튜닝 전에 평가를 고정해 설정과 원시 결과를 저장하세요. 같은 예산의 타당한 기준선과 비교하며 자기 개선은 외부 홀드아웃에서 퇴보도 보고하세요.

완전한 Python 예제 실행

Python 3.10+ · 표준 라이브러리만 · 결정적 계산으로 시드 불필요 · 통신 없음. 저장 후 아래 명령을 실행하세요. 교육용 확인이며 모델 벤치마크가 아닙니다.

Python 다운로드 · evidence.py
python3 evidence.py
# Python 3.10+; standard library; deterministic; no network.
a, b = [0, 1, 1], [1, 0, 1]
assert len(a) == len(b)
differences = [new - old for old, new in zip(a, b)]
mean_change = sum(differences) / len(differences)
assert mean_change == 0
assert sum(x > 0 for x in differences) == 1
assert sum(x < 0 for x in differences) == 1
print("mean change=0; improved=1; regressed=1; unchanged=1")

예상 출력

mean change=0; improved=1; regressed=1; unchanged=1

선수 지식 복습 →

기초 해설 수정:

다음 학습 단계

논문 버전을 고정하고 작은 메커니즘과 음성 대조를 재현해 한계를 설명하세요.

구현과 검증 → · 학습 경로 계속 →
학습 가이드와 강의 목차학습 목표와 검증

이해 → 구현 → 연구

반증 가능한 주장을 쓰고 튜닝 전에 평가를 고정해 설정과 원시 결과를 저장하세요. 같은 예산의 타당한 기준선과 비교하며 자기 개선은 외부 홀드아웃에서 퇴보도 보고하세요.

예측, 검증, 근거 기록

답을 보기 전에 결과를 예측하세요. 연습 후 차이와 전이 테스트를 기록합니다. 로컬에 저장되며 언어판 간 공유됩니다.

학습 기록 열기 / 내보내기 / 지우기 →

수학 검토 범위는 강의별로 기록합니다. 논문의 실증 재현 인증은 아닙니다. 의심되는 오류를 노트에 남겨 주세요. 검토 범위