Token Lab검색

← 전체 연구 실험
PHD / RESEARCH ENGINEERING

정보 이론 실험실

Understand entropy, cross-entropy, both KL directions and support mismatch.

로컬 계산과 합성 예제입니다. 유료 API나 실제 모델 벤치마크를 사용하지 않습니다.

기술 본문은 영어와 중국어로 제공됩니다. 이 언어에서는 영어 기술 본문을 표시합니다.

Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →

01 · 이해와 유도

사전 지식 · Discrete probabilities, logarithms, expectations.

For fixed target P, minimizing cross-entropy minimizes forward KL because H(P) is constant. Reversing KL changes which distribution weights the penalty. Inputs are weights normalized to sum to one; all outputs use nats.

Text / 原始公式
H(P) = −Σ pᵢ ln pᵢ
H(P,Q) = −Σ pᵢ ln qᵢ
KL(P||Q) = H(P,Q) − H(P)
JS(P,Q) = [KL(P||M)+KL(Q||M)]/2, M=(P+Q)/2
Step-by-step derivation and interpretation
  1. Expand ln p−Σp ln q. Rearranging yields , with the usual support conditions. P must remain fixed when interpreting optimization equivalence.
  2. For one observed class y and softmax logits z, cross-entropy is −z_y+log Σexp(z). Its derivative is softmax(z)_i−1[ This connects information theory to the training signal.
  3. Entropy measures uncertainty within a distribution. Calibration compares predicted probabilities with observed frequencies over labeled examples; a confident wrong prediction can have low entropy.

Common pitfall: Changing temperature at inference is not automatically calibration: fit and validate against labeled held-out data.

02 · 이해도 확인

What is the entropy of a uniform distribution over 4 categories, in nats?

유도 과정 보기

−4 × 0.25 × ln(0.25) = ln(4) ≈ 1.386294

03 · 통제 실험 실행

실험 과제

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Controlled experiment protocol
  1. Use ,0.2,0.1 and ,0.3,0.2. Verify H(P,Q)−H(P) equals forward KL.
  2. Swap P and Q; predict which quantities change. Multiply all P weights by 10 and verify normalization leaves the result unchanged.
  3. Run ,0 and ,1. Explain infinite cross-entropy and KL, while Export preserves infinity as a string instead of silently replacing it with zero.

Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.

힌트 보기

A positive pᵢ at incurs infinite forward KL. Zero pᵢ contributes zero. The mixture M retains both supports.

04 · 근거 확인

실행하면 결과가 표시됩니다.

변수를 통제하여 결과를 비교하세요. 내보내기에는 입력과 방법이 포함됩니다.

05 · 추론 설명

자기 평가: 원리와 가정을 설명하고 구별 가능한 실험을 제안하세요. LLM 자동 채점은 없습니다.

1. Why is entropy constant in supervised cross-entropy minimization?

참고 답변과 평가 기준

The target distribution is fixed with respect to model parameters; the prediction distribution changes.

2. Can a low-entropy model be badly wrong?

참고 답변과 평가 기준

Yes. It can confidently concentrate mass on the wrong class. Evaluate accuracy, calibration and task costs separately.

3. Why does changing KL direction change optimization?

참고 답변과 평가 기준

The expectation and support penalties change. Statements about mode coverage depend on the approximating family and target, not just a slogan.

가정과 한계

This compares categorical distributions, not empirical model calibration. Calibration also needs predictions paired with observed outcomes; entropy alone is not correctness.

참고 문헌

arXiv:1706.04599 · 2017

On Calibration of Modern Neural Networks

Guo et al. · Year denotes first arXiv submission

Read: Reliability diagrams, calibration metrics, temperature scaling.

Reading checkpoint: Distinguish class ranking from confidence quality; fit temperature on validation data, never the final test set.

Analysis & visualization →arXiv abstract ↗PDF ↗
BibTeX
@misc{arxiv170604599,
  title = {On Calibration of Modern Neural Networks},
  year = {2017},
  eprint = {1706.04599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.04599}
}

arXiv:2305.18290 · 2023

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafailov et al. · Year denotes first arXiv submission

Read: KL-constrained reward objective, policy reparameterization and preference loss.

Reading checkpoint: Identify the reference-policy log ratio and β. The current information lab teaches KL, not a DPO trainer.

Analysis & visualization →arXiv abstract ↗PDF ↗
BibTeX
@misc{arxiv230518290,
  title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
  year = {2023},
  eprint = {2305.18290},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.18290}
}
MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

This compares categorical distributions, not empirical model calibration. Calibration also needs predictions paired with observed outcomes; entropy alone is not correctness.

Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.

Nonnegative weights P and Q → Normalize distributions separately → Check support before logarithms → H, cross entropy, KL and JS in nats
Mechanism map: follow the transformations from top to bottom.

Before running: predict a boundary case

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node information-theory.mjs. No third-party runtime dependencies.

import {runPhd} from './phd-core.mjs';
const params={
  "p": "0.7,0.2,0.1",
  "q": "0.5,0.3,0.2"
};
console.log(runPhd('information-theory',params));
Download this examplephd-core.mjsacademy-core.mjsresearch-core.mjs

Test and transfer

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope