Token Lab検索

← 研究ラボ一覧
PHD / RESEARCH ENGINEERING

情報理論ラボ

Understand entropy, cross-entropy, both KL directions and support mismatch.

ローカル計算と合成データです。有料 API や実モデルのベンチマークは使用しません。

技術本文は英語と中国語に対応しています。この言語では英語の技術本文を表示します。

Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →

01 · 理解と導出

前提知識 · Discrete probabilities, logarithms, expectations.

For fixed target P, minimizing cross-entropy minimizes forward KL because H(P) is constant. Reversing KL changes which distribution weights the penalty. Inputs are weights normalized to sum to one; all outputs use nats.

Text / 原始公式
H(P) = −Σ pᵢ ln pᵢ
H(P,Q) = −Σ pᵢ ln qᵢ
KL(P||Q) = H(P,Q) − H(P)
JS(P,Q) = [KL(P||M)+KL(Q||M)]/2, M=(P+Q)/2
Step-by-step derivation and interpretation
  1. Expand ln p−Σp ln q. Rearranging yields , with the usual support conditions. P must remain fixed when interpreting optimization equivalence.
  2. For one observed class y and softmax logits z, cross-entropy is −z_y+log Σexp(z). Its derivative is softmax(z)_i−1[ This connects information theory to the training signal.
  3. Entropy measures uncertainty within a distribution. Calibration compares predicted probabilities with observed frequencies over labeled examples; a confident wrong prediction can have low entropy.

Common pitfall: Changing temperature at inference is not automatically calibration: fit and validate against labeled held-out data.

02 · 理解を確認

What is the entropy of a uniform distribution over 4 categories, in nats?

導出を表示

−4 × 0.25 × ln(0.25) = ln(4) ≈ 1.386294

03 · 対照実験を実行

実験課題

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Controlled experiment protocol
  1. Use ,0.2,0.1 and ,0.3,0.2. Verify H(P,Q)−H(P) equals forward KL.
  2. Swap P and Q; predict which quantities change. Multiply all P weights by 10 and verify normalization leaves the result unchanged.
  3. Run ,0 and ,1. Explain infinite cross-entropy and KL, while Export preserves infinity as a string instead of silently replacing it with zero.

Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.

ヒントを表示

A positive pᵢ at incurs infinite forward KL. Zero pᵢ contributes zero. The mixture M retains both supports.

04 · 根拠を確認

実行すると結果が表示されます。

条件を統制して実行結果を比較してください。出力に入力と手法を含みます。

05 · 推論を説明

自己評価：仕組みと前提を説明し、仮説を区別できる実験を提案してください。LLM 自動採点はありません。

1. Why is entropy constant in supervised cross-entropy minimization?

参考解答と評価観点

The target distribution is fixed with respect to model parameters; the prediction distribution changes.

2. Can a low-entropy model be badly wrong?

参考解答と評価観点

Yes. It can confidently concentrate mass on the wrong class. Evaluate accuracy, calibration and task costs separately.

3. Why does changing KL direction change optimization?

参考解答と評価観点

The expectation and support penalties change. Statements about mode coverage depend on the approximating family and target, not just a slogan.

前提と限界

This compares categorical distributions, not empirical model calibration. Calibration also needs predictions paired with observed outcomes; entropy alone is not correctness.

参考文献

arXiv:1706.04599 · 2017

On Calibration of Modern Neural Networks

Guo et al. · Year denotes first arXiv submission

Read: Reliability diagrams, calibration metrics, temperature scaling.

Reading checkpoint: Distinguish class ranking from confidence quality; fit temperature on validation data, never the final test set.

Analysis & visualization →arXiv abstract ↗PDF ↗
BibTeX
@misc{arxiv170604599,
  title = {On Calibration of Modern Neural Networks},
  year = {2017},
  eprint = {1706.04599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.04599}
}

arXiv:2305.18290 · 2023

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafailov et al. · Year denotes first arXiv submission

Read: KL-constrained reward objective, policy reparameterization and preference loss.

Reading checkpoint: Identify the reference-policy log ratio and β. The current information lab teaches KL, not a DPO trainer.

Analysis & visualization →arXiv abstract ↗PDF ↗
BibTeX
@misc{arxiv230518290,
  title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
  year = {2023},
  eprint = {2305.18290},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.18290}
}
MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

This compares categorical distributions, not empirical model calibration. Calibration also needs predictions paired with observed outcomes; entropy alone is not correctness.

Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.

Nonnegative weights P and Q → Normalize distributions separately → Check support before logarithms → H, cross entropy, KL and JS in nats
Mechanism map: follow the transformations from top to bottom.

Before running: predict a boundary case

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node information-theory.mjs. No third-party runtime dependencies.

import {runPhd} from './phd-core.mjs';
const params={
  "p": "0.7,0.2,0.1",
  "q": "0.5,0.3,0.2"
};
console.log(runPhd('information-theory',params));
Download this examplephd-core.mjsacademy-core.mjsresearch-core.mjs

Test and transfer

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope