Token LabSearch

← All research labs
PHD / RESEARCH ENGINEERING

Information Theory Lab

Understand entropy, cross-entropy, both KL directions and support mismatch.

Local calculations and synthetic examples. No paid API or live-model benchmark.
Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →

01 · Understand & derive

Prerequisites · Discrete probabilities, logarithms, expectations.

For fixed target P, minimizing cross-entropy minimizes forward KL because H(P) is constant. Reversing KL changes which distribution weights the penalty. Inputs are weights normalized to sum to one; all outputs use nats.

Text / 原始公式
H(P) = −Σ pᵢ ln pᵢ
H(P,Q) = −Σ pᵢ ln qᵢ
KL(P||Q) = H(P,Q) − H(P)
JS(P,Q) = [KL(P||M)+KL(Q||M)]/2, M=(P+Q)/2
Step-by-step derivation and interpretation
  1. Expand ln p−Σp ln q. Rearranging yields , with the usual support conditions. P must remain fixed when interpreting optimization equivalence.
  2. For one observed class y and softmax logits z, cross-entropy is −z_y+log Σexp(z). Its derivative is softmax(z)_i−1[ This connects information theory to the training signal.
  3. Entropy measures uncertainty within a distribution. Calibration compares predicted probabilities with observed frequencies over labeled examples; a confident wrong prediction can have low entropy.

Common pitfall: Changing temperature at inference is not automatically calibration: fit and validate against labeled held-out data.

02 · Check your understanding

What is the entropy of a uniform distribution over 4 categories, in nats?

Show derivation

−4 × 0.25 × ln(0.25) = ln(4) ≈ 1.386294

03 · Run a controlled experiment

Experiment assignment

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Controlled experiment protocol
  1. Use ,0.2,0.1 and ,0.3,0.2. Verify H(P,Q)−H(P) equals forward KL.
  2. Swap P and Q; predict which quantities change. Multiply all P weights by 10 and verify normalization leaves the result unchanged.
  3. Run ,0 and ,1. Explain infinite cross-entropy and KL, while Export preserves infinity as a string instead of silently replacing it with zero.

Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.

Show a hint

A positive pᵢ at incurs infinite forward KL. Zero pᵢ contributes zero. The mixture M retains both supports.

04 · Inspect evidence

Run the experiment to see evidence.

Only compare runs with controlled inputs. Exports include inputs and methods.

05 · Defend your reasoning

Self-assessment: explain the mechanism, identify assumptions, and propose a discriminating test. No automatic LLM grading.

1. Why is entropy constant in supervised cross-entropy minimization?

Show answer & scoring criteria

The target distribution is fixed with respect to model parameters; the prediction distribution changes.

2. Can a low-entropy model be badly wrong?

Show answer & scoring criteria

Yes. It can confidently concentrate mass on the wrong class. Evaluate accuracy, calibration and task costs separately.

3. Why does changing KL direction change optimization?

Show answer & scoring criteria

The expectation and support penalties change. Statements about mode coverage depend on the approximating family and target, not just a slogan.

Assumptions & limits

This compares categorical distributions, not empirical model calibration. Calibration also needs predictions paired with observed outcomes; entropy alone is not correctness.

References

arXiv:1706.04599 · 2017

On Calibration of Modern Neural Networks

Guo et al. · Year denotes first arXiv submission

Read: Reliability diagrams, calibration metrics, temperature scaling.

Reading checkpoint: Distinguish class ranking from confidence quality; fit temperature on validation data, never the final test set.

Analysis & visualization →arXiv abstract ↗PDF ↗
BibTeX
@misc{arxiv170604599,
  title = {On Calibration of Modern Neural Networks},
  year = {2017},
  eprint = {1706.04599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.04599}
}

arXiv:2305.18290 · 2023

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafailov et al. · Year denotes first arXiv submission

Read: KL-constrained reward objective, policy reparameterization and preference loss.

Reading checkpoint: Identify the reference-policy log ratio and β. The current information lab teaches KL, not a DPO trainer.

Analysis & visualization →arXiv abstract ↗PDF ↗
BibTeX
@misc{arxiv230518290,
  title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
  year = {2023},
  eprint = {2305.18290},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.18290}
}
MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

This compares categorical distributions, not empirical model calibration. Calibration also needs predictions paired with observed outcomes; entropy alone is not correctness.

Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.

Nonnegative weights P and Q → Normalize distributions separately → Check support before logarithms → H, cross entropy, KL and JS in nats
Mechanism map: follow the transformations from top to bottom.

Before running: predict a boundary case

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node information-theory.mjs. No third-party runtime dependencies.

import {runPhd} from './phd-core.mjs';
const params={
  "p": "0.7,0.2,0.1",
  "q": "0.5,0.3,0.2"
};
console.log(runPhd('information-theory',params));
Download this examplephd-core.mjsacademy-core.mjsresearch-core.mjs

Test and transfer

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope