Token LabBuscar

← Volver a los laboratorios
PHD / RESEARCH ENGINEERING

Laboratorio de teoría de la información

Understand entropy, cross-entropy, both KL directions and support mismatch.

Cálculos locales y ejemplos sintéticos. Sin API de pago ni benchmark de modelos reales.

El contenido técnico está disponible en inglés y chino; esta versión usa el texto técnico en inglés.

Study guide & lesson outlineLEARNING CONTRACT

Understand → implement → investigate

Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.

Prerequisite route →Related implementation practice →

01 · Comprender y derivar

Requisitos previos · Discrete probabilities, logarithms, expectations.

For fixed target P, minimizing cross-entropy minimizes forward KL because H(P) is constant. Reversing KL changes which distribution weights the penalty. Inputs are weights normalized to sum to one; all outputs use nats.

Text / 原始公式
H(P) = −Σ pᵢ ln pᵢ
H(P,Q) = −Σ pᵢ ln qᵢ
KL(P||Q) = H(P,Q) − H(P)
JS(P,Q) = [KL(P||M)+KL(Q||M)]/2, M=(P+Q)/2
Step-by-step derivation and interpretation
  1. Expand ln p−Σp ln q. Rearranging yields , with the usual support conditions. P must remain fixed when interpreting optimization equivalence.
  2. For one observed class y and softmax logits z, cross-entropy is −z_y+log Σexp(z). Its derivative is softmax(z)_i−1[ This connects information theory to the training signal.
  3. Entropy measures uncertainty within a distribution. Calibration compares predicted probabilities with observed frequencies over labeled examples; a confident wrong prediction can have low entropy.

Common pitfall: Changing temperature at inference is not automatically calibration: fit and validate against labeled held-out data.

02 · Comprobar conocimientos

What is the entropy of a uniform distribution over 4 categories, in nats?

Mostrar derivación

−4 × 0.25 × ln(0.25) = ln(4) ≈ 1.386294

03 · Experimento controlado

Tarea experimental

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Controlled experiment protocol
  1. Use ,0.2,0.1 and ,0.3,0.2. Verify H(P,Q)−H(P) equals forward KL.
  2. Swap P and Q; predict which quantities change. Multiply all P weights by 10 and verify normalization leaves the result unchanged.
  3. Run ,0 and ,1. Explain infinite cross-entropy and KL, while Export preserves infinity as a string instead of silently replacing it with zero.

Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.

Mostrar pista

A positive pᵢ at incurs infinite forward KL. Zero pᵢ contributes zero. The mixture M retains both supports.

04 · Analizar evidencia

Ejecuta el experimento para ver la evidencia.

Compara ejecuciones con variables controladas. Las exportaciones incluyen entradas y métodos.

05 · Defender el razonamiento

Autoevaluación: explica el mecanismo, identifica los supuestos y propone una prueba. Sin calificación automática por LLM.

1. Why is entropy constant in supervised cross-entropy minimization?

Ver respuesta y criterios

The target distribution is fixed with respect to model parameters; the prediction distribution changes.

2. Can a low-entropy model be badly wrong?

Ver respuesta y criterios

Yes. It can confidently concentrate mass on the wrong class. Evaluate accuracy, calibration and task costs separately.

3. Why does changing KL direction change optimization?

Ver respuesta y criterios

The expectation and support penalties change. Statements about mode coverage depend on the approximating family and target, not just a slogan.

Supuestos y límites

This compares categorical distributions, not empirical model calibration. Calibration also needs predictions paired with observed outcomes; entropy alone is not correctness.

Referencias

arXiv:1706.04599 · 2017

On Calibration of Modern Neural Networks

Guo et al. · Year denotes first arXiv submission

Read: Reliability diagrams, calibration metrics, temperature scaling.

Reading checkpoint: Distinguish class ranking from confidence quality; fit temperature on validation data, never the final test set.

Analysis & visualization →arXiv abstract ↗PDF ↗
BibTeX
@misc{arxiv170604599,
  title = {On Calibration of Modern Neural Networks},
  year = {2017},
  eprint = {1706.04599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.04599}
}

arXiv:2305.18290 · 2023

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafailov et al. · Year denotes first arXiv submission

Read: KL-constrained reward objective, policy reparameterization and preference loss.

Reading checkpoint: Identify the reference-policy log ratio and β. The current information lab teaches KL, not a DPO trainer.

Analysis & visualization →arXiv abstract ↗PDF ↗
BibTeX
@misc{arxiv230518290,
  title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
  year = {2023},
  eprint = {2305.18290},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.18290}
}
MATHEMATICAL REVIEW · 2026-10-26

Assumptions, implementation & acceptance

This compares categorical distributions, not empirical model calibration. Calibration also needs predictions paired with observed outcomes; entropy alone is not correctness.

Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.

Nonnegative weights P and Q → Normalize distributions separately → Check support before logarithms → H, cross entropy, KL and JS in nats
Mechanism map: follow the transformations from top to bottom.

Before running: predict a boundary case

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Hint: isolate the assumption

Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node information-theory.mjs. No third-party runtime dependencies.

import {runPhd} from './phd-core.mjs';
const params={
  "p": "0.7,0.2,0.1",
  "q": "0.5,0.3,0.2"
};
console.log(runPhd('information-theory',params));
Download this examplephd-core.mjsacademy-core.mjsresearch-core.mjs

Test and transfer

Set , then use ,0 and ,1. Explain the infinite KL and finite Jensen–Shannon divergence without introducing an arbitrary epsilon.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.

Download complete kit (.tar.gz) · Complete reference kit and regression checks

Predict, test and keep evidence

Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.

Open notebook / export / clear →

The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope