1. Why can harder negatives be harmful?
参考解答と評価観点
They may be false negatives or annotation errors. High probability gives them substantial repulsive gradient.
対照学習ラボ
Inspect InfoNCE, temperature and the exact gradient applied to a possible false negative.
ローカル計算と合成データです。有料 API や実モデルのベンチマークは使用しません。技術本文は英語と中国語に対応しています。この言語では英語の技術本文を表示します。
Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
01 · 理解と導出
前提知識 · Softmax, cross-entropy, derivatives.
The positive is pulled up in similarity while negatives are pushed down by gradient descent. A high-scoring negative gets a stronger push. If that negative is actually relevant, the training signal conflicts with the desired semantics.
Text / 原始公式
L = −log[exp(s₊/τ) / Σⱼ exp(sⱼ/τ)] ∂L/∂sᵢ = (pᵢ − 1[i=positive]) / τ positive index is zero-based; excluded candidates leave the denominator
Step-by-step derivation and interpretation
- Write
₊/τ+log Σexp(sⱼ/τ). Differentiating the log-sum-exp gives pᵢ/τ, so ∂L/∂ =+])/τ. The gradients sum to zero. - Gradient descent reduces the score of a negative with positive derivative. A false negative is therefore pushed away even when it should be relevant. A hard negative is not automatically a valid negative.
- Temperature changes both the 1/τ scale and the softmax probabilities. Saturation can offset the larger scale; do not assume every gradient monotonically increases as temperature falls.
Common pitfall: Removing candidates makes the classification task easier; reduced loss alone does not prove improved representation learning.
02 · 理解を確認
With 3 equal similarity scores and one positive, what is InfoNCE loss in nats?
導出を表示
−ln(1/3) = ln(3) ≈ 1.098612
03 · 対照実験を実行
実験課題
Treat index 1 as a known false negative. Run with all candidates, then exclude index 1 and compare its gradient and the positive probability. Sweep temperature 0.05, 0.2 and 1.
Controlled experiment protocol
- Use default scores with positive index 0. Record the probability and gradient of candidate 1 before excluding anything.
- Assume candidate 1 is independently labeled relevant; exclude it and rerun. Its gradient becomes zero and the positive probability rises.
- Repeat at temperatures 0.05,0.2,1. Submit a gradient comparison and propose a fixed held-out retrieval evaluation for an actual encoder.
Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.
ヒントを表示
Removing a candidate changes task difficulty. A lower loss after removal cannot establish improved embeddings; evaluate a fixed held-out retrieval task.
04 · 根拠を確認
実行すると結果が表示されます。
条件を統制して実行結果を比較してください。出力に入力と手法を含みます。
05 · 推論を説明
自己評価：仕組みと前提を説明し、仮説を区別できる実験を提案してください。LLM 自動採点はありません。
2. Does a lower temperature always increase every gradient?
参考解答と評価観点
No. The 1/τ factor grows, but softmax probabilities also change and can saturate. Inspect both terms.
3. How would you test a new negative-mining strategy?
参考解答と評価観点
Keep encoder, data budget and evaluation fixed, audit negative labels, compare multiple seeds and measure held-out retrieval rather than only training loss.
前提と限界
Exact single-anchor loss and score gradients with supplied similarities; no encoder is trained. The tool cannot identify false negatives automatically. Compare retrieval quality using a separate dataset.
参考文献
arXiv:1807.03748 · 2018
Representation Learning with Contrastive Predictive Coding
van den Oord, Li & Vinyals · Year denotes first arXiv submission
Read: Section 2.3: InfoNCE; Appendix A.1: mutual-information argument.
Reading checkpoint: Track the negative-sampling assumptions before interpreting a mutual-information bound.
BibTeX
@misc{arxiv180703748,
title = {Representation Learning with Contrastive Predictive Coding},
year = {2018},
eprint = {1807.03748},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1807.03748}
}arXiv:2002.05709 · 2020
A Simple Framework for Contrastive Learning of Visual Representations
Chen et al. · Year denotes first arXiv submission
Read: Framework, augmentation ablations, projection head and contrastive objective.
Reading checkpoint: List which augmentation invariances are useful or harmful for your domain. A lower training loss is not the only evaluation.
BibTeX
@misc{arxiv200205709,
title = {A Simple Framework for Contrastive Learning of Visual Representations},
year = {2020},
eprint = {2002.05709},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2002.05709}
}Assumptions, implementation & acceptance
Exact single-anchor loss and score gradients with supplied similarities; no encoder is trained. The tool cannot identify false negatives automatically. Compare retrieval quality using a separate dataset.
Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.
Before running: predict a boundary case
Treat index 1 as a known false negative. Run with all candidates, then exclude index 1 and compare its gradient and the positive probability. Sweep temperature 0.05, 0.2 and 1.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node contrastive-learning.mjs. No third-party runtime dependencies.
import {runPhd} from './phd-core.mjs';
const params={
"scores": "0.8,0.75,0.2,-0.1",
"positive": 0,
"temperature": 0.2,
"exclude": -1
};
console.log(runPhd('contrastive-learning',params));Test and transfer
Treat index 1 as a known false negative. Run with all candidates, then exclude index 1 and compare its gradient and the positive probability. Sweep temperature 0.05, 0.2 and 1.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Download complete kit (.tar.gz) · Complete reference kit and regression checksPredict, test and keep evidence
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
Open notebook / export / clear →The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope