1. Why can harder negatives be harmful?
Show answer & scoring criteria
They may be false negatives or annotation errors. High probability gives them substantial repulsive gradient.
Contrastive Learning Lab
Inspect InfoNCE, temperature and the exact gradient applied to a possible false negative.
Local calculations and synthetic examples. No paid API or live-model benchmark.Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
01 · Understand & derive
Prerequisites · Softmax, cross-entropy, derivatives.
The positive is pulled up in similarity while negatives are pushed down by gradient descent. A high-scoring negative gets a stronger push. If that negative is actually relevant, the training signal conflicts with the desired semantics.
Text / 原始公式
L = −log[exp(s₊/τ) / Σⱼ exp(sⱼ/τ)] ∂L/∂sᵢ = (pᵢ − 1[i=positive]) / τ positive index is zero-based; excluded candidates leave the denominator
Step-by-step derivation and interpretation
- Write
₊/τ+log Σexp(sⱼ/τ). Differentiating the log-sum-exp gives pᵢ/τ, so ∂L/∂ =+])/τ. The gradients sum to zero. - Gradient descent reduces the score of a negative with positive derivative. A false negative is therefore pushed away even when it should be relevant. A hard negative is not automatically a valid negative.
- Temperature changes both the 1/τ scale and the softmax probabilities. Saturation can offset the larger scale; do not assume every gradient monotonically increases as temperature falls.
Common pitfall: Removing candidates makes the classification task easier; reduced loss alone does not prove improved representation learning.
02 · Check your understanding
With 3 equal similarity scores and one positive, what is InfoNCE loss in nats?
Show derivation
−ln(1/3) = ln(3) ≈ 1.098612
03 · Run a controlled experiment
Experiment assignment
Treat index 1 as a known false negative. Run with all candidates, then exclude index 1 and compare its gradient and the positive probability. Sweep temperature 0.05, 0.2 and 1.
Controlled experiment protocol
- Use default scores with positive index 0. Record the probability and gradient of candidate 1 before excluding anything.
- Assume candidate 1 is independently labeled relevant; exclude it and rerun. Its gradient becomes zero and the positive probability rises.
- Repeat at temperatures 0.05,0.2,1. Submit a gradient comparison and propose a fixed held-out retrieval evaluation for an actual encoder.
Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.
Show a hint
Removing a candidate changes task difficulty. A lower loss after removal cannot establish improved embeddings; evaluate a fixed held-out retrieval task.
04 · Inspect evidence
Run the experiment to see evidence.
Only compare runs with controlled inputs. Exports include inputs and methods.
05 · Defend your reasoning
Self-assessment: explain the mechanism, identify assumptions, and propose a discriminating test. No automatic LLM grading.
2. Does a lower temperature always increase every gradient?
Show answer & scoring criteria
No. The 1/τ factor grows, but softmax probabilities also change and can saturate. Inspect both terms.
3. How would you test a new negative-mining strategy?
Show answer & scoring criteria
Keep encoder, data budget and evaluation fixed, audit negative labels, compare multiple seeds and measure held-out retrieval rather than only training loss.
Assumptions & limits
Exact single-anchor loss and score gradients with supplied similarities; no encoder is trained. The tool cannot identify false negatives automatically. Compare retrieval quality using a separate dataset.
References
arXiv:1807.03748 · 2018
Representation Learning with Contrastive Predictive Coding
van den Oord, Li & Vinyals · Year denotes first arXiv submission
Read: Section 2.3: InfoNCE; Appendix A.1: mutual-information argument.
Reading checkpoint: Track the negative-sampling assumptions before interpreting a mutual-information bound.
BibTeX
@misc{arxiv180703748,
title = {Representation Learning with Contrastive Predictive Coding},
year = {2018},
eprint = {1807.03748},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1807.03748}
}arXiv:2002.05709 · 2020
A Simple Framework for Contrastive Learning of Visual Representations
Chen et al. · Year denotes first arXiv submission
Read: Framework, augmentation ablations, projection head and contrastive objective.
Reading checkpoint: List which augmentation invariances are useful or harmful for your domain. A lower training loss is not the only evaluation.
BibTeX
@misc{arxiv200205709,
title = {A Simple Framework for Contrastive Learning of Visual Representations},
year = {2020},
eprint = {2002.05709},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2002.05709}
}Assumptions, implementation & acceptance
Exact single-anchor loss and score gradients with supplied similarities; no encoder is trained. The tool cannot identify false negatives automatically. Compare retrieval quality using a separate dataset.
Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.
Before running: predict a boundary case
Treat index 1 as a known false negative. Run with all candidates, then exclude index 1 and compare its gradient and the positive probability. Sweep temperature 0.05, 0.2 and 1.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node contrastive-learning.mjs. No third-party runtime dependencies.
import {runPhd} from './phd-core.mjs';
const params={
"scores": "0.8,0.75,0.2,-0.1",
"positive": 0,
"temperature": 0.2,
"exclude": -1
};
console.log(runPhd('contrastive-learning',params));Test and transfer
Treat index 1 as a known false negative. Run with all candidates, then exclude index 1 and compare its gradient and the positive probability. Sweep temperature 0.05, 0.2 and 1.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Download complete kit (.tar.gz) · Complete reference kit and regression checksPredict, test and keep evidence
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
Open notebook / export / clear →The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope