1. Why can harder negatives be harmful?
참고 답변과 평가 기준
They may be false negatives or annotation errors. High probability gives them substantial repulsive gradient.
대조 학습
Inspect InfoNCE, temperature and the exact gradient applied to a possible false negative.
로컬 계산과 합성 예제입니다. 유료 API나 실제 모델 벤치마크를 사용하지 않습니다.기술 본문은 영어와 중국어로 제공됩니다. 이 언어에서는 영어 기술 본문을 표시합니다.
Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
01 · 이해와 유도
사전 지식 · Softmax, cross-entropy, derivatives.
The positive is pulled up in similarity while negatives are pushed down by gradient descent. A high-scoring negative gets a stronger push. If that negative is actually relevant, the training signal conflicts with the desired semantics.
Text / 原始公式
L = −log[exp(s₊/τ) / Σⱼ exp(sⱼ/τ)] ∂L/∂sᵢ = (pᵢ − 1[i=positive]) / τ positive index is zero-based; excluded candidates leave the denominator
Step-by-step derivation and interpretation
- Write
₊/τ+log Σexp(sⱼ/τ). Differentiating the log-sum-exp gives pᵢ/τ, so ∂L/∂ =+])/τ. The gradients sum to zero. - Gradient descent reduces the score of a negative with positive derivative. A false negative is therefore pushed away even when it should be relevant. A hard negative is not automatically a valid negative.
- Temperature changes both the 1/τ scale and the softmax probabilities. Saturation can offset the larger scale; do not assume every gradient monotonically increases as temperature falls.
Common pitfall: Removing candidates makes the classification task easier; reduced loss alone does not prove improved representation learning.
02 · 이해도 확인
With 3 equal similarity scores and one positive, what is InfoNCE loss in nats?
유도 과정 보기
−ln(1/3) = ln(3) ≈ 1.098612
03 · 통제 실험 실행
실험 과제
Treat index 1 as a known false negative. Run with all candidates, then exclude index 1 and compare its gradient and the positive probability. Sweep temperature 0.05, 0.2 and 1.
Controlled experiment protocol
- Use default scores with positive index 0. Record the probability and gradient of candidate 1 before excluding anything.
- Assume candidate 1 is independently labeled relevant; exclude it and rerun. Its gradient becomes zero and the positive probability rises.
- Repeat at temperatures 0.05,0.2,1. Submit a gradient comparison and propose a fixed held-out retrieval evaluation for an actual encoder.
Deliver: input configurations, exported results, controlled variables, observations, alternative explanations and the next validation step.
힌트 보기
Removing a candidate changes task difficulty. A lower loss after removal cannot establish improved embeddings; evaluate a fixed held-out retrieval task.
04 · 근거 확인
실행하면 결과가 표시됩니다.
변수를 통제하여 결과를 비교하세요. 내보내기에는 입력과 방법이 포함됩니다.
05 · 추론 설명
자기 평가: 원리와 가정을 설명하고 구별 가능한 실험을 제안하세요. LLM 자동 채점은 없습니다.
2. Does a lower temperature always increase every gradient?
참고 답변과 평가 기준
No. The 1/τ factor grows, but softmax probabilities also change and can saturate. Inspect both terms.
3. How would you test a new negative-mining strategy?
참고 답변과 평가 기준
Keep encoder, data budget and evaluation fixed, audit negative labels, compare multiple seeds and measure held-out retrieval rather than only training loss.
가정과 한계
Exact single-anchor loss and score gradients with supplied similarities; no encoder is trained. The tool cannot identify false negatives automatically. Compare retrieval quality using a separate dataset.
참고 문헌
arXiv:1807.03748 · 2018
Representation Learning with Contrastive Predictive Coding
van den Oord, Li & Vinyals · Year denotes first arXiv submission
Read: Section 2.3: InfoNCE; Appendix A.1: mutual-information argument.
Reading checkpoint: Track the negative-sampling assumptions before interpreting a mutual-information bound.
BibTeX
@misc{arxiv180703748,
title = {Representation Learning with Contrastive Predictive Coding},
year = {2018},
eprint = {1807.03748},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1807.03748}
}arXiv:2002.05709 · 2020
A Simple Framework for Contrastive Learning of Visual Representations
Chen et al. · Year denotes first arXiv submission
Read: Framework, augmentation ablations, projection head and contrastive objective.
Reading checkpoint: List which augmentation invariances are useful or harmful for your domain. A lower training loss is not the only evaluation.
BibTeX
@misc{arxiv200205709,
title = {A Simple Framework for Contrastive Learning of Visual Representations},
year = {2020},
eprint = {2002.05709},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2002.05709}
}Assumptions, implementation & acceptance
Exact single-anchor loss and score gradients with supplied similarities; no encoder is trained. The tool cannot identify false negatives automatically. Compare retrieval quality using a separate dataset.
Objective, derivatives, worked examples and implementation reviewed. A passing finite-difference or manifest check is not a paper reproduction certificate.
Before running: predict a boundary case
Treat index 1 as a known false negative. Run with all candidates, then exclude index 1 and compare its gradient and the positive probability. Sweep temperature 0.05, 0.2 and 1.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node contrastive-learning.mjs. No third-party runtime dependencies.
import {runPhd} from './phd-core.mjs';
const params={
"scores": "0.8,0.75,0.2,-0.1",
"positive": 0,
"temperature": 0.2,
"exclude": -1
};
console.log(runPhd('contrastive-learning',params));Test and transfer
Treat index 1 as a known false negative. Run with all candidates, then exclude index 1 and compare its gradient and the positive probability. Sweep temperature 0.05, 0.2 and 1.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Download complete kit (.tar.gz) · Complete reference kit and regression checksPredict, test and keep evidence
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
Open notebook / export / clear →The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope