Candidate starvation
Even an oracle cannot recover documents excluded before reranking.
Local calculation / simulationLocal educational experiment. No external API calls. Synthetic fixtures are not production measurements.
Study guide & lesson outline
LEARNING CONTRACTUnderstand → implement → investigate
Goal: explain the mechanism in this lesson, reproduce one example and test a failure case. Reading alone does not mark mastery.
- Understand: state assumptions, shapes/units and a counterexample.
- Implement: reproduce a numerical example and inspect boundaries.
- Investigate: compare one alternative under a matched budget.
02 · Predict & configure
Technical scenario notes, parameter names and reference answers are currently in English.
03 · Inspect the evidence
Methods & references
Lexical score counts matching query terms (not BM25). Dense score is cosine on supplied 2D vectors, not a live embedding model. RRF combines ranks. Optional reranking uses known relevance labels as an oracle upper-bound, not a trained reranker. Empty query ties preserve document order.
Text / 原始公式
score=w/(k+lexicalRank)+(1−w)/(k+denseRank); DCG=Σ(2^rel−1)/log₂(rank+1)
An oracle reranker leaks labels intentionally; never report it as learned model performance.
Reciprocal Rank Fusion · SIGIR 2009 ↗Practice questions
retrieval-5 / 01What does “Candidate starvation” test?
Even an oracle cannot recover documents excluded before reranking.
Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗
Open lab →retrieval-5 / 02Which computation produces the result in “Candidate starvation”?
Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗
Open lab →retrieval-5 / 03What is the baseline configuration for “Candidate starvation”?
{"query":"KV cache memory","documents":"[\n {\n \"id\": \"D1\",\n \"text\": \"KV cache stores keys and values for generation\",\n \"vector\": [\n 1,\n 0.1\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D2\",\n \"text\": \"Cache restaurant menu and opening hours\",\n \"vector\": [\n 0.1,\n 1\n ],\n \"relevance\": 0\n },\n {\n \"id\": \"D3\",\n \"text\": \"Grouped query attention reduces memory\",\n \"vector\": [\n 0.95,\n 0.2\n ],\n \"relevance\": 2\n },\n {\n \"id\": \"D4\",\n \"text\": \"KV cache memory footprint depends on sequence length\",\n \"vector\": [\n 0.8,\n 0.3\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D5\",\n \"text\": \"Tokenization splits input text\",\n \"vector\": [\n 0.4,\n 0.8\n ],\n \"relevance\": 0\n }\n]","x":1,"y":0.1,"weight":0.5,"k":60,"retrieve":1,"top":3,"rerank":1}
Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗
Open lab →retrieval-5 / 04What numerical result should you check for “Candidate starvation”?
{"Recall@k":0.3333333333333333,"nDCG@k":0.541942129480027}
Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗
Open lab →retrieval-5 / 05How would you run a controlled comparison for “Candidate starvation”?
Keep the supplied fixture and other settings fixed; change weight. Compare Recall@k and nDCG@k while varying candidate pool and fusion weight.
Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗
Open lab →retrieval-5 / 06What failure could invalidate a conclusion from “Candidate starvation”?
An oracle reranker leaks labels intentionally; never report it as learned model performance.
Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗
Open lab →retrieval-5 / 07Is “Candidate starvation” a benchmark of a deployed language model?
No. Lexical score counts matching query terms (not BM25). Dense score is cosine on supplied 2D vectors, not a live embedding model. RRF combines ranks. Optional reranking uses known relevance labels as an oracle upper-bound, not a trained reranker. Empty query ties preserve document order.
Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗
Open lab →retrieval-5 / 08Which evidence should you report for “Candidate starvation”?
Compare Recall@k and nDCG@k while varying candidate pool and fusion weight. Export inputs, engine, result and method; distinguish synthetic fixtures from observed production data.
Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗
Open lab →retrieval-5 / 09How can another researcher reproduce “Candidate starvation”?
Open retrieval-5, reset the preset, retain the exact inputs and run. The calculation is deterministic for identical inputs.
Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗
Open lab →retrieval-5 / 10What is the next validation step after “Candidate starvation”?
An oracle reranker leaks labels intentionally; never report it as learned model performance. Test the hypothesis on a representative held-out workload and document a baseline, success metric and stopping rule before claiming deployment readiness.
Original teaching answer / local derivation. Method reference: Reciprocal Rank Fusion · SIGIR 2009 ↗
Open lab →Assumptions, implementation & acceptance
Weighted reciprocal-rank fusion uses ranks starting at 1. Supplied relevance is an oracle reranker, not a trained model. Zero-vector cosine uses a software sentinel; labels must not enter a real retrieval index.
Engine formulas, scenario presets and numerical safeguards reviewed. Seeded synthetic inputs are not a production benchmark.
Before running: predict a boundary case
Even an oracle cannot recover documents excluded before reranking.
Hint: isolate the assumption
Keep the baseline input fixed, change one parameter or boundary, and compare both the numerical result and the claimed units. A passing example verifies this fixture only.
Minimal executable example
Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node retrieval-5.mjs. No third-party runtime dependencies.
import {calculate,network} from './academy-core.mjs';
const params={
"query": "KV cache memory",
"documents": "[\n {\n \"id\": \"D1\",\n \"text\": \"KV cache stores keys and values for generation\",\n \"vector\": [\n 1,\n 0.1\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D2\",\n \"text\": \"Cache restaurant menu and opening hours\",\n \"vector\": [\n 0.1,\n 1\n ],\n \"relevance\": 0\n },\n {\n \"id\": \"D3\",\n \"text\": \"Grouped query attention reduces memory\",\n \"vector\": [\n 0.95,\n 0.2\n ],\n \"relevance\": 2\n },\n {\n \"id\": \"D4\",\n \"text\": \"KV cache memory footprint depends on sequence length\",\n \"vector\": [\n 0.8,\n 0.3\n ],\n \"relevance\": 3\n },\n {\n \"id\": \"D5\",\n \"text\": \"Tokenization splits input text\",\n \"vector\": [\n 0.4,\n 0.8\n ],\n \"relevance\": 0\n }\n]",
"x": 1,
"y": 0.1,
"weight": 0.5,
"k": 60,
"retrieve": 1,
"top": 3,
"rerank": 1
};
const result=calculate('retrieval',params);
console.log(result.metrics);Test and transfer
Even an oracle cannot recover documents excluded before reranking.
Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.
Download complete kit (.tar.gz) · Complete reference kit and regression checksPredict, test and keep evidence
Before revealing an answer, predict one result. After the exercise, record the discrepancy and a transfer test. Saved locally across language editions.
Open notebook / export / clear →The mathematical review scope is recorded per lesson. It does not establish empirical paper reproduction; report suspected errors in your notes. Review scope