Self-improving agents: reward gains are not capability gains

Start with a prediction

A coding agent’s rubric score rises from 0.70 to 0.95 after you change its instructions. Hidden-test success falls from 60% to 50%. Has the agent improved? Write your criterion before looking at the experiment.

Prerequisites: sample means, variance, evaluation splits and basic RL rewards. By the end, you should distinguish policy updates from runtime updates, detect a reward group with no relative signal, and design a comparison that separates better scoring from better task completion.

Three papers, three different meanings of improvement

The engineering distinction

In a coding assistant, a harness change might improve how test output is retained between turns. A policy update changes the model’s parameters. A judge update changes what receives reward. These interventions can all raise a dashboard number, but they support different conclusions. Product behavior alone does not reveal which method a proprietary assistant uses.

One numerical mechanism: reward discrimination

For a group of G scalar rewards, use population variance. All quantities below are dimensionless. This is a teaching normalization, not a complete GRPO loss: it omits token-level likelihood ratios, clipping and regularization.

For rewards [0, 1], the mean is 0.5, variance 0.25 and advantages [−1, 1]. For [1, 1, 1], variance is zero; our implementation returns zero rather than dividing by zero. This says there is no within-group relative signal, not that every answer is good. Uniformly incorrect answers have the same problem. A variance floor can stabilize near-zero variance but does not create useful ranking information.

Run the local experiment

Predict which candidate is selected. The baseline solves 6/10 development tasks in 20 calls. A proxy-optimized candidate solves 5/10 with a higher rubric score. A second candidate solves 8/10 in 20 calls. Our deliberately simple development gate requires strictly more correct tasks and no extra calls. It is not a significance test.

Ready. Everything runs locally; no model calls or uploads.

Expected: seven PASS checks. Reject the proxy-only improvement, accept the second candidate on development, then observe 6/10 versus 6/10 on the separate synthetic audit. The fixture demonstrates the logic of a failure, not its frequency in real agents.

Download complete implementation · Node.js 22+, no dependencies, no randomness or seed needed.

node --input-type=module -e "import {run} from './adaptive-harness.mjs'; console.log(run())"

// Original deterministic teaching fixture. No model training or external calls.
export function advantages(rewards){
 if(!rewards.length||rewards.some(x=>!Number.isFinite(x))) throw Error('Finite nonempty rewards required');
 const mean=rewards.reduce((a,b)=>a+b,0)/rewards.length;
 const variance=rewards.reduce((a,b)=>a+(b-mean)**2,0)/rewards.length;
 return rewards.map(x=>variance===0?0:(x-mean)/Math.sqrt(variance));
}
export function decide(base,candidate){
 for(const r of [base,candidate]) if(!Number.isInteger(r.correct)||!Number.isInteger(r.n)||r.n<1||r.correct<0||r.correct>r.n||!Number.isFinite(r.calls)||r.calls<0) throw Error('Invalid measurement');
 if(base.n!==candidate.n) throw Error('Use matched tasks');
 return candidate.correct>base.correct&&candidate.calls<=base.calls;
}
export function run(){
 const checks=[];const check=(name,ok)=>{if(!ok)throw Error(name);checks.push('PASS: '+name)};
 check('uniform rewards have no relative signal',advantages([1,1,1]).every(x=>x===0));
 const a=advantages([0,1]);check('population normalization',a[0]===-1&&a[1]===1);
 const baseline={n:10,correct:6,calls:20};
 const scoreHack={n:10,correct:5,calls:20,proxy:0.95};
 const candidate={n:10,correct:8,calls:20,proxy:0.8};
 check('reject proxy-only gain',!decide(baseline,scoreHack));
 check('accept development improvement at matched budget',decide(baseline,candidate));
 check('reject extra compute',!decide(baseline,{...candidate,calls:21}));
 let rejects=0;for(const bad of [[],[NaN],[Infinity]])try{advantages(bad)}catch{rejects++}
 check('reject invalid reward vectors',rejects===3);
 // Only inspected after selection: development success does not guarantee transfer.
 const heldOut={baselineCorrect:6,candidateCorrect:6,n:10};
 check('no demonstrated held-out gain',heldOut.baselineCorrect===heldOut.candidateCorrect);
 return {checks,baseline,scoreHack,candidate,decision:'candidate selected on development only',heldOut,conclusion:'A development gain is not evidence of transferable capability.'};
}

From fixture to a fair research experiment

1. Pre-register task success and the budget: model tokens, tool calls, wall time and any evaluator calls. Equal tool counts alone do not imply equal compute.
2. Freeze a model checkpoint. Run baseline and candidate on matched development tasks. Keep evaluator version and task fixtures constant.
3. Inspect failure cases, then select the configuration. Include a no-change control and an evaluator-only change.
4. Lock the configuration before opening held-out tasks. Record paired per-task outcomes and repeat stochastic runs with declared seeds.
5. Report confidence intervals, failures by category and transfer to a different harness. An improvement on ten tasks is too weak for a general capability claim.

A useful ablation matrix crosses two model checkpoints with two harnesses under one frozen evaluator. This separates checkpoint effects, harness effects and their interaction. If you repeatedly tune against the audit set, retire it and obtain a fresh audit set. Do not relabel the same exposed data “held out.”

Interview practice · original Token Lab questions

Transfer exercise

A candidate solves 8/10 tasks but uses 30 calls instead of 20. Modify the fixture to compare success under a fixed total-call budget. Explain why simply dividing success rate by calls can reward a system that stops too early.

Continue the learning path

First: implement a bounded agent harness → Next: verification and inference budgets → RL knowledge library

Source scope: the three linked arXiv abstracts and version histories were checked on 26 October 2026. No conference acceptance or independent reproduction is claimed. Read the full papers before reproducing their training protocols.