Comparación de evaluaciones pareadas

Assumptions, implementation & acceptance

Resample paired independent seed/run means, not correlated rows from one run. Percentile bootstrap is not a causal claim or a p-value; a degenerate fixture can produce a zero-width interval.

Reviewed counting units, statistical assumptions and numerical examples. Provider accounting and actual GPU allocation still require external measurement.

Before running: predict a boundary case

Resample paired independent seed/run means, not correlated rows from one run. Percentile bootstrap is not a causal claim or a p-value; a degenerate fixture can produce a zero-width interval.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node eval-compare.mjs. npm install gpt-tokenizer@4.1.0

import {encode as rawEncode} from 'gpt-tokenizer/encoding/cl100k_base';
import {promptDiff,inspectDataset,chunkDocument,compareScores,estimateVRAM,samplingLab,speculativeLab} from './research-core.mjs';
const encode=s=>rawEncode(s,{disallowedSpecial:new Set()});
const result=compareScores('id,a,b\nx,1,2\ny,4,5\nz,0,1','higher',42);
console.log(result.delta,result.ci); // 1 [1,1]; degenerate fixture, not evidence of zero uncertainty in general

Test and transfer

Resample paired independent seed/run means, not correlated rows from one run. Percentile bootstrap is not a causal claim or a p-value; a degenerate fixture can produce a zero-width interval.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.