プロンプト差分とトークン変化

Assumptions, implementation & acceptance

Line edits and token deltas are different quantities. Whitespace affects tokenization; a line edit is not semantic equivalence.

Reviewed counting units, statistical assumptions and numerical examples. Provider accounting and actual GPU allocation still require external measurement.

Before running: predict a boundary case

Line edits and token deltas are different quantities. Whitespace affects tokenization; a line edit is not semantic equivalence.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node prompt-diff.mjs. npm install gpt-tokenizer@4.1.0

import {encode as rawEncode} from 'gpt-tokenizer/encoding/cl100k_base';
import {promptDiff,inspectDataset,chunkDocument,compareScores,estimateVRAM,samplingLab,speculativeLab} from './research-core.mjs';
const encode=s=>rawEncode(s,{disallowedSpecial:new Set()});
const result=promptDiff('a\nb\nc','a\nx\nc',encode);
console.log([result.inserted,result.removed,result.unchanged]); // [1,1,2]

Test and transfer

Line edits and token deltas are different quantities. Whitespace affects tokenization; a line edit is not semantic equivalence.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.