JSONLデータセット検査

Assumptions, implementation & acceptance

Exact normalized text overlap is a leakage signal, not proof that all near duplicates are detected. Token counts exclude provider chat-template overhead.

Reviewed counting units, statistical assumptions and numerical examples. Provider accounting and actual GPU allocation still require external measurement.

Before running: predict a boundary case

Exact normalized text overlap is a leakage signal, not proof that all near duplicates are detected. Token counts exclude provider chat-template overhead.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node jsonl-inspector.mjs. npm install gpt-tokenizer@4.1.0

import {encode as rawEncode} from 'gpt-tokenizer/encoding/cl100k_base';
import {promptDiff,inspectDataset,chunkDocument,compareScores,estimateVRAM,samplingLab,speculativeLab} from './research-core.mjs';
const encode=s=>rawEncode(s,{disallowedSpecial:new Set()});
const result=inspectDataset('{"text":"x"}\n{"text":"x"}', '{"text":"x"}',50,encode);
console.log([result.train.duplicates,result.crossSplitMatches]); // [1,1]

Test and transfer

Exact normalized text overlap is a leakage signal, not proof that all near duplicates are detected. Token counts exclude provider chat-template overhead.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.