RAGチャンク実験室

Assumptions, implementation & acceptance

Chunk size and overlap are Unicode code points, not tokens or UTF-8 bytes. Reconstruct the input without duplicating overlaps; then separately count tokens.

Reviewed counting units, statistical assumptions and numerical examples. Provider accounting and actual GPU allocation still require external measurement.

Before running: predict a boundary case

Chunk size and overlap are Unicode code points, not tokens or UTF-8 bytes. Reconstruct the input without duplicating overlaps; then separately count tokens.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node rag-chunker.mjs. npm install gpt-tokenizer@4.1.0

import {encode as rawEncode} from 'gpt-tokenizer/encoding/cl100k_base';
import {promptDiff,inspectDataset,chunkDocument,compareScores,estimateVRAM,samplingLab,speculativeLab} from './research-core.mjs';
const encode=s=>rawEncode(s,{disallowedSpecial:new Set()});
const text='你好🌍 abc '.repeat(8);
const result=chunkDocument(text,16,5,'fixed',encode);
const rebuilt=result.chunks.map(c=>Array.from(c.text).slice(c.overlapWithPrevious).join('')).join('');
console.log(rebuilt===text); // true

Test and transfer

Chunk size and overlap are Unicode code points, not tokens or UTF-8 bytes. Reconstruct the input without duplicating overlaps; then separately count tokens.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.