토큰과 토크나이저: 텍스트에서 모델 입력까지

Assumptions, implementation & acceptance

Round-trip decode(encode(text)) equals the original text. Individual tokens may split UTF-8 sequences; do not require each token to decode independently.

Shared code, English mathematical explanation and key multilingual numeric passages reviewed. This is not a full translation proofread.

Before running: predict a boundary case

Round-trip decode(encode(text)) equals the original text. Individual tokens may split UTF-8 sequences; do not require each token to decode independently.

Minimal executable example

Save the example and linked modules in one folder. Use Node.js 22 or newer, then run node tokens-and-tokenizers.mjs. npm install gpt-tokenizer@4.1.0

import { encode, decode } from
  'gpt-tokenizer/encoding/cl100k_base';

const text = 'Hello, world!';
const ids = encode(text);
// [9906, 11, 1917, 0]
console.log(ids.length); // 4
console.log(decode(ids) === text); // true

Test and transfer

Round-trip decode(encode(text)) equals the original text. Individual tokens may split UTF-8 sequences; do not require each token to decode independently.

Record input, code version, expected result and observed result in the notebook below. Then construct a counterexample that violates the assumption above and explain why the original conclusion no longer follows.