Tokens and tokenizers: from text to model input

A token is not a word

A language model receives token IDs rather than a string of words. A tokenizer maps text to those integers using a vocabulary and encoding rules. A common word can be one token; an unfamiliar name may be several. Spaces, punctuation, and newlines also matter.

Byte Pair Encoding (BPE) builds useful subword units through learned merge rules. The visible fragments are a debugging aid, not a linguistic segmentation of the sentence.

Run a reproducible example

Choose cl100k_base and enter Hello, world! exactly, including the comma and space. The result is four IDs: 9906, 11, 1917, 0. Select each colored token to inspect its fragment. Then remove the comma or change capitalization and compare the sequence.

Switching to o200k_base changes the vocabulary and may change IDs or counts. An ID only has meaning with its encoding. Keep both the encoding name and IDs when exporting a result.

import { encode, decode } from
  'gpt-tokenizer/encoding/cl100k_base';

const text = 'Hello, world!';
const ids = encode(text);
// [9906, 11, 1917, 0]
console.log(ids.length); // 4
console.log(decode(ids) === text); // true

Verify before integrating

Decode the entire sequence and compare it with the original text. A single token may split a UTF-8 character and display a replacement character on its own, even when the full sequence decodes correctly.

This tool counts encoded text. It does not reproduce an entire API request: role markers, tool definitions, media, and other service-specific fields may add usage. Compare your local estimate with the API response before using it for billing or hard limits.