Why languages, emoji, and JSON change token counts

Measure three different things

Unicode code points describe text, UTF-8 bytes describe its storage, and tokens describe how a particular vocabulary represents it. One visible emoji can contain several code points; a single code point can require several bytes or tokens.

An English rule of thumb such as four characters per token is not a multilingual budgeting rule. Count the actual prompt in the encoding you will use.

Compare without changing the input

Load Languages & emoji and use Compare encodings. Both counts come from exactly the same text. Then try your real Spanish, Japanese, Korean, or Chinese prompt; do not assume the relationship stays constant across samples.

JSON adds keys, punctuation, quoting, and whitespace. Compare a formatted object with a compact version. Removing whitespace can reduce bytes, but the token change depends on the encoding and surrounding text; always measure.

import { encode } from 'gpt-tokenizer/encoding/cl100k_base';
// npm install gpt-tokenizer@4.1.0
const samples = [
  'Hello, world!', '你好，世界！',
  '¡Hola, mundo!', 'こんにちは、世界！',
  '안녕하세요, 세계!', '👨‍👩‍👧‍👦'
];
for (const text of samples) {
  const ids = encode(text);
  console.log(text, ids.length);
}

Optimize meaning, not appearance

First remove unnecessary fields, repeated documents, and irrelevant history. Preserve task constraints and evidence. A shorter-looking prompt is not necessarily fewer tokens, and fewer tokens do not guarantee an equally good answer.

Keep a small multilingual regression set. After a prompt change, compare token counts and task correctness together. Do not use these two encodings as exact counters for unrelated model tokenizers.