Why does the same text cost more in Vietnamese?

The same document costs 1.45x more in Vietnamese than in English — 237 tokens against 163 for identical meaning — so a Vietnamese user pays 45% more to send the same request. Under the older cl100k_base tokenizer it was 2.69x. That is the headline finding and also the good news: the gap is real, and it has narrowed sharply.

LLM pricing is quoted per token, never per word or per idea. A token is whatever the tokenizer decides it is, and every production BPE tokenizer was fitted to a corpus that is mostly English. The consequence is a per-language tax that nobody puts on the pricing page. Our users write in Vietnamese, so we measured it.

Hardware: Apple M3, 16 GB, macOS 26.4.1. Node v23.5.0 arm64, gpt-tokenizer 4.0.0, cross-checked against js-tiktoken 1.0.21. Everything runs offline — the token counts are computed locally by the same BPE merge tables the API uses, not estimated and not billed. All fourteen counts matched exactly between the two libraries, so the numbers below are not an artifact of one implementation.

On the corpus. I wrote two short documents — a support-and-billing notice (143 English words) and a product-care notice (49 words) — and then wrote each one in Vietnamese, Chinese, Japanese, Thai, French and German myself, keeping the same eleven statements in the same order. Japanese renders one of those statements as two sentences, and Thai does not mark sentence ends with a period; the content is matched, not the punctuation. I am not a professional translator; the wording is plain and idiomatic enough to be fair, but a certified translation would move these numbers by a few percent. It would not move them by a factor. Two independent documents are reported precisely so you can see which findings survive a change of subject matter.

The short answer

• Vietnamese costs 1.45x more tokens than English for the same document (237 against 163) under o200k_base, the encoding used by GPT-4o and current models. Across both test documents the combined figure is 1.37x.
• The old tokenizer was far worse. On cl100k_base the same Vietnamese text was 439 tokens — 2.69x English. Moving to o200k_base cut Vietnamese by 46% and Thai by 59%, while leaving English unchanged at 0.0%.
• Chinese is now essentially at parity with English — 170 tokens against 163, a 1.04x ratio — and it is cheaper than French (1.28x) or German (1.26x). That was not the expected result.
• Stripping Vietnamese accents now makes text more expensive, not less. Removing diacritics raised the count from 237 to 262 tokens (+10.5%) on o200k_base, while on cl100k_base the identical operation saved 28.5%. The common advice reversed when the tokenizer changed.
• A 128,000-token context holds 785 copies of the English document and 540 of the Vietnamese one — 0.69x the meaning for the same window, and 0.60x in Thai.

How many tokens is a Vietnamese word?

Usually two, where the English equivalent is one. The mechanism is visible at the level of a single phrase. Here is o200k_base on a greeting, with each token quoted:

"Xin" " ch" "ào" "," " hôm" " nay" " bạn" " có" " khỏe" " không" "?"     11 tokens
"Hello" "," " how" " are" " you" " today" "?"                             7 tokens

Xin chào is two words in Vietnamese and three tokens; Hello is one word and one token. Common vocabulary splits the same way — thanh toán (payment) is 2 tokens against 1 for payment, lời nhắc (reminder) is 3 against 1, and khách hàng (customer) is 2 against 1.

Here is the full corpus. Characters are Unicode code points, bytes are UTF-8:

Tokens per character is the wrong metric to reason about cost, and the table shows why: Chinese has the second-highest tokens-per-character in the list and the lowest total. Han characters carry far more meaning each, so 257 of them say what takes 793 Latin characters. The only fair unit is the whole document, because that is the only place the meaning is held constant.

What is the ratio against English?

Vietnamese moves the most between the two documents — 1.45x on the longer, prose-heavy notice and 1.14x on the short imperative one. Ratios of this kind are a property of the text as much as the language, so treat any single number as ±0.15 and the ordering as the durable part. The ordering is stable across both documents: English cheapest, Chinese next, then the European pair, then Vietnamese, Japanese and Thai.

The European control is the interesting row. French and German pay a 24–26% premium for the same meaning, which is close to Vietnamese on the short document and worse than Chinese on both. This is not a Latin-script-versus-everything-else story. Whatever is happening is about how much of a language's vocabulary made it into the merge table, not about its writing system.

Did o200k fix this?

Largely, yes — and it is worth saying plainly, because the fair version of this article is not an accusation. Vendors did not choose to overcharge non-English users. BPE learns merges from frequency, an English-dominated corpus produces English-dominated merges, and enlarging the vocabulary from roughly 100k to 200k entries buys room for everyone else's words.

The second document reproduces the pattern: Vietnamese −42.3%, Thai −56.6%, English 0.0%. English gained nothing from the new tokenizer, and everyone else gained between 14% and 59%. The entire budget of the vocabulary expansion went to non-English text. That is a deliberate and effective fix, and it means any cost estimate built on cl100k_base numbers — most of what is written about this online — now overstates the penalty by roughly a factor of two.

Do accents cost extra?

No — and removing them costs you 10.5% more tokens. This is the measurement that came out backwards, and it is the most actionable thing here.

The folk advice is that Vietnamese is expensive because of its diacritics: each ế or ạ is three UTF-8 bytes, so surely stripping them saves money. It used to. It no longer does.

The second document agrees: +12.5% on o200k_base, −27.9% on cl100k_base. The reason shows up at word level. Under o200k_base, người is one token and nguoi is two; được is one and duoc is two; những is one and nhung is two. The accented forms are real Vietnamese words that appeared often enough in training to earn their own merges. The unaccented forms are not words in any language, so they fall back to fragments.

So the practical rule is the opposite of the old one. Send correctly accented Vietnamese. It is cheaper, and the model understands it better. If you have a pipeline that normalizes diacritics away before hitting the API — a legacy search index, an SMS gateway, a database column that is still latin1 — it is now costing you money on both sides of the call.

The one caveat is that this is an o200k_base result. If you are calling a model still on cl100k_base, stripping accents does save 28.5%, at the price of text your users find harder to read. Check which encoding your model uses before acting on either number.

What does it cost in money?

Example only, and the price is the part that will go stale first: $2.50 per million input tokens, the published standard input rate for GPT-4o, checked on 31 August 2026. Substitute your own. The shape of the table is what matters.

Sending 100,000 copies of the support notice as input:

Eighteen dollars on a forty-dollar workload. At the scale most teams operate, the absolute number is small enough to ignore and the percentage is not: it is a flat 45% surcharge on every Vietnamese customer interaction, and it compounds with anything that multiplies your token count — retries, few-shot examples, a system prompt repeated on every call.

How much context do you lose?

More than the price difference, because context is a hard wall rather than a meter. Filling a 128,000-token window with copies of the same document:

A Vietnamese RAG system fits 69% of the source material an English one does, at the same window and the same retrieval settings. If you sized your chunk budget on English documents, you are silently retrieving less evidence per query in Vietnamese. This is worth folding into how you chunk documents for retrieval: a 500-token chunk holds about a third less Vietnamese content than the English chunk you tested with, so a fixed token budget quietly changes what a chunk is when the language changes.

Is the tokenizer itself slower?

Yes, and Vietnamese is the worst case in this corpus. Encoding the same corpus repeated 40 times, median of 21 runs, single core:

Vietnamese encodes at roughly half the English rate by either measure. The BPE merge loop does more work when the byte sequence is long and the winning merges are late in the table, which is exactly the multi-byte accented Latin case. In absolute terms this only matters if you tokenize at scale — counting tokens before dispatch, splitting documents, or enforcing a budget across millions of records. At 24 MB/s a gigabyte of Vietnamese text takes about 41 seconds of CPU; at 48 MB/s the English gigabyte takes 21. Note also that o200k_base is slower than cl100k_base everywhere: the larger vocabulary produces fewer tokens but costs more time to find them.

What should you actually do?

Four things, in order of payoff.

Check which encoding your model uses, and count tokens locally before you send. The difference between cl100k_base and o200k_base is a factor of two on Vietnamese, so a budget estimated against the wrong one is not approximately right, it is wrong. Counting offline is free and instant.

Stop stripping diacritics. It is now a cost increase as well as a quality loss.

Budget context in documents, not tokens, when the language varies. Cap your retrieval at "twelve support notices", not "8,000 tokens", or the Vietnamese path silently gets less evidence than the English one.

Keep the structured parts of your prompt in English. Field names, enum values and instructions are not user-facing, and they tokenize best in English — one of the cheaper reasons that asking a model for JSON with an explicit schema pays for itself. Put Vietnamese where it belongs, in the content, and let the scaffolding be cheap.

What does not help is the storage-side lever. Embeddings for Vietnamese cost the same bytes as embeddings for English — a 384-dimension vector is 1,536 bytes whatever produced it — so quantising them to int8 saves exactly as much in either language and does nothing about the token count. The tokenizer tax lands on the API call, not on the disk.

Check it yourself

One dependency, no API key, no network after the install. Runs in about a second.

mkdir tokcost && cd tokcost
npm init -y >/dev/null && npm i gpt-tokenizer@4.0.0

Save as tokcost.mjs and run node tokcost.mjs. Replace the strings with your own text — the ratios are the point, not these particular sentences.

// tokcost.mjs -- what the same meaning costs in different languages.
// gpt-tokenizer 4.0.0. Offline: these are the real BPE tables, not an estimate.
import { encode as o200k } from 'gpt-tokenizer/encoding/o200k_base';
import { encode as cl100k } from 'gpt-tokenizer/encoding/cl100k_base';

const PRICE = 2.50;   // USD per 1M input tokens. EXAMPLE. Substitute yours.

const doc = {
  English: `The battery lasts about ten hours. Charge it fully before the first use. Do not leave the device in a hot car. If the screen does not turn on, hold the side button for ten seconds. Keep the receipt; the warranty lasts two years from the day of purchase.`,
  Vietnamese: `Pin dùng được khoảng mười giờ. Hãy sạc đầy trước lần sử dụng đầu tiên. Không để thiết bị trong ô tô nóng. Nếu màn hình không sáng, hãy giữ nút bên cạnh trong mười giây. Giữ lại hóa đơn; bảo hành kéo dài hai năm kể từ ngày mua.`,
  Chinese: `电池可以使用大约十个小时。首次使用前请充满电。不要把设备留在炎热的车内。如果屏幕不亮，请按住侧面按钮十秒钟。请保留收据；保修期为自购买之日起两年。`,
  Thai: `แบตเตอรี่ใช้งานได้ประมาณสิบชั่วโมง ชาร์จให้เต็มก่อนใช้งานครั้งแรก อย่าทิ้งอุปกรณ์ไว้ในรถที่ร้อน หากหน้าจอไม่ติด ให้กดปุ่มด้านข้างค้างไว้สิบวินาที เก็บใบเสร็จไว้ การรับประกันมีอายุสองปีนับจากวันที่ซื้อ`,
  French: `La batterie dure environ dix heures. Chargez-la complètement avant la première utilisation. Ne laissez pas l'appareil dans une voiture chaude. Si l'écran ne s'allume pas, maintenez le bouton latéral enfoncé pendant dix secondes. Conservez le reçu ; la garantie dure deux ans à compter du jour de l'achat.`,
};

const EN = o200k(doc.English).length;
console.log(`node ${process.versions.node}  o200k_base vs cl100k_base  $${PRICE}/1M tokens\n`);
console.log('lang         bytes   o200k  vs EN   cl100k  vs EN   $/100k docs   128k fits');
for (const [lang, text] of Object.entries(doc)) {
  const o = o200k(text).length, c = cl100k(text).length;
  console.log(
    lang.padEnd(12) +
    String(Buffer.byteLength(text, 'utf8')).padStart(6) +
    String(o).padStart(8) + (o / EN).toFixed(2).padStart(7) + 'x' +
    String(c).padStart(9) + (c / cl100k(doc.English).length).toFixed(2).padStart(7) + 'x' +
    ('$' + ((o * 100000 / 1e6) * PRICE).toFixed(2)).padStart(14) +
    String(Math.floor(128000 / o)).padStart(12));
}

// The accent test. Strip diacritics and re-count.
const strip = (s) => s.normalize('NFD').replace(/[̀-ͯ]/g, '')
  .replace(/đ/g, 'd').replace(/Đ/g, 'D').normalize('NFC');
const vi = doc.Vietnamese, bare = strip(vi);
console.log('\nAccent test (Vietnamese):');
console.log(`  o200k : ${o200k(vi).length} with accents -> ${o200k(bare).length} without` +
  `  (${o200k(bare).length > o200k(vi).length ? 'WORSE' : 'better'})`);
console.log(`  cl100k: ${cl100k(vi).length} with accents -> ${cl100k(bare).length} without` +
  `  (${cl100k(bare).length > cl100k(vi).length ? 'WORSE' : 'better'})`);
for (const w of ['người', 'được', 'những']) {
  console.log(`  ${w} = ${o200k(' ' + w).length} tok   ${strip(w)} = ${o200k(' ' + strip(w)).length} tok`);
}

On the M3 laptop above that prints:

node 23.5.0  o200k_base vs cl100k_base  $2.5/1M tokens

lang         bytes   o200k  vs EN   cl100k  vs EN   $/100k docs   128k fits
English        253      56   1.00x       56   1.00x        $14.00        2285
Vietnamese     303      64   1.14x      111   1.98x        $16.00        2000
Chinese        219      60   1.07x       82   1.46x        $15.00        2133
Thai           588      79   1.41x      182   3.25x        $19.75        1620
French         311      67   1.20x       87   1.55x        $16.75        1910

Accent test (Vietnamese):
  o200k : 64 with accents -> 72 without  (WORSE)
  cl100k: 111 with accents -> 80 without  (better)
  người = 1 tok   nguoi = 2 tok
  được = 1 tok   duoc = 2 tok
  những = 1 tok   nhung = 2 tok

The line to watch is the cl100k vs EN column against o200k vs EN. Thai drops from 3.25x to 1.41x and English does not move at all. Change one string to your own product copy and the ratio you get is the surcharge your non-English users are already paying.