How do you tell what language a message is in?

A character n-gram profile built from roughly 700 characters per language identified 100% of our 72 test tickets at 100 characters of input, 95.8% at 20 characters and 86.1% at 10 — at 266 microseconds per detection, with no library, no network and no model. Unicode script detection, the thing most people reach for first, resolved only 33.3% of the same messages on its own.

Routing a ticket, choosing a prompt template, picking a tokenizer, deciding whether a cheap model will do — all of it comes after the language question. The usual answers are "call a library" or "ask an LLM". Both work. Neither tells you where the answer stops being trustworthy, which is what matters when your inbox is full of twenty-character messages.

Hardware: Apple M3, 16 GB, macOS 26.4.1 (build 25E253). Python 3.14.6 arm64, standard library only, fully offline.

On the corpus. I wrote it by hand: per language, one profile text of roughly 700 characters and eight held-out ticket bodies of about 200 characters — 72 test documents, all of them things a support inbox actually receives. I am a native speaker of two of these nine, so the Thai, Japanese and Indonesian are plain rather than idiomatic and the absolute accuracy figures are optimistic. The transferable finding is the relative pattern — how accuracy moves with input length and script family — not the absolute percentages.

The short answer

• A 300-entry character n-gram profile with Cavnar-Trenkle rank distance got 100% at 100 characters, 95.8% at 20 and 86.1% at 10 across nine languages, in 266 microseconds per 100-character message on one core.
• Unicode script detection resolved 33.3% of messages to a single language, leaving a mean of 3.89 candidates. Perfect on Thai, Chinese and Japanese; worth nothing on the five languages that share the Latin script.
• Vietnamese is the Latin-script exception: its diacritics identify it from script alone, 100% of the time from 20 characters up. The first Vietnamese-only letter appears after a median of 7 characters.
• Strip those diacritics and it collapses. Script detection goes 100% to 0% — every unaccented Vietnamese ticket is called English — and the n-gram method 100% to 87.5% at full length, 37.5% at 20 characters.
• On mixed English/Vietnamese tickets the n-gram method picked English on 4 of 6 messages that were roughly half Vietnamese. A margin threshold of 0.02 kept 96.5% of answers and made the kept ones 100% correct.

What are the four methods?

Unicode script detection checks which code blocks the characters fall in — Thai in U+0E00–U+0E7F, kana in U+3040–U+30FF, Han in U+4E00–U+9FFF. A lookup, not a classifier.

Character n-gram profiles, the Cavnar-Trenkle method from 1994, do the real work. Split the text into whitespace tokens, pad each with underscores, count every substring of length 1 to 5, sort by frequency and keep the top 300: that ranked list is the profile. To classify, build the same profile for the document and sum, over each of its n-grams, the gap between its rank there and its rank in the language, with a fixed penalty of 300 for anything absent. Lowest total wins. An English profile starts _ e a t r e_ i n o s h l; Vietnamese starts _ n t h i _t n_ c g ng g_ ng_. That is the entire signal.

Stop-word matching counts function words — the and to of, và của là không, yang dan di ke — matching particles as substrings for the three languages without spaces. The combination narrows by script, ranks by n-gram distance inside the candidate set, and falls back to the stop-word score when the top two are within 5% of each other.

How short can the text be?

Everyone tests language detection on paragraphs and ships it on chat messages, subject lines and search queries. Accuracy over all nine languages, 72 documents at each length:

The n-gram method is essentially finished by 100 characters — one sentence — and already at 95.8% by 20, which is a subject line. The stop-word method is the one that falls apart: 52.8% at 10 characters, because a ten-character fragment often contains no function word at all, so every language scores zero and the answer is a coin flip. Function words are a property of sentences; n-grams are a property of letters, and letters are all a short string has.

One caveat: our Chinese documents are 63–74 characters and the Japanese ones 86–100, so for those two the 100, 200 and full columns are the same text — the same density effect that makes Chinese cheaper than French to tokenise.

Where is script detection enough, and where is it useless?

Both, sharply, in the same corpus. Split the runs by script family:

For Thai, Chinese and Japanese, script detection is not merely adequate, it is the correct answer: 100% at ten characters, 4.8 microseconds per message, and it cannot be wrong the way a statistical model can. A range check is enough.

For the Latin six it resolves nothing. Asked to name one language it was right 33.3% of the time, and even that is inflated: it comes from Vietnamese (caught by diacritics) plus English, which only "wins" because English is the tie-break default. French, German, Spanish and Indonesian scored 0.0% at every length. What it does deliver is a candidate set containing the true language 100% of the time, averaging 3.89 candidates. Its real job is not answering, but narrowing.

Which letters are actually Vietnamese-only?

This one contradicted my expectation and cost a re-run, so it goes in. My first rule was ă â đ ê ô ơ ư — the vowels you learn as distinctively Vietnamese. It caught 100% of Vietnamese and also flagged 4 of 8 French tickets as Vietnamese, because French uses ê, ô and â too.

The correct set is ă đ ơ ư plus Latin Extended Additional (U+1E00–U+1EFF), where the precomposed ế ạ ộ ầ live and where no other language here goes. A "distinctive letter" heuristic is only as good as the list of languages you checked it against: add Portuguese and ã stops being a Vietnamese signal too.

What about unaccented Vietnamese?

It collapses, and this is the most useful number here for a Vietnamese inbox. People type khong co for không có constantly — on phones without a Vietnamese keyboard, in search boxes, and in any pipeline that has already mangled the encoding.

Script detection does not degrade — it inverts. All eight unaccented tickets were classified as English, confidently, because with the diacritics gone the text is plain ASCII and there is nothing left to key on. The n-gram method survives at full length (87.5%, one ticket lost to Indonesian) and is nearly useless at 20 characters.

So never make the script signal your only detector: the moment a user types without accents your router quietly starts sending them to the English queue. Note too that the combination scores worse than plain n-grams on unaccented text at full length, 75% against 87.5%. The script stage is confidently wrong, so narrowing first amplifies its error. A stage that is perfect on clean input can be a liability on dirty input.

What happens with a mixed English/Vietnamese message?

Code-switching is the normal register in a Vietnamese support inbox: "Chào shop, mình đặt hàng tuần trước nhưng tracking vẫn báo pending. Could you please check the status?" I wrote six such messages, 139–163 characters, with 7.2%–11.6% of their characters carrying Vietnamese diacritics. The n-gram method picked English on 4 of 6 and Vietnamese on 2; stop-word matching picked English on all 6; script detection and the combination picked Vietnamese on all 6, because a single ầ short-circuits the rule. Distances on the first message:

English wins by 13% over a message a human would call bilingual. The methods are not malfunctioning; the question has no single right answer. The correct output for a mixed message is two labels or none, and none of these methods can express that. The useful signal: margins on mixed text ran 0.013 to 0.132, against a median of 0.171 for correct single-language answers at full length. A low margin is your "this might be code-switched" flag — a reason to route to a bilingual agent, not a failure.

Can it say "I do not know"?

Yes, and cheaply. Use the relative gap between the best and second-best rank distance, (d2 - d1) / d1. At 20 characters the 69 correct answers had a median margin of 0.131 and the 3 wrong ones a median of 0.009 — barely overlapping distributions, which is what makes a threshold work. Pooled over 288 classifications at 20, 50, 100 and full length:

A threshold of 0.02 abstains on 3.5% of messages and removes every error here. On a corpus this small "100%" means "no counterexamples yet", but the shape is what to copy: abstaining on a few percent of inputs is far cheaper than misrouting them. The errors it catches, all at 20 characters, are instructive — Ma facture du mois d read as English, Le fichier d'export as German, Aplikasi di ponsel s as Spanish. Short Latin fragments with no distinctive letters are genuinely ambiguous, and saying so beats guessing.

How does this compare with an LLM call?

Not on accuracy — on everything else.

Best of 15 runs, single core, pure Python. Building all nine profiles from 5,389 characters of training text takes 6.7 ms, once at process start, with no model file to ship.

One example rate for scale: $1.00 per million input tokens, the published standard input price for Claude Haiku 4.5, checked on 1 September 2026. A 200-character ticket plus an instruction is comfortably over 50 tokens, so a million classifications is tens of dollars of input alone. The n-gram detector costs nothing per call and adds 266 microseconds instead of a round trip — if that round trip were 30 ms, the local detector runs about 110 times inside it. Classifying the language is not work that needs a model, the same way lexical search beats an embedding call when the query is a product code.

What should you actually do?

Check the script first and let it answer when it can — Thai, Chinese, Japanese, Korean, Cyrillic and Arabic are settled by a range check at 4.8 microseconds. For Latin text run the n-gram profile and keep the margin; abstain below 0.02 and send abstentions to a default language or a human. Do not make Vietnamese diacritics your only Vietnamese signal, because your users will remove them. And be honest about mixed messages: replying in the language the customer opened with is a business rule, not a classifier output.

Check it yourself

Save as langdetect.py and run python3 langdetect.py. No dependencies, no network. Two languages, so the mechanism is visible; add more via TRAIN.

# langdetect.py - Cavnar-Trenkle language ID, standard library only.
import re, unicodedata
from collections import Counter

SIZE, MAXN = 300, 5
WORD = re.compile(r"[^\w']+", re.UNICODE)

def ngrams(text):
    c = Counter()
    for tok in filter(None, WORD.split(text.lower())):
        s = "_" + tok + "_"
        for n in range(1, MAXN + 1):
            for i in range(len(s) - n + 1):
                c[s[i:i + n]] += 1
    return c

def profile(text):
    top = sorted(ngrams(text).items(), key=lambda kv: (-kv[1], kv[0]))[:SIZE]
    return {g: i for i, (g, _) in enumerate(top)}

def distance(doc, lang):
    return sum(SIZE if lang.get(g) is None else abs(r - lang[g])
               for g, r in doc.items()) / max(1, len(doc))

# script rules: these letters appear in no other language in this set
VI_ONLY = set("ăđơưĂĐƠƯ")
def script(text):
    t = unicodedata.normalize("NFC", text)
    if any("฀" <= c <= "๿" for c in t):  return ["th"]
    if any("぀" <= c <= "ヿ" for c in t):  return ["ja"]
    if any("一" <= c <= "鿿" for c in t):  return ["zh", "ja"]
    if any(c in VI_ONLY or "Ḁ" <= c <= "ỿ" for c in t): return ["vi"]
    return ["en", "fr", "de", "es", "id", "vi"]

TRAIN = {
 "en": "Thank you for contacting our support team. We have received your "
       "request and a member of staff will reply within one business day. "
       "Please keep this ticket number for reference. The payment has been "
       "refunded to the original card and should appear within five days.",
 "vi": "Cảm ơn bạn đã liên hệ với bộ phận hỗ trợ của chúng tôi. Chúng tôi đã "
       "nhận được yêu cầu của bạn và một nhân viên sẽ trả lời trong vòng một "
       "ngày làm việc. Vui lòng giữ lại mã số này để tiện tra cứu. Khoản "
       "thanh toán đã được hoàn về thẻ ban đầu và sẽ xuất hiện sau năm ngày.",
}
P = {l: profile(t) for l, t in TRAIN.items()}

def detect(text, threshold=0.02):
    cand = [l for l in script(text) if l in P] or list(P)
    if len(cand) == 1:
        return cand[0], 999.0
    d = profile(text)
    sc = sorted(((distance(d, P[l]), l) for l in cand))
    margin = (sc[1][0] - sc[0][0]) / sc[0][0]
    return (sc[0][1] if margin >= threshold else "unknown"), margin

for msg in [
    "The printer stopped responding after the last firmware update.",
    "Máy in ngừng phản hồi sau lần cập nhật phần mềm gần nhất.",
    "May in ngung phan hoi sau lan cap nhat phan mem gan nhat.",   # no accents
    "Chào bạn, could you check the order status? Cảm ơn nhiều.",   # mixed
    "hello",
]:
    lang, m = detect(msg)
    print(f"{lang:>7}  margin {m:6.3f}  {msg[:44]}")

Expected output on Python 3.14.6:

     en  margin  0.205  The printer stopped responding after the las
     vi  margin 999.000  Máy in ngừng phản hồi sau lần cập nhật phần
     vi  margin  0.042  May in ngung phan hoi sau lan cap nhat phan
     vi  margin 999.000  Chào bạn, could you check the order status?
     en  margin  0.096  hello

A margin of 999 means the script rule decided outright and the n-gram stage never ran — which is why line four says Vietnamese for a half-English message. Add the other seven languages and watch line three stop being easy.