Why is splitting text into sentences so hard?

On a hand-labelled test set of 187 sentences, text.split('.') produced 263 boundaries where 186 exist — 101 of them, 38% of everything it emitted, are wrong. The same document split that way and chunked for retrieval leaves 37.5% of chunks with a broken sentence at an edge. Intl.Segmenter, a standard-library built-in that needs no dependency, gets 0.939 F1 on the same set for one line of code — but it splits after "Dr." and there is no locale flag that stops it.

Everything below was run on Node v23.5.0 (ICU 76.1, V8 12.9) on an Apple M3, macOS 26.4.1, arm64. No NLP library, no model, no network.

The short answer

• text.split('.') scores precision 0.616, recall 0.871, F1 0.722. It emits 101 false boundaries on 186 real ones — inside $1.5M, v2.0.1, https://cstsolution.com, U.S.A. and e.g. — and misses every ? and !.
• A regex on [.!?]\s+ gets F1 0.869. Adding an abbreviation list and digit guards takes it to 0.933. A 60-line rules engine that also understands quotes, brackets, URLs, code fences and bullet lists reaches 0.978.
• Intl.Segmenter with granularity: 'sentence' scores 0.939 F1 in one line — better than any regex we would actually have written, and it has the highest recall of anything tested (0.995, one miss in 186).
• Its one weakness is abbreviations: ICU's default sentence rules split after Dr., and the ICU suppression option (en-u-ss-standard) is not honoured by Node — output is byte-identical with and without it. Filtering its output through a 15-line abbreviation guard lifts it to 0.963 and costs 30.5 ms per megabyte.
• On Chinese, every period-based method returns one segment for the whole document (F1 0.000); Intl.Segmenter scores 1.000. On a chat log with no terminal punctuation at all, every regex returns one segment; Intl.Segmenter recovers 9 of 9 boundaries from the line breaks.

How was this measured?

The test set is synthetic — we wrote every sentence in it, so the F1 numbers describe this corpus, not the internet. What transfers is which case classes break each method, and those classes are not synthetic: they are what appears in real invoices, discharge summaries, support threads and PDFs.

The corpus is 90 cases, 187 sentences, 6,449 characters, in 13 adversarial classes: plain prose, honorifics (Dr., Prof.), Latin abbreviations (e.g., etc., vs.), dotted acronyms (U.S.A., a.m., J. R. Pham), decimals and money and versions (3.14, $1.5M, v2.0.1), URLs and email addresses, ellipses, quotations with the stop inside and outside the quote, questions and exclamations, line breaks mid-sentence, bullet and numbered lists, and a fenced code block.

Labelling is mechanical: each case is an array of sentences, and the ground truth is the end of each string in that array, so joining them produces the document and the boundary offsets at once. A boundary is the offset immediately after a sentence's last non-whitespace character, and every method returns offsets in that same normalised form, so nothing is punished for keeping or dropping the trailing space. The document's end is excluded — every method gets it free. Scoring is exact-position precision, recall and F1 on the remaining 186.

What does split('.') actually get wrong?

Everything with a dot in it that is not the end of a sentence:

> "Dr. Smith earned $1.5M in 2024. See https://cstsolution.com/blog/ for the v2.0.1 notes.".split(".")

["Dr"," Smith earned $1","5M in 2024"," See https://cstsolution",
 "com/blog/ for the v2","0","1 notes",""]

Seven fragments where there are two sentences. $1.5M becomes $1 and 5M in 2024; the version number becomes three pieces; the URL is cut in half. Note also the empty string at the end — a bug that ships to production constantly, because "a.".split(".") is ["a", ""].

Here is the damage by case class. Each cell is boundaries found / boundaries that exist, plus the false boundaries invented inside that class.

Two rows matter most. Questions: split('.') finds 8 of 15, because a sentence ending in ? has no dot to split on — question and answer are welded into one segment. Lists: every period-based method scores 0 of 12. Bullet items are separate retrieval units that end in no punctuation, so nothing looking for a full stop will ever separate them.

So which method wins?

The hand-written rules engine wins English by 0.015 F1 over the hybrid, and by 0.039 over Intl.Segmenter alone. That is a real win, and it is also the most expensive result in the table: 60 lines that we now own, that encode an English-shaped view of punctuation, and that — as the next two sections show — fall apart the moment the text is not English or not punctuated.

Does Intl.Segmenter solve it?

It comes closer than we expected, and it fails in one specific, fixable place.

const seg = new Intl.Segmenter('en', { granularity: 'sentence' })
const sentences = [...seg.segment(text)].map((s) => s.segment)

That is the whole implementation. It finds 185 of 186 boundaries — recall 0.995, the best of anything measured — because it implements the Unicode UAX #29 sentence-break algorithm, which knows about closing quotes, brackets, ?, !, paragraph breaks and CJK punctuation without being told. Its one miss is the end of a fenced code block.

Its 23 false positives are almost all one thing:

> [...seg.segment("Dr. Smith earned $1.5M in 2024. See the v2.0.1 notes.")]
["Dr. ", "Smith earned $1.5M in 2024. ", "See the v2.0.1 notes."]

It handles $1.5M and v2.0.1 perfectly and then splits after Dr.. UAX #29 leaves abbreviations to a per-locale suppression list, and ICU ships one; CLDR exposes it as the ss extension, so new Intl.Segmenter('en-u-ss-standard') should fix exactly this. On Node 23.5.0 it does nothing:

en                   ["Dr. ","Smith earned $1.5M in 2024. ","Half of it went to hardware."]
en-u-ss-standard     ["Dr. ","Smith earned $1.5M in 2024. ","Half of it went to hardware."]
en-US-u-ss-standard  ["Dr. ","Smith earned $1.5M in 2024. ","Half of it went to hardware."]

Byte-identical. The data is in ICU; V8 does not wire the option through. This surprised us most: the documented fix for the segmenter's one real weakness is inert.

The repair is fifteen lines — run the segmenter, then drop any boundary whose preceding token is a known abbreviation or a single capital letter. That is the Intl.Segmenter + abbrev filter row: F1 0.963, and unlike the rules engine it keeps everything UAX #29 already does for other scripts.

What does a bad split cost downstream?

A sentence splitter is rarely the product. It is the stage before chunking, and a chunk that starts mid-sentence retrieves worse — the embedding is computed over a fragment whose subject is in the previous chunk, and the snippet a user reads begins with sk whether a high number is.

So: split the same document each way, pack the resulting segments into chunks of at most 400 characters, and count how many chunks have an edge that is not a real sentence boundary.

The chunk count is identical in all six cases — the size budget decides that. What changes is where the cuts land. Six of sixteen chunks damaged is not a rounding error at corpus scale; it is thousands of retrieval units that begin in the middle of a thought, and the retriever has no way to know. Same failure mode as a search that ranks the right document below the wrong one: the ranking is fine, the unit being ranked is broken.

Note that the abbreviation-guarded regex reaches 0% broken edges while scoring lower F1 than Intl.Segmenter. Missing a boundary costs you a long chunk; inventing one costs you a broken chunk. For chunking, precision is the metric that matters.

What about Vietnamese, Chinese and Thai?

Our readers are mostly Vietnamese, so this is not an appendix.

Vietnamese uses the full stop as English does, so it inherits every English failure and adds its own: always-abbreviated honorifics (TS., BS., ThS.), place prefixes (TP., Q.), v.v. for "etc.", and — worst — a dot as the thousands separator.

> "Hợp đồng trị giá 1.500.000 đồng. TS. Nguyễn đã ký.".split(".")
["Hợp đồng trị giá 1","500","000 đồng"," TS"," Nguyễn đã ký",""]

One contract value becomes three sentences. Across three writing systems:

Chinese ends sentences with 。 and puts no space after it, so [.!?]\s+ matches nothing and the document comes back as one segment — not a degraded result, zero boundaries. Intl.Segmenter gets 16 of 16, because UAX #29 lists 。, ！ and ？ as terminators. The rules engine's 0.609 is not comprehension either: it found only the blank lines between test cases.

Thai is where everything loses. Thai ends sentences with a space and no punctuation, and telling that space apart from an intra-sentence space needs a dictionary. Intl.Segmenter returns the whole Thai paragraph as one segment — its 0.625 comes entirely from blank lines — and so does everything else. If you index Thai, segmentation is a dictionary problem and none of these six methods is the answer, which is worth knowing before you promise it.

What happens with no terminal punctuation at all?

Chat logs, meeting transcripts and support tickets often have none. We built a ten-line chat log with zero periods, question marks or exclamation marks:

son: the index rebuild finished
lan: how long did it take
son: about nine minutes

Four methods return the entire transcript as one segment. A chunker fed that has nothing to work with and falls back to cutting on character count, which is where we came in. Intl.Segmenter treats the line breaks as sentence boundaries — UAX #29 rule SB4 — and recovers all nine. The failure here is total, not gradual, and it happens on the most common unstructured input in a support product.

How fast is each one?

A 1,290,198-character document (200 copies of the corpus, 37,400 sentences), median of five runs:

We expected Intl.Segmenter to be the slow option — it crosses into ICU and allocates an object per segment. It is instead 4.5× faster than our own rules engine, at 13.4 ms per megabyte. Across a 100,000-document corpus averaging 20 KB, fastest and slowest here differ by roughly 7 seconds and 93 seconds in total. Speed is not the axis on which to choose, which is easier to see beside what the tokens for those documents cost: segmentation is free, and the thing it feeds is not.

What we use

Intl.Segmenter with the correct locale, plus a short abbreviation filter over its output. It is the only configuration measured that is above 0.96 on English, above 0.98 on Vietnamese, exactly 1.000 on Chinese and non-zero on a chat log. The rules engine beats it on English by 0.015 F1 and loses everywhere else, which is the trade: sixty lines of English punctuation folklore, or one line of Unicode and fifteen lines of abbreviations.

And if you take one thing: stop writing .split('.'). It has been wrong in your pipeline the whole time, quietly, one chunk in three.

Check it yourself

No dependencies, Node 18+. Save the first block as sbd.mjs, append the second block to the same file, and run it. This corpus is a 14-case excerpt of the 90-case set — enough to exercise every class and every method, small enough to read — so its F1 values are lower than the tables above and the shape of the ranking is the same. The exact output on the machine described at the top is printed below the script; if yours differs, the difference is your Node's ICU version.

// ---- the labelled corpus. Ground truth = the end of each string in `sents`.
const CASES = [
  { cls: 'plain', sents: ['The server accepted the upload.', 'The parser ran for eleven seconds.', 'Nothing else happened that night.'] },
  { cls: 'plain', sents: ['We keep the index on disk.', 'It is rebuilt once a week.'] },
  { cls: 'abbrev_title', sents: ['Dr. Nguyen reviewed the dosing table.', 'She found one error.'] },
  { cls: 'abbrev_latin', sents: ['Some fields are optional, e.g. the middle name.', 'Others are not.'] },
  { cls: 'acronym', sents: ['The backup starts at 2 a.m. and finishes before staff arrive.'] },
  { cls: 'number', sents: ['The round closed at $1.5M.', 'Half of it went to hardware.'] },
  { cls: 'url_email', sents: ['Send the export to data.team@cstsolution.com before Friday.', 'Nothing arrives after that.'] },
  { cls: 'ellipsis', sents: ['The log just stopped... Nobody could say why.'] },
  { cls: 'quote', sents: ['He said, "the index is stale."', 'He was right.'] },
  { cls: 'question', sents: ['How large should a chunk be?', 'It depends on the document.'] },
  { cls: 'linebreak', sents: ['The parser reads the file in one pass\nand keeps the offsets in memory.', 'Memory is the limit.'] },
  { cls: 'list', sents: ['The pipeline has three stages:', '- extract the text', '- split it into sentences'], join: '\n' },
  { cls: 'code', sents: ['The call is one line:\n\n```js\nconst parts = text.split(".")\n```', 'That line is the bug.'] },
  { cls: 'mixed', sents: ['The U.S.A. office opens at 8 a.m. and closes at 4 p.m.', 'Vietnam is twelve hours ahead.'] },
]

const VI_CASES = [
  { cls: 'vi_number', sents: ['Hợp đồng trị giá 1.500.000 đồng.', 'Một nửa dành cho phần cứng.'] },
  { cls: 'vi_title', sents: ['TS. Nguyễn đã xem lại bảng liều.', 'Ông tìm thấy một lỗi.'] },
]
const ZH_CASES = [
  { cls: 'zh', sents: ['服务器已经接收了上传的文件。', '解析器运行了十一秒。'], join: '' },
  { cls: 'zh', sents: ['一个片段应该多长？', '这取决于文档本身。'], join: '' },
]
const TH_CASES = [
  { cls: 'th', sents: ['เซิร์ฟเวอร์ได้รับไฟล์แล้ว', 'ตัวแยกวิเคราะห์ทำงานสิบเอ็ดวินาที'] },
]
const CHAT_CASES = [
  { cls: 'chat', sents: ['son: the index rebuild finished', 'lan: how long did it take', 'son: about nine minutes', 'lan: thats slower than yesterday'], join: '\n' },
]

function build(cases) {
  let text = ''
  const truth = []
  cases.forEach((c, ci) => {
    const join = c.join === undefined ? ' ' : c.join
    c.sents.forEach((s, si) => {
      text += s
      truth.push({ pos: text.length, cls: c.cls })
      if (si < c.sents.length - 1) text += join
    })
    if (ci < cases.length - 1) text += '\n\n'
  })
  truth.pop()          // the end of the document is free for every method
  return { text, truth }
}

// ---- the six splitters. Each returns boundary offsets, normalised.
const norm = (text, arr) => {
  const end = text.trimEnd().length
  const out = new Set()
  for (let p of arr) {
    while (p > 0 && /\s/.test(text[p - 1])) p--
    if (p > 0 && p < end) out.add(p)
  }
  return [...out].sort((a, b) => a - b)
}

function m1_splitDot(text) {
  const b = []
  for (let i = text.indexOf('.'); i !== -1; i = text.indexOf('.', i + 1)) b.push(i + 1)
  return norm(text, b)
}

function m2_regex(text) {
  const b = []
  const re = /[.!?]\s+/g
  let m
  while ((m = re.exec(text))) b.push(m.index + 1)
  return norm(text, b)
}

const ABBREV = new Set([
  'dr', 'mr', 'mrs', 'ms', 'prof', 'sr', 'jr', 'st', 'mt', 'no', 'fig', 'eq',
  'vol', 'ch', 'sec', 'dept', 'univ', 'inc', 'ltd', 'co', 'corp', 'est',
  'approx', 'min', 'max', 'e.g', 'i.e', 'etc', 'vs', 'cf', 'al', 'ca', 'ph.d',
  'a.m', 'p.m', 'u.s', 'u.s.a', 'u.k', 'n.h.s', 'jan', 'feb', 'mar', 'apr',
  'jun', 'jul', 'aug', 'sep', 'sept', 'oct', 'nov', 'dec',
  'ts', 'bs', 'ths', 'gs', 'pgs', 'tp', 'q', 'p', 'đ', 'hình', 'v.v', 'tr',
])
const TOKCH = /[\p{L}.]/u
const prevToken = (text, i) => {
  let a = i
  while (a > 0 && TOKCH.test(text[a - 1])) a--
  const t = text.slice(a, i)
  return /^\p{L}/u.test(t) ? t.toLowerCase() : ''
}
const nextNonSpace = (text, i) => {
  while (i < text.length && /\s/.test(text[i])) i++
  return text[i] ?? ''
}

function guarded(text, index, punct) {
  if (punct !== '.') return true
  const tok = prevToken(text, index)
  if (tok && (ABBREV.has(tok) || /^\p{Lu}$/u.test(tok))) return false     // Dr. / J.
  if (/\d/.test(text[index - 1] ?? '') && /\d/.test(nextNonSpace(text, index + 1))) return false
  const lineStart = text.lastIndexOf('\n', index - 1) + 1
  if (/^\s*\d+$/.test(text.slice(lineStart, index))) return false          // "1. item"
  return true
}

function m3_abbrev(text) {
  const b = []
  const re = /[.!?]\s+/g
  let m
  while ((m = re.exec(text))) if (guarded(text, m.index, text[m.index])) b.push(m.index + 1)
  return norm(text, b)
}

function fenceRanges(text) {
  const r = []
  const re = /```[\s\S]*?```/g
  let m
  while ((m = re.exec(text))) r.push([m.index, m.index + m[0].length])
  return r
}
const inRanges = (ranges, i) => ranges.some(([a, z]) => i >= a && i < z)
function tokenAround(text, i) {
  let a = i, z = i
  while (a > 0 && !/\s/.test(text[a - 1])) a--
  while (z < text.length && !/\s/.test(text[z])) z++
  return { tok: text.slice(a, z), a, z }
}

function m4_rules(text) {
  const b = []
  const fences = fenceRanges(text)
  const qpar = new Uint8Array(text.length + 1)
  const depth = new Int32Array(text.length + 1)
  for (let i = 0; i < text.length; i++) {
    const c = text[i]
    qpar[i + 1] = qpar[i] ^ (c === '"' ? 1 : 0)
    depth[i + 1] = depth[i] + (c === '(' || c === '[' || c === '{' ? 1 : 0) -
      (c === ')' || c === ']' || c === '}' ? 1 : 0)
  }
  const para = /\n[ \t]*\n/g
  let m
  while ((m = para.exec(text))) if (!inRanges(fences, m.index)) b.push(m.index)
  const bullet = /\n[ \t]*(?:[-*•]\s|\d+[.)]\s)/g
  while ((m = bullet.exec(text))) if (!inRanges(fences, m.index)) b.push(m.index)

  const re = /([.!?]+)(["')\]}”’]*)(\s+|$)/g
  while ((m = re.exec(text))) {
    const punct = m[1]
    const i = m.index
    const end = m.index + punct.length + m[2].length
    if (inRanges(fences, i)) continue
    if (!guarded(text, i, punct.length === 1 ? punct : punct[0])) continue
    const { tok, z } = tokenAround(text, i)
    const isUrlish = /^(https?:\/\/|www\.)/i.test(tok) || /@/.test(tok) ||
      /^[\w.-]+\.(com|org|net|io|dev|vn|co|gov|edu)\b/i.test(tok)
    if (isUrlish && z > i + punct.length) continue
    const nxt = nextNonSpace(text, end)
    if (punct.length > 1 && punct[0] === '.' && !/[\p{Lu}"'“]/u.test(nxt)) continue
    if (qpar[i] === 1 && !m[2].includes('"')) continue
    if (depth[i] > 0 && !/[)\]}]/.test(m[2])) continue
    if (nxt && /\p{Ll}/u.test(nxt) && !/\n[ \t]*\n/.test(m[3])) continue
    b.push(end)
  }
  return norm(text, b)
}

function m5_segmenter(text, locale = 'en') {
  const seg = new Intl.Segmenter(locale, { granularity: 'sentence' })
  const b = []
  for (const s of seg.segment(text)) b.push(s.index + s.segment.length)
  b.pop()
  return norm(text, b)
}

function m6_hybrid(text, locale = 'en') {
  const fences = fenceRanges(text)
  return m5_segmenter(text, locale).filter((p) => {
    if (inRanges(fences, p - 1)) return false
    if (text[p - 1] !== '.') return true
    return guarded(text, p - 1, '.')
  })
}

const METHODS = [
  ["split('.')", m1_splitDot],
  ['regex [.!?]\\s+', m2_regex],
  ['+ abbrev/digits', m3_abbrev],
  ['rules engine', m4_rules],
  ['Intl.Segmenter', m5_segmenter],
  ['Segmenter+abbrev', m6_hybrid],
]

// ---- scoring
const f3 = (x) => (Number.isFinite(x) ? x.toFixed(3) : '0.000')
const pad = (s, n) => String(s).padEnd(n)
const lpad = (s, n) => String(s).padStart(n)

function score(truth, pred) {
  const T = new Set(truth.map((t) => t.pos))
  const P = new Set(pred)
  let tp = 0
  for (const p of P) if (T.has(p)) tp++
  const precision = P.size ? tp / P.size : 0
  const recall = T.size ? tp / T.size : 0
  const f1 = precision + recall ? (2 * precision * recall) / (precision + recall) : 0
  return { tp, fp: P.size - tp, fn: T.size - tp, precision, recall, f1 }
}

const { text: DOC, truth: TRUTH } = build(CASES)
console.log(`corpus ${DOC.length} chars, ${TRUTH.length} labelled boundaries\n`)
console.log(pad('method', 18), lpad('pred', 5), lpad('TP', 5), lpad('FP', 5), lpad('FN', 5), lpad('prec', 7), lpad('rec', 7), lpad('F1', 7))
for (const [name, fn] of METHODS) {
  const s = score(TRUTH, fn(DOC))
  console.log(pad(name, 18), lpad(fn(DOC).length, 5), lpad(s.tp, 5), lpad(s.fp, 5), lpad(s.fn, 5),
    lpad(f3(s.precision), 7), lpad(f3(s.recall), 7), lpad(f3(s.f1), 7))
}

// ---- downstream: chunks with a broken edge
function chunkFromBoundaries(text, bounds, max = 400) {
  const edges = [0, ...bounds, text.trimEnd().length]
  const chunks = []
  let start = 0
  for (let i = 1; i < edges.length; i++) {
    if (edges[i] - start >= max || i === edges.length - 1) {
      chunks.push({ start, end: edges[i] })
      start = edges[i]
    }
  }
  return chunks
}
const TSET = new Set([0, DOC.trimEnd().length, ...TRUTH.map((t) => t.pos)])
console.log('\nchunks with a broken edge (max 400 chars)')
for (const [name, fn] of METHODS) {
  const cs = chunkFromBoundaries(DOC, fn(DOC))
  const broken = cs.filter((c) => !TSET.has(c.start) || !TSET.has(c.end)).length
  console.log(pad(name, 18), lpad(cs.length, 4), 'chunks', lpad(broken, 4), 'broken',
    lpad(((broken / cs.length) * 100).toFixed(1) + '%', 8))
}

// ---- other languages, and the chat log
for (const [label, cases, locale] of [['VI', VI_CASES, 'vi'], ['ZH', ZH_CASES, 'zh'],
  ['TH', TH_CASES, 'th'], ['CHAT', CHAT_CASES, 'en']]) {
  const { text, truth } = build(cases)
  console.log(`\n${label} (${truth.length} boundaries)`)
  for (const [name, fn] of METHODS) {
    const pred = name.startsWith('Intl') ? m5_segmenter(text, locale)
      : name.startsWith('Segmenter') ? m6_hybrid(text, locale) : fn(text)
    const s = score(truth, pred)
    console.log(pad(name, 18), 'TP', lpad(s.tp, 3), ' F1', f3(s.f1))
  }
}

// ---- the ICU suppression option that does nothing
console.log('\nICU sentence-break suppressions:')
for (const loc of ['en', 'en-u-ss-standard']) {
  const s = new Intl.Segmenter(loc, { granularity: 'sentence' })
  console.log(pad(loc, 20), JSON.stringify([...s.segment('Dr. Smith earned $1.5M in 2024. Half went to hardware.')].map((x) => x.segment)))
}

// ---- speed, median of five
const BIG = Array(200).fill(DOC).join('\n\n')
console.log(`\nspeed on ${BIG.length} chars, median of 5`)
for (const [name, fn] of METHODS) {
  const ts = []
  for (let k = 0; k < 5; k++) {
    const t0 = process.hrtime.bigint()
    fn(BIG)
    ts.push(Number(process.hrtime.bigint() - t0) / 1e6)
  }
  ts.sort((a, b) => a - b)
  console.log(pad(name, 18), lpad(ts[2].toFixed(1), 7), 'ms',
    lpad((BIG.length / 1e6 / (ts[2] / 1000)).toFixed(1), 7), 'MB/s')
}
console.log('\nnode', process.version, '| ICU', process.versions.icu)

node sbd.mjs

corpus 983 chars, 27 labelled boundaries

method              pred    TP    FP    FN    prec     rec      F1
split('.')            41    21    20     6   0.512   0.778   0.618
regex [.!?]\s+        28    22     6     5   0.786   0.815   0.800
+ abbrev/digits       22    21     1     6   0.955   0.778   0.857
rules engine          27    25     2     2   0.926   0.926   0.926
Intl.Segmenter        32    26     6     1   0.813   0.963   0.881
Segmenter+abbrev      28    25     3     2   0.893   0.926   0.909

chunks with a broken edge (max 400 chars)
split('.')            3 chunks    2 broken    66.7%
regex [.!?]\s+        3 chunks    0 broken     0.0%
...
ZH (3 boundaries)
split('.')         TP   0  F1 0.000
regex [.!?]\s+     TP   0  F1 0.000
+ abbrev/digits    TP   0  F1 0.000
rules engine       TP   1  F1 0.500
Intl.Segmenter     TP   3  F1 1.000
Segmenter+abbrev   TP   3  F1 1.000

CHAT (3 boundaries)
split('.')         TP   0  F1 0.000
...
Intl.Segmenter     TP   3  F1 1.000

ICU sentence-break suppressions:
en                   ["Dr. ","Smith earned $1.5M in 2024. ","Half went to hardware."]
en-u-ss-standard     ["Dr. ","Smith earned $1.5M in 2024. ","Half went to hardware."]

node v23.5.0 | ICU 76.1

Then paste your own documents in place of CASES — labelling is just writing each sentence as its own string — and run it against the text you actually index. If your corpus is invoices, the decimal row is your whole problem; if it is chat, only the segmenter rows are real options.

Where this goes next

Segments feed the chunker, and the chunker feeds an index that must work with no network. In MedSearch that chain runs entirely on the device, which is why every stage of it has to be measurable offline — including this one.