Chunking a document without destroying its meaning
Split on structure first — headings, paragraphs, tables — and fall back to character count only when a single block is genuinely too big. Fixed-size chunking cuts tables in half and hands your retriever a fragment that no embedding model can repair.
Split on structure — headings, paragraphs, tables, code blocks — and only fall back to character count when a single block is genuinely too large. Then attach the heading path to every chunk. A chunk that starts mid-sentence is a chunk your retriever can find and your reader cannot use, and no embedding model recovers a table header that was cut off two hundred characters earlier.
Everything below was run on Node 23.5.0. The document under test is 1,549 characters of a paediatric fever reference: prose, a dosing table, and a SQL snippet.
The problem
The first chunker anyone writes is four lines:
export function fixedChunks(text, size = 400) {
const out = []
for (let i = 0; i < text.length; i += size) out.push(text.slice(i, i + size))
return out
}
It is fast, it is obviously correct as code, and it produces this:
[0] "# Paediatric fever\n\n## When to seek help the same day\n\nA temperature at or abo…"
[1] "sk whether a high number is dangerous in itself. It is not.\nFever is a respons…"
[2] "| 15 mg/kg | 4-6 h | 60 mg/kg |\n| Ibuprofen | 10 mg/kg | 6-8 h | 30 mg/kg |\n\nD…"
[3] "ded in significant liver disease.\n\n## Recording a reading\n\n```sql\nINSERT INTO …"
sizes {"n":4,"min":349,"p50":400,"p90":400,"max":400}
defects {"chunks":4,"endMidSentence":3,"tableRowsOrphaned":1}
Three of the four chunks end in the middle of a sentence. One begins with the word sk. And here is the boundary that should end the argument:
..."r 24h |\n|---|---|---|---|\n| Paracetamol " | "| 15 mg/kg | 4-6 h | 60 mg/kg |\n| Ibupro"...
The cut lands inside a table row. Chunk 1 ends with the drug name and no dose. Chunk 2 begins with | 15 mg/kg | 4-6 h | 60 mg/kg | — a complete row with no drug attached to it — and contains the ibuprofen row with no header, so nothing in that chunk says which of 10 mg/kg and 30 mg/kg is the single dose and which is the daily maximum.
That is not a ranking problem. Retrieve that chunk with a perfect retriever and you have still retrieved something that cannot be answered from.
Why the obvious approach fails
The usual response is to fix the boundary rather than the method: split on sentence ends, or on double newlines, or add overlap so that whatever got cut appears again in the next chunk.
Each of those helps, and none of them addresses the actual failure, which is that character count is not a unit of meaning and a document already tells you what its units are. A markdown document has headings, paragraphs, list items, tables and fenced code. Those boundaries were chosen by a human being who was deciding what belonged with what. Ignoring them in favour of a number you picked is throwing away the only structural signal in the input.
The second reason is subtler. A fixed-size splitter has no concept of a block it must not break, so it will break the ones where breaking is most expensive. Tables and code are the highest-information regions of most technical documents and the ones that degrade most sharply when cut — half a table is not half as useful as a whole table, it is useless. Prose degrades gracefully. Tables do not.
The fix: structure first, size second
Parse to blocks, then pack blocks up to a size budget, and never pack across a heading:
const HEADING = /^(#{1,6})\s+(.+)$/
const TABLE_ROW = /^\s*\|/
const FENCE = /^\s*```/
export function parseBlocks(md) {
const blocks = []
const path = []
let buf = [], kind = 'prose', fenced = false
const flush = () => {
const text = buf.join('\n').trim()
if (text) blocks.push({ kind, path: [...path], text })
buf = []; kind = 'prose'
}
for (const line of md.split('\n')) {
if (FENCE.test(line)) {
if (fenced) { buf.push(line); fenced = false; flush() }
else { flush(); fenced = true; kind = 'code'; buf.push(line) }
continue
}
if (fenced) { buf.push(line); continue }
const h = line.match(HEADING)
if (h) {
flush()
const depth = h[1].length
path.length = Math.min(path.length, depth - 1)
while (path.length < depth - 1) path.push('')
path[depth - 1] = h[2].trim()
continue
}
if (line.trim() === '') { flush(); continue }
const isRow = TABLE_ROW.test(line)
if (isRow && kind !== 'table') { flush(); kind = 'table' }
if (!isRow && kind === 'table') { flush() }
buf.push(line)
}
flush()
return blocks
}
Then the packer. Two rules do most of the work: a chunk never spans two heading paths, and tables and code are atomic.
const SENTENCE = /(?<=[.!?])\s+(?=[A-Z(])/
export function splitProse(text, max) {
const out = []
let cur = ''
for (const s of text.split(SENTENCE)) {
if (cur && (cur + ' ' + s).length > max) { out.push(cur); cur = s }
else cur = cur ? cur + ' ' + s : s
}
if (cur) out.push(cur)
return out
}
export function chunk(md, { max = 700, overlapSentences = 0 } = {}) {
const chunks = []
let cur = null
const push = () => { if (cur) { chunks.push(cur); cur = null } }
for (const b of parseBlocks(md)) {
// tables and code are atomic: a half table is worse than an oversized chunk
const pieces = (b.kind === 'prose' && b.text.length > max)
? splitProse(b.text, max) : [b.text]
for (const piece of pieces) {
const samePath = cur && cur.path.join(' > ') === b.path.join(' > ')
if (samePath && cur.text.length + piece.length + 2 <= max) {
cur.text += '\n\n' + piece
if (b.kind !== 'prose') cur.kind = b.kind
} else {
push()
cur = { path: b.path.filter(Boolean), kind: b.kind, text: piece }
}
}
}
push()
if (overlapSentences > 0) {
for (let i = 1; i < chunks.length; i++) {
const prev = chunks[i - 1], here = chunks[i]
if (prev.kind !== 'prose' || here.kind !== 'prose') continue
if (prev.path.join(' > ') !== here.path.join(' > ')) continue
const tail = prev.text.split(SENTENCE).slice(-overlapSentences).join(' ')
here.text = tail + ' ' + here.text
here.overlapped = true
}
}
return chunks
}
Same document, same target size class:
[0] prose 512 Paediatric fever > When to seek help the same day
[1] table 481 Paediatric fever > Antipyretic dosing
[2] prose 135 Paediatric fever > Antipyretic dosing > Contraindications
[3] code 289 Paediatric fever > Recording a reading
sizes {"n":4,"min":135,"p50":481,"p90":512,"max":512}
defects {"chunks":4,"endMidSentence":0,"tableRowsOrphaned":0}
Zero chunks ending mid-sentence, zero orphaned table rows, and the dosing table now sits in one chunk together with the sentence that qualifies it ("per single dose, by weight, for otherwise healthy children") and the sentence that warns against alternating. That grouping was not something the chunker inferred. It is what the blank lines under a heading already said.
The heading path is the metadata that matters
A retrieved fragment arrives with no context. It does not know what document it came from, and more importantly it does not know what section it was under — which for a reference document is frequently the difference between a correct answer and a dangerous one. "15 mg/kg" under Antipyretic dosing is a fact; the same string with no path is a number.
So carry the path on the chunk, as data:
path : ["Paediatric fever","Antipyretic dosing"]
text : "| Drug | Dose | Interval | Max per 24h | / |---|---|---|---| / | Paracetamol | 15 mg/kg | 4-6 h | 60 mg/kg | / | Ibuprofen | 10 mg/kg | 6-8 h | 30 mg/kg |"
That path earns its storage three separate times. It is a filter (restrict retrieval to one section). It is a citation (show the user where the answer came from, which is what makes an answer checkable). And it is text worth prepending to the chunk before you embed it, because "Paediatric fever > Antipyretic dosing" contains query terms that the table body does not.
The same argument applies to the row you store beside it — a small set of explicit fields beats a blob, for the reasons in asking a model for JSON and actually getting JSON.
Overlap, and what it actually buys
Overlap is usually described in vague terms — "so context is not lost". It has a specific job: repairing pairs of adjacent sentences that a boundary separated, so that neither half is retrievable without the other. That is countable. Run the same document at max: 260, and count how many source-adjacent sentence pairs no longer co-occur in any chunk:
overlap 0: {"pairs":8,"separated":1} chunks 9 stored 1408 chars (0.91x the document)
overlap 1: {"pairs":8,"separated":0} chunks 9 stored 1642 chars (1.06x the document)
overlap 2: {"pairs":8,"separated":0} chunks 9 stored 1784 chars (1.15x the document)
One sentence of overlap repaired the single broken adjacency in this document and cost 17% more stored text. A second sentence repaired nothing further and cost another 9%. That is the shape of the trade: overlap has a knee, one unit was enough here, and the only way to find yours is to count rather than to pick a percentage.
The repair is visible:
chunk 1 without overlap: "Above six months, the number matters less than how the child looks: poor feeding, drowsiness, or…"
chunk 1 with overlap 1 : "Between three and six months the threshold is 39 C. Above six months, the number matters less th…"
Note what overlap does not fix. It does not restore a table header, because the overlap here is applied only between prose chunks under the same heading — copying half a table into the next chunk would produce two half-tables instead of one. Overlap is a prose repair. Structure is the table repair.
Measure the distribution, do not guess the number
"Chunk size 500" is not a property of your output. It is a request. What you actually get depends on the document:
max n min p50 p90 max over-target
120 15 39 82 148 181 4
200 9 119 148 196 196 0
300 7 135 187 289 289 0
400 6 135 283 323 323 0
600 4 135 481 512 512 0
1000 4 135 481 512 512 0
Three things fall out of that table, and none of them were guessable.
Above 600 the parameter stops doing anything. The document has four structural groups; asking for 1000 gives you the same four chunks as 600. Tuning that knob upward past the natural block size is tuning nothing.
At 120 the target is violated four times. Those are the atomic blocks — the table at 148 characters, and single sentences up to 181. The chunker is working correctly: max is a target, and keeping a table whole beats honouring it. But you only find out that your "120-character chunks" are sometimes 181 by measuring.
The minimum is unbounded. At max: 120 the smallest chunk is 39 characters. A 39-character chunk is nearly pure noise in a retrieval index; it will match short queries on almost no evidence. If you ship this, add a minimum and merge below it — and set that minimum from a measured distribution, not from a round number.
Print sizes() and defects() on your real corpus before you tune anything. Both are a dozen lines, and they turn an argument into a table.
Chunking decides retrieval more than the model does
This is the claim people push back on, so state it precisely, without pretending to a benchmark.
We have not measured embedding models here, and this article makes no claim about their relative accuracy. The claim is structural, and it is stronger than a benchmark would be: retrieval cannot return information that is not in any chunk. In the fixed-size run above, no chunk contains the string | Paracetamol | 15 mg/kg |. That row does not exist in the index. There is no embedding model, no reranker, no hybrid search and no amount of top-k that retrieves it, because retrieval selects among chunks and that fact is not in one.
Chunking is upstream of everything else in the pipeline, and it is the only stage that can permanently destroy information. Model choice moves results around within what chunking made available. That is why it is worth spending an afternoon on a parser and ten minutes on the model, and it is the same ordering argument as taking the certain fields with a regex before involving a model: do the deterministic work first, because the deterministic work is where correctness is decided.
What this parser does not handle
Four limits, each confirmed by running the code above on the input described:
| Input | What happens |
|---|---|
An HTML <table> |
Classified as prose, not table. It stays whole only because there are no sentence boundaries inside it to split on — which is luck, not design. |
A setext heading (Doses over =====) |
Not recognised. Every chunk gets an empty heading path, so the whole document packs as one section. |
| A document with no headings at all | Same: path: [] on every chunk, and the only boundaries left are blank lines. |
| A bullet list followed by a paragraph | Merged into a single prose chunk, because both sit under the same heading and fit the budget. Usually what you want, and never a decision the parser made deliberately. |
Indented pipe tables are handled — TABLE_ROW is /^\s*\|/, so leading whitespace is fine. That one is worth checking rather than assuming, which is the general rule here: defects() reporting zero orphaned rows on your own corpus proves nothing until you have confirmed the detector recognises your table syntax in the first place. It only knows pipes.
Anything beyond this wants a real markdown parser producing an AST, and at that point the chunker becomes a tree walk over blocks rather than a line loop. The shape of the answer does not change; the parser gets better.
Check it yourself
Save it all as chunkdemo.mjs: the document below, then the js blocks above in order — fixedChunks, parseBlocks, splitProse, chunk — with the export keywords dropped, then the three measurement functions and the harness. No network, no API key, no model call.
The document under test, so your numbers match the ones above (four-backtick fence, because the document itself contains a fenced block):
const DOC = `# Paediatric fever
## When to seek help the same day
A temperature at or above 38 C in an infant under three months needs same-day
assessment. Between three and six months the threshold is 39 C. Above six
months, the number matters less than how the child looks: poor feeding,
drowsiness, or a rash that does not fade under pressure all outrank the reading
on the thermometer.
Parents frequently ask whether a high number is dangerous in itself. It is not.
Fever is a response, not the illness, and the height of it correlates poorly
with how serious the cause is.
## Antipyretic dosing
Doses below are per single dose, by weight, for otherwise healthy children.
Confirm against the product label before giving anything.
| Drug | Dose | Interval | Max per 24h |
|---|---|---|---|
| Paracetamol | 15 mg/kg | 4-6 h | 60 mg/kg |
| Ibuprofen | 10 mg/kg | 6-8 h | 30 mg/kg |
Do not alternate the two on a fixed schedule as a matter of routine. Alternating
doubles the number of opportunities to make an arithmetic error, and the
evidence for it improving comfort is weak.
### Contraindications
Ibuprofen is avoided in dehydration, in chickenpox, and in known renal
impairment. Paracetamol is avoided in significant liver disease.
## Recording a reading
\`\`\`sql
INSERT INTO observations(patient_id, kind, value_c, taken_at)
VALUES (?, 'temperature', ?, datetime('now'));
\`\`\`
Record the route as well as the number. An axillary reading runs roughly 0.5 C
below a rectal one, and a comparison between the two over time is meaningless
without it.
`
export function sizes(chunks) {
const l = chunks.map((c) => (c.text ?? c).length).sort((a, b) => a - b)
const q = (p) => l[Math.min(l.length - 1, Math.floor(p * l.length))]
return { n: l.length, min: l[0], p50: q(0.5), p90: q(0.9), max: l[l.length - 1] }
}
const ENDS_CLEAN = /[.!?:|`]\s*$/
export function defects(chunks) {
const texts = chunks.map((c) => c.text ?? c)
const midSentence = texts.filter((t) => !ENDS_CLEAN.test(t.trim())).length
const rowsWithoutHeader = texts.filter(
(t) => /^\s*\|/m.test(t) && !/\|\s*-{2,}/.test(t)).length
return { chunks: texts.length, endMidSentence: midSentence, tableRowsOrphaned: rowsWithoutHeader }
}
export function brokenAdjacencies(md, chunks) {
const texts = chunks.map((c) => c.text ?? c)
let broken = 0, total = 0
for (const b of parseBlocks(md)) {
if (b.kind !== 'prose') continue
const s = b.text.split(SENTENCE)
for (let i = 0; i + 1 < s.length; i++) {
total++
const a = s[i].replace(/\s+/g, ' '), z = s[i + 1].replace(/\s+/g, ' ')
const ok = texts.some((t) => {
const flat = t.replace(/\s+/g, ' ')
return flat.includes(a) && flat.includes(z)
})
if (!ok) broken++
}
}
return { pairs: total, separated: broken }
}
// ---- harness ----
const show = (t, n = 78) => JSON.stringify(t.length > n ? t.slice(0, n) + '…' : t)
console.log('document:', DOC.length, 'chars\n')
const fixed = fixedChunks(DOC, 400)
fixed.forEach((c, i) => console.log(` [${i}] ${show(c.trim())}`))
console.log(' sizes ', JSON.stringify(sizes(fixed)))
console.log(' defects', JSON.stringify(defects(fixed)))
console.log(' the table row that got cut:')
console.log(' ...' + JSON.stringify(fixed[1].slice(-40)) + ' | ' + JSON.stringify(fixed[2].slice(0, 40)) + '...\n')
const st = chunk(DOC, { max: 700 })
st.forEach((c, i) => console.log(` [${i}] ${c.kind.padEnd(5)} ${String(c.text.length).padStart(4)} ${c.path.join(' > ')}`))
console.log(' sizes ', JSON.stringify(sizes(st)))
console.log(' defects', JSON.stringify(defects(st)), '\n')
const tbl = st.find((c) => c.kind === 'table')
console.log(' path :', JSON.stringify(tbl.path))
console.log(' text :', show(tbl.text.split('\n').filter((l) => l.startsWith('|')).join(' / '), 200))
console.log(' fixed-size index contains the paracetamol row:',
fixed.some((c) => c.includes('| Paracetamol | 15 mg/kg |')), '\n')
console.log(' max n min p50 p90 max over-target')
for (const max of [120, 200, 300, 400, 600, 1000]) {
const cs = chunk(DOC, { max })
const s = sizes(cs)
const over = cs.filter((c) => c.text.length > max).length
console.log(` ${String(max).padEnd(6)}${String(s.n).padEnd(4)}${String(s.min).padEnd(5)}${String(s.p50).padEnd(5)}${String(s.p90).padEnd(5)}${String(s.max).padEnd(6)}${over}`)
}
for (const ov of [0, 1, 2]) {
const cs = chunk(DOC, { max: 260, overlapSentences: ov })
const chars = cs.reduce((a, c) => a + c.text.length, 0)
console.log(` overlap ${ov}: ${JSON.stringify(brokenAdjacencies(DOC, cs))}` +
` chunks ${cs.length} stored ${chars} chars (${(chars / DOC.length).toFixed(2)}x the document)`)
}
node chunkdemo.mjs
Then paste in one of your own documents and read the sweep. If it is flat, as it is here above 600, your max is already larger than the document's natural blocks and every hour you spend tuning that number is wasted.
Where this goes next
Chunks have to land somewhere that can be searched without a network. In MedSearch that is a SQLite file on the device, one row per chunk with the heading path in its own column — the index side of this is full-text search that works on a plane, where the heading path becomes a weighted column in BM25 and the chunk body becomes the snippet.
The privacy argument arrives with it. A chunk you never send is a chunk nobody else stores, which is the same reasoning as scanning an inbox without leaking the inbox.
