Why does my search miss the obvious answer?

Because you picked one retriever. On 60 labelled queries against the same 300 documents, BM25 put the correct document first for 100% of exact-identifier searches and 13.3% of paraphrases; a local embedding model scored 40.0% and 73.3% on those same two sets. Overall recall@1 was 63.3% for BM25 and 66.7% for the vectors — a 3.4-point gap concealing an 86.7-point swing inside BM25 alone.

Run on an Apple M3, 16 GB, macOS 26.4.1: Node v22.22.2, better-sqlite3 13.0.3 (SQLite 3.53.4, FTS5), @xenova/transformers 2.17.2 running Xenova/all-MiniLM-L6-v2 quantized at 384 dimensions. No API key, no network after the model is cached.

The corpus is synthetic and that matters. The absolute percentages belong to this corpus and no other. The transferable finding is the pattern by query type: which method collapses on which shape of query. Read the columns against each other, not the numbers alone.

The short answer

• BM25 scored 100.0% recall@1 on exact identifiers and 13.3% on paraphrases; the embedding model scored 40.0% and 73.3%. Each retriever fails one class of query near-totally, and the two classes do not overlap.
• Aggregate scores hide this completely. Overall recall@1 differed by 3.4 points between the two methods while the by-type numbers differed by 60. Never accept a retrieval number without a breakdown by query shape.
• Vector search does not rescue typos in identifiers. Misspell a part number and vector recall@1 was 14.3% against BM25's 0.0% — both useless. Spelled correctly: BM25 100.0%, vector 28.6%.
• A normalised weighted sum beat Reciprocal Rank Fusion (MRR 0.785 against 0.770), and w = 0.50 — the value everyone starts with — sits on a cliff: 0.48 gave 75.0% recall@1, 0.50 gave 66.7%.
• Fusion is insurance, not improvement. Hybrid ranked the answer worse than the better single method on 9 to 13 of 60 queries, and better on at most
  1. It wins by never being the wrong choice, not by finding anything new.

How was this measured?

A synthetic service knowledge base: 300 documents, 83,411 characters, median length 278. Each document is one (component, symptom) pair drawn without repetition from 20 components and 16 symptoms, so every pair is unique and the correct answer to a query about it is exactly one document — ground truth with no judgement calls and no copyrighted text. Each carries three identifiers unique across the corpus: an error code, a part number, a firmware version.

ERR-7938: servo fails to authenticate
Applies to part CW-2325-N running firmware v4.10.29. The servo fails to
authenticate after the first hour of operation. Photograph the label before
removing it. The bench procedure takes about forty minutes.

The 60 queries split into five types, each with one correct document:

The paraphrase set is load-bearing, so it is built adversarially: every symptom is written twice, once for the document and once for the query, with no content word shared between the two phrasings. emits a high-pitched whine becomes makes a squealing noise. The script asserts it:

paraphrase lexical-overlap check: 0 of 15 queries share a symptom word with their target doc

The component name is left in the query deliberately. Without it BM25 returns nothing and the comparison is a strawman; with it, BM25 has a foothold and must choose among the fifteen documents about that component.

Keyword side: FTS5 external-content table, bm25() with default column weights, top 50 — the mechanics are in full-text search that works on a plane. Vector side: mean-pooled, L2-normalised MiniLM embeddings of title + body, brute-force cosine over all 300, instant at this size for the reasons in do you actually need a vector database. The FTS5 index built in 0.8 ms; embedding the documents took 3,043 ms, 10.1 ms each.

One query-builder decision changes everything. FTS5 treats a b c as an implicit AND:

=== FTS5 zero-result rate: implicit AND vs explicit OR ===
  exact-id     AND: 0/15 empty   OR: 0/15 empty
  paraphrase   AND: 15/15 empty   OR: 0/15 empty
  keyword      AND: 0/15 empty   OR: 0/15 empty
  typo-word    AND: 8/8 empty   OR: 0/8 empty
  typo-id      AND: 7/7 empty   OR: 7/7 empty

Every paraphrase and every misspelled query returns nothing. All BM25 numbers below therefore use an explicit OR of quoted terms. If you have ever concluded keyword search is hopeless on natural language, check whether you measured BM25 or measured implicit AND.

What does BM25 miss?

Paraphrases — and it misses them not by returning nothing, but by returning fifteen plausible neighbours in the wrong order.

  [paraphrase] "thermostat makes a squealing noise"   gold = ERR-6926: thermostat emits a high-pitched whine
     BM25 rank 15, vector rank 1, RRF k=60 rank 6
     BM25 top 3   : ERR-1650: thermostat vibrates excessively | ERR-8836: thermostat leaks coolant | ERR-1099: thermostat reports inaccurate readings
     vector top 3 : ERR-6926: thermostat emits a high-pitched whine | ERR-1650: thermostat vibrates excessively | ERR-3927: thermostat triggers a false alarm

BM25 ranked the answer fifteenth — last among the thermostat documents, because thermostat is the only term it can match and the tie-break is document length. The results look relevant; they are all thermostat faults. That is the failure mode nobody reports as a bug: a page of near-misses reads as "the answer isn't in there".

What does vector search miss?

Identifiers, and it misses them by confidently returning different identifiers.

  [exact-id] "ERR-7938"   gold = ERR-7938: servo fails to authenticate
     BM25 rank 1, vector rank 4, RRF k=60 rank 1
     BM25 top 3   : ERR-7938: servo fails to authenticate
     vector top 3 : ERR-7924: actuator freezes during firmware update | ERR-8940: compressor corrupts stored logs | ERR-5421: actuator loses its network address

ERR-7938 and ERR-7924 are one digit apart and mean nothing to each other. The tokenizer splits both into subword pieces, mean pooling averages them, and what survives is "this is an error code" — the category, not the value. Embeddings are built to blur, and an identifier is the one input where blurring is the whole error.

Both effects side by side — the table this article exists for:

The last row is why teams ship the wrong thing. Judged on it alone the two methods are within four points and you would choose on latency or cost. Judged by type, BM25's own recall@1 swings 86.7 points on what the user typed.

Does vector search handle typos?

This is where the expectation was clearest and the measurement least kind. The typo sets are small — 8 and 7 queries — so each was run again spelled correctly, as a matched control:

  set          n   BM25 typo -> clean      vector typo -> clean     trigram BM25 (typo)
  typo-word    8    75.0% -> 100.0%      87.5% -> 100.0%      75.0%
  typo-id      7     0.0% -> 100.0%      14.3% ->  28.6%       0.0%

Three things fall out, and only the first is the expected one.

On a misspelled ordinary word, vector beat BM25 — by one query in eight. That is 7 of 8 against 6 of 8; do not build a plan on it. BM25's 75% is not robustness either: those queries are three or four words long, one word is wrong, and the OR query still matches the rest.

On a misspelled identifier, neither method works. BM25 goes to 0.0% because a mistyped token matches no token at all. Vector goes to 14.3% — from a clean baseline of only 28.6%, because vectors were never good at identifiers. The case where people most expect embeddings to save them is the case where embeddings have least to offer, because the signal the typo destroys is exactly the signal embeddings already discard.

The trigram tokenizer does not fix it either, which is the usual next suggestion. FTS5's tokenize='trigram' gives substring matching, not fuzzy matching:

  match "ERR-8790"     -> 1        match "790: servo"   -> 1
  match "8790"         -> 1        match "ERR-879"      -> 1
  match "ERR-879O"     -> 0        match "servo"        -> 1
  match "ERR-8709"     -> 0        match "srvo"         -> 0

Substrings hit; a single wrong character misses. Over the whole query set a trigram index scored 100.0% recall@1 on exact identifiers and 0.0% on mistyped ones — the same shape as ordinary BM25. Typo tolerance is a spelling-correction or edit-distance problem. It is not a retrieval-model problem, and reaching for embeddings to solve it is reaching for the wrong tool.

Is RRF better than a weighted sum?

Reciprocal Rank Fusion adds 1 / (k + rank) across lists and ignores scores entirely. A weighted sum min-max normalises each list's scores to [0, 1], gives absent documents 0, and adds w · vector + (1 − w) · bm25. Both swept:

=== RRF k sweep ===            === weighted-sum sweep, w on the vector side ===
  k=1     68.3%  88.3%  0.770    w=0.00  65.0%  76.7%  0.702
  k=5     65.0%  81.7%  0.728    w=0.20  66.7%  80.0%  0.723
  k=10    65.0%  81.7%  0.716    w=0.40  71.7%  80.0%  0.764
  k=20    65.0%  76.7%  0.712    w=0.50  68.3%  80.0%  0.752
  k=60    65.0%  75.0%  0.711    w=0.60  65.0%  81.7%  0.720
  k=100   65.0%  75.0%  0.711    w=0.80  66.7%  76.7%  0.724
  k=200   65.0%  75.0%  0.711    w=1.00  66.7%  76.7%  0.719

Four results, three of which contradict the usual advice.

Tuning beat the default on both. RRF's famous k=60 was the worst setting swept: k=1 moved recall@5 from 75.0% to 88.3%. The larger k is, the more it flattens rank 1 against rank 20 — sensible across many long lists from many systems, wasteful with two lists of fifty over 300 documents.

The weighted sum won. MRR 0.785 against 0.770 at their best, 0.752 against 0.711 at their defaults. RRF is recommended because it needs no normalisation and no tuning; here, what it throws away — how far ahead the top hit was — was worth keeping.

Even w = 0.00 beat BM25 alone (MRR 0.702 against 0.681) while using only BM25's scores — the gain is purely the vector list supplying candidates BM25 never returned. Union the candidate sets before you argue about weights.

And w = 0.50 is the worst place to put the weight. The 0.02 grid:

    w=0.44  75.0%  80.0%  0.785
    w=0.46  75.0%  80.0%  0.785
    w=0.48  75.0%  80.0%  0.785
    w=0.50  66.7%  80.0%  0.744
    w=0.52  66.7%  80.0%  0.738

An 8.3-point cliff between 0.48 and 0.50, and the cause is a general defect of normalised weighted sums:

  "firmware v5.19.5" w=0.48  #10(gold) 0.5200 [bm25 1.00 vec 0.00]   #144 0.4800 [bm25 0.00 vec 1.00]
  "firmware v5.19.5" w=0.50  #144 0.5000 [bm25 0.00 vec 1.00]   #10(gold) 0.5000 [bm25 1.00 vec 0.00]

Min-max normalisation forces every list's top hit to exactly 1.0. At w = 0.50 a document first on one list and absent from the other ties exactly with a document first on the other — 0.5000 against 0.5000 — and an arbitrary tie-break decides your search result. RRF has no equivalent.

One honest caveat: the sweep was run on the same 60 queries it is scored against. w = 0.48 is fitted to this test set, so do not copy it. Run your own sweep on your own labelled queries — and note that the transferable finding is the cliff at the default, which is a property of the method, not the corpus.

Does fusion ever beat both methods?

Almost never, and this changed how we describe hybrid search to clients:

  RRF k=60           worse than the better single method on 13/60, better on 1/60
  RRF k=1            worse than the better single method on 13/60, better on 0/60
  weighted w=0.48    worse than the better single method on 9/60, better on 0/60

Take, per query, the better rank of the two single methods. Fusion beat that best-of-both on one query in 60 across every setting tried, and lost to it on 9 to 13.

Hybrid finds no document that neither retriever found. It is a hedge against not knowing which kind of query the user will type, and since a real search box gets all the kinds, the hedge is worth having. The by-type table is the honest sales pitch: hybrid took paraphrase recall@1 from BM25's 13.3% to 53.3% and identifier recall@1 from the vectors' 40.0% to 100.0%.

What does hybrid cost per query?

Median of 60 queries × 5 runs. Across three separate runs the query-embedding figure ranged 1.35–1.96 ms and BM25 stayed at 0.031–0.033 ms.

Embedding the query costs 59x the entire keyword search, and is about 89% of hybrid latency. The retrieval is free; the model is the bill. Caching query embeddings is therefore the highest-value optimisation available, and running the two retrievers concurrently saves nothing — one of them costs 0.033 ms.

Check it yourself

Two files, no API key, and no network after the first run — which downloads about 23 MB of quantized ONNX weights into ./.models.

mkdir hybrid-search && cd hybrid-search && npm init -y
npm install better-sqlite3 @xenova/transformers

better-sqlite3 rather than the built-in node:sqlite for one reason: at Node v23.5.0 the built-in reported SQLite 3.47.2 with no FTS5 compiled in (no such module: fts5). Check yours before assuming.

Save this as hybrid.mjs and run node hybrid.mjs. Corpus and queries are seeded, so every retrieval number reproduces exactly and only timings move.

// hybrid.mjs -- BM25 (SQLite FTS5) vs vector (all-MiniLM-L6-v2) vs hybrid,
// on a synthetic knowledge base with unambiguous ground truth.
import Database from 'better-sqlite3'
import { pipeline, env } from '@xenova/transformers'

env.cacheDir = './.models'
const rnd = (s) => () => (s = (s + 0x6d2b79f5) | 0, ((Math.imul(s ^ (s >>> 15), 1 | s) ^ (Math.imul(s ^ (s >>> 7), 61 | s) + s)) >>> 0) / 4294967296)

// ---------------------------------------------------------------- corpus
const COMPONENTS = ['pump','valve','sensor','controller','relay','thermostat','compressor',
  'actuator','encoder','gateway','chiller','blower','injector','manifold','servo',
  'igniter','condenser','regulator','turbine','diffuser']

// [ phrasing used in the DOCUMENT , phrasing used in the PARAPHRASE QUERY ]
// The two halves share no content word. That is the whole point of the test.
const SYMPTOMS = [
  ['refuses to start',            'will not power up'],
  ['leaks coolant',               'fluid escaping'],
  ['overheats under load',        'runs too hot when busy'],
  ['drops packets intermittently','loses network traffic now and then'],
  ['reports inaccurate readings', 'measurements are wrong'],
  ['vibrates excessively',        'shakes far too much'],
  ['fails to authenticate',       'login is rejected'],
  ['loses calibration over time', 'drifts out of adjustment'],
  ['consumes excessive current',  'draws too much power'],
  ['emits a high-pitched whine',  'makes a squealing noise'],
  ['freezes during firmware update','hangs while upgrading software'],
  ['corrupts stored logs',        'damages saved records'],
  ['sticks in the closed position','jams shut'],
  ['responds slowly to commands', 'is sluggish when told what to do'],
  ['triggers a false alarm',      'warns about a trouble that is not real'],
  ['loses its network address',   'forgets its IP assignment'],
]

const FILLER = [
  'Escalate to field service if the condition persists after two attempts.',
  'Note the serial number and the date of the visit in the service history.',
  'Spares for this unit are held in the regional depot.',
  'Covered by the standard warranty for the first eighteen months.',
  'A replacement gasket kit ships with every site call.',
  'Isolate the circuit before opening the enclosure.',
  'Torque the mounting bolts to twelve newton metres.',
  'The bench procedure takes about forty minutes.',
  'Customers on the extended plan receive a loan unit.',
  'Photograph the label before removing it.',
]

const STOP = new Set(['the','a','an','is','are','it','its','to','of','in','on','and','or','not',
  'that','when','has','have','been','be','too','far','out','over','do','does','with','for','after','my','i','while','about','what','told'])
const toks = (s) => (s.toLowerCase().match(/[a-z0-9]+/g) || [])

function buildCorpus() {
  const r = rnd(20260831)
  const pairs = []
  for (const c of COMPONENTS) for (let s = 0; s < SYMPTOMS.length; s++) pairs.push([c, s])
  for (let i = pairs.length - 1; i > 0; i--) { const j = Math.floor(r() * (i + 1)); [pairs[i], pairs[j]] = [pairs[j], pairs[i]] }
  const chosen = pairs.slice(0, 300)

  const errs = new Set(), skus = new Set(), vers = new Set()
  const uniq = (set, gen) => { let v; do { v = gen() } while (set.has(v)); set.add(v); return v }
  const L = 'ABCDEFGHJKLMNPRSTVWXYZ'

  return chosen.map(([comp, sIdx], i) => {
    const err = uniq(errs, () => 'ERR-' + (1000 + Math.floor(r() * 9000)))
    const sku = uniq(skus, () => L[Math.floor(r()*L.length)] + L[Math.floor(r()*L.length)] + '-' +
      (1000 + Math.floor(r()*9000)) + '-' + L[Math.floor(r()*L.length)])
    const ver = uniq(vers, () => `v${1+Math.floor(r()*6)}.${Math.floor(r()*20)}.${Math.floor(r()*40)}`)
    const f1 = FILLER[Math.floor(r() * FILLER.length)]
    let f2 = FILLER[Math.floor(r() * FILLER.length)]
    if (f2 === f1) f2 = FILLER[(FILLER.indexOf(f1) + 1) % FILLER.length]
    const title = `${err}: ${comp} ${SYMPTOMS[sIdx][0]}`
    const body = `Applies to part ${sku} running firmware ${ver}. The ${comp} ${SYMPTOMS[sIdx][0]} after the first hour of operation. ${f1} ${f2}`
    return { id: i + 1, comp, sIdx, err, sku, ver, title, body, text: title + '. ' + body }
  })
}

// ---------------------------------------------------------------- queries
function typoWord(w, r) {
  const k = Math.floor(r() * 4), i = 1 + Math.floor(r() * Math.max(1, w.length - 2))
  if (k === 0) return w.slice(0, i) + w[i + 1] + w[i] + w.slice(i + 2)      // transpose
  if (k === 1) return w.slice(0, i) + w.slice(i + 1)                        // delete
  if (k === 2) return w.slice(0, i) + w[i] + w.slice(i)                     // double
  return w.slice(0, i) + 'aeiourstn'[Math.floor(r() * 9)] + w.slice(i + 1)  // substitute
}
// one character wrong, structure kept: O/0 and l/1 confusion, or two digits swapped
const typoId = (s, r) => {
  const map = { '0': 'O', 'O': '0', '1': 'l', 'l': '1', '5': 'S', 'S': '5', '8': 'B', 'B': '8' }
  const idx = [...s].map((c, i) => (map[c] ? i : -1)).filter((i) => i >= 0)
  if (idx.length && r() < 0.6) {
    const i = idx[Math.floor(r() * idx.length)]
    return s.slice(0, i) + map[s[i]] + s.slice(i + 1)
  }
  const d = [...s].map((c, i) => (/[0-9]/.test(c) && /[0-9]/.test(s[i + 1] || '') ? i : -1)).filter((i) => i >= 0)
  const i = d[Math.floor(r() * d.length)]
  return s.slice(0, i) + s[i + 1] + s[i] + s.slice(i + 2)
}

function buildQueries(docs) {
  const r = rnd(77)
  const pick = (n, skip = 0) => docs.filter((_, i) => i % 7 === skip).slice(0, n)
  const Q = []
  // 1. exact identifier: 5 error codes, 5 part numbers, 5 firmware versions
  pick(5, 0).forEach((d) => Q.push({ type: 'exact-id', q: d.err, gold: d.id }))
  pick(5, 1).forEach((d) => Q.push({ type: 'exact-id', q: d.sku, gold: d.id }))
  pick(5, 2).forEach((d) => Q.push({ type: 'exact-id', q: `firmware ${d.ver}`, gold: d.id }))
  // 2. paraphrase: component + a restatement sharing no content word with the doc
  pick(15, 3).forEach((d) => Q.push({ type: 'paraphrase', q: `${d.comp} ${SYMPTOMS[d.sIdx][1]}`, gold: d.id }))
  // 3. keyword: the document's own words
  pick(15, 4).forEach((d) => Q.push({ type: 'keyword', q: `${d.comp} ${SYMPTOMS[d.sIdx][0]}`, gold: d.id }))
  // 4. typo: 8 in a natural-language word, 7 in an identifier
  pick(8, 5).forEach((d) => {
    const w = [d.comp, ...SYMPTOMS[d.sIdx][0].split(' ')].filter((x) => x.length > 4)
    const t = w[Math.floor(r() * w.length)]
    const clean = `${d.comp} ${SYMPTOMS[d.sIdx][0]}`
    Q.push({ type: 'typo-word', q: clean.replace(t, typoWord(t, r)), clean, gold: d.id })
  })
  pick(7, 6).forEach((d) => {
    const clean = r() < 0.5 ? d.err : d.sku
    Q.push({ type: 'typo-id', q: typoId(clean, r), clean, gold: d.id })
  })
  return Q
}

// ---------------------------------------------------------------- retrieval
const K = 50
function fts5Query(q, mode = 'or') {
  const t = q.split(/\s+/).map((w) => w.replace(/"/g, '')).filter((w) => w && !STOP.has(w.toLowerCase()))
  if (!t.length) return null
  return t.map((w) => `"${w}"`).join(mode === 'or' ? ' OR ' : ' ')
}
const cos = (a, b) => { let s = 0; for (let i = 0; i < a.length; i++) s += a[i] * b[i]; return s }
function topK(scores, k) {
  return scores.map((s, i) => [s, i + 1]).sort((a, b) => b[0] - a[0]).slice(0, k)
}

// ---------------------------------------------------------------- fusion
function rrf(lists, k) {
  const m = new Map()
  for (const l of lists) l.forEach(([, id], i) => m.set(id, (m.get(id) || 0) + 1 / (k + i + 1)))
  return [...m].sort((a, b) => b[1] - a[1]).map(([id]) => id)
}
function normalise(list) {                       // min-max within the list
  const m = new Map()
  if (!list.length) return m
  const v = list.map(([s]) => s), lo = Math.min(...v), hi = Math.max(...v)
  list.forEach(([s, id]) => m.set(id, hi === lo ? 1 : (s - lo) / (hi - lo)))
  return m
}
function weighted(bm, ve, w) {                   // w = weight on the vector side
  const B = normalise(bm), V = normalise(ve), out = new Map()
  for (const id of new Set([...B.keys(), ...V.keys()]))
    out.set(id, w * (V.get(id) || 0) + (1 - w) * (B.get(id) || 0))
  return [...out].sort((a, b) => b[1] - a[1]).map(([id]) => id)
}

// ---------------------------------------------------------------- metrics
const rankOf = (ids, gold) => { const i = ids.indexOf(gold); return i < 0 ? 0 : i + 1 }
function metrics(ranks) {
  const n = ranks.length
  return {
    n,
    r1: ranks.filter((r) => r === 1).length / n,
    r5: ranks.filter((r) => r >= 1 && r <= 5).length / n,
    mrr: ranks.reduce((a, r) => a + (r ? 1 / r : 0), 0) / n,
  }
}
const pct = (x) => (x * 100).toFixed(1).padStart(5)
const f3 = (x) => x.toFixed(3).padStart(5)

// ---------------------------------------------------------------- run
const docs = buildCorpus()
const queries = buildQueries(docs)
const L2 = docs.map((d) => d.text.length).sort((a, b) => a - b)
console.log(`node ${process.version}  better-sqlite3 ${(await import('better-sqlite3/package.json', { with: { type: 'json' } })).default.version}` +
  `  transformers.js ${(await import('@xenova/transformers/package.json', { with: { type: 'json' } })).default.version}`)
console.log(`corpus: ${docs.length} docs, ${docs.reduce((a, d) => a + d.text.length, 0)} chars, ` +
  `doc length min ${L2[0]} / median ${L2[150]} / max ${L2[299]}`)
console.log(`queries: ${queries.length}  ` + JSON.stringify(
  Object.fromEntries([...new Set(queries.map(q=>q.type))].map(t=>[t,queries.filter(q=>q.type===t).length]))))

// sanity: no paraphrase content word (other than the component) is in the target doc
let leak = 0
for (const q of queries.filter((x) => x.type === 'paraphrase')) {
  const gold = docs.find((d) => d.id === q.gold)
  const dt = new Set(toks(gold.text))
  const bad = toks(SYMPTOMS[gold.sIdx][1]).filter((w) => !STOP.has(w) && dt.has(w))
  if (bad.length) { leak++; console.log('  LEAK', q.q, bad) }
}
console.log(`paraphrase lexical-overlap check: ${leak} of 15 queries share a symptom word with their target doc`)

const db = new Database(':memory:')
db.exec(`create table docs(id integer primary key, title text, body text);
         create virtual table docs_fts using fts5(title, body, content='docs', content_rowid='id');
         create virtual table docs_tri using fts5(title, body, content='docs', content_rowid='id', tokenize='trigram');`)
const ins = db.prepare('insert into docs values (?,?,?)')
db.transaction(() => docs.forEach((d) => ins.run(d.id, d.title, d.body)))()
const tF = process.hrtime.bigint()
db.exec("insert into docs_fts(rowid, title, body) select id, title, body from docs")
db.exec("insert into docs_fts(docs_fts) values('optimize')")
const ftsBuildMs = Number(process.hrtime.bigint() - tF) / 1e6
db.exec("insert into docs_tri(rowid, title, body) select id, title, body from docs")
db.exec("insert into docs_tri(docs_tri) values('optimize')")
const SEL = db.prepare(`select rowid as id, bm25(docs_fts) as s from docs_fts
                        where docs_fts match ? order by rank limit ${K}`)
const TRI = db.prepare(`select rowid as id, bm25(docs_tri) as s from docs_tri
                        where docs_tri match ? order by rank limit ${K}`)
const bm25 = (q, mode = 'or') => { const m = fts5Query(q, mode); if (!m) return []
  try { return SEL.all(m).map((r) => [-r.s, r.id]) } catch { return [] } }
const trigram = (q) => {
  const t = q.split(/\s+/).map((w) => w.replace(/"/g, '')).filter((w) => w.length >= 3 && !STOP.has(w.toLowerCase()))
  if (!t.length) return []
  try { return TRI.all(t.map((w) => `"${w}"`).join(' OR ')).map((r) => [-r.s, r.id]) } catch { return [] } }
console.log(`FTS5 index build: ${ftsBuildMs.toFixed(1)} ms for ${docs.length} docs`)

const t0 = Date.now()
const pipe = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', { quantized: true })
console.log(`model load: ${((Date.now()-t0)/1000).toFixed(1)} s`)

const embed = async (texts) => {
  const o = await pipe(texts, { pooling: 'mean', normalize: true })
  const d = o.dims[1]
  return Array.from({ length: o.dims[0] }, (_, i) => o.data.slice(i * d, (i + 1) * d))
}
const tEmb = Date.now()
const dvecs = []
for (let i = 0; i < docs.length; i += 32) dvecs.push(...await embed(docs.slice(i, i + 32).map((d) => d.text)))
const embedMs = Date.now() - tEmb
console.log(`indexed ${docs.length} docs in ${embedMs} ms (${(embedMs/docs.length).toFixed(1)} ms/doc), dim ${dvecs[0].length}\n`)

// per-query candidate lists
const R = []
for (const q of queries) {
  const b = bm25(q.q)
  const [qv] = await embed([q.q])
  const v = topK(dvecs.map((d) => cos(d, qv)), K)
  let bc = null, vc = null
  if (q.clean) { bc = bm25(q.clean); const [cv] = await embed([q.clean]); vc = topK(dvecs.map((d) => cos(d, cv)), K) }
  R.push({ ...q, qv, b, v, t: trigram(q.q), bAnd: bm25(q.q, 'and'), bc, vc })
}

const TYPES = ['exact-id', 'paraphrase', 'keyword', 'typo-word', 'typo-id']
const by = (f, sel = () => true) => metrics(R.filter(sel).map((r) => rankOf(f(r), r.gold)))

const RUNS = {
  'BM25 only':         (r) => r.b.map(([, id]) => id),
  'Vector only':       (r) => r.v.map(([, id]) => id),
  'Hybrid RRF k=60':   (r) => rrf([r.b, r.v], 60),
  'Hybrid weighted .5':(r) => weighted(r.b, r.v, 0.5),
}
console.log('=== overall, 60 queries ===')
console.log('  method                 recall@1  recall@5    MRR')
for (const [name, f] of Object.entries(RUNS)) {
  const m = by(f)
  console.log(`  ${name.padEnd(22)}${pct(m.r1)}%   ${pct(m.r5)}%  ${f3(m.mrr)}`)
}

console.log('\n=== by query type: recall@1 / recall@5 / MRR ===')
console.log('  query type    n   ' + Object.keys(RUNS).map((k) => k.padEnd(22)).join(''))
for (const t of TYPES) {
  const cells = Object.values(RUNS).map((f) => {
    const m = by(f, (r) => r.type === t)
    return `${pct(m.r1)} ${pct(m.r5)} ${f3(m.mrr)}`.padEnd(22)
  })
  console.log(`  ${t.padEnd(12)}${String(R.filter(r=>r.type===t).length).padStart(3)}   ` + cells.join(''))
}

console.log('\n=== FTS5 zero-result rate: implicit AND vs explicit OR ===')
for (const t of TYPES) {
  const s = R.filter((r) => r.type === t)
  console.log(`  ${t.padEnd(12)} AND: ${s.filter((r)=>!r.bAnd.length).length}/${s.length} empty   OR: ${s.filter((r)=>!r.b.length).length}/${s.length} empty`)
}

console.log('\n=== RRF k sweep (recall@1 / recall@5 / MRR) ===')
for (const k of [1, 5, 10, 20, 60, 100, 200]) {
  const m = by((r) => rrf([r.b, r.v], k))
  console.log(`  k=${String(k).padEnd(4)} ${pct(m.r1)}% ${pct(m.r5)}%  ${f3(m.mrr)}`)
}

console.log('\n=== weighted-sum sweep, w = weight on the vector score ===')
let bestW = null
for (let w = 0; w <= 1.0001; w += 0.1) {
  const m = by((r) => weighted(r.b, r.v, w))
  if (!bestW || m.mrr > bestW.m.mrr) bestW = { w, m }
  console.log(`  w=${w.toFixed(2)} ${pct(m.r1)}% ${pct(m.r5)}%  ${f3(m.mrr)}`)
}
let fine = null
for (let w = 0; w <= 1.0001; w += 0.02) {
  const m = by((r) => weighted(r.b, r.v, w))
  if (!fine || m.mrr > fine.m.mrr) fine = { w, m }
}
console.log(`  best on the 0.10 grid: w=${bestW.w.toFixed(2)}  r@1 ${pct(bestW.m.r1)}%  r@5 ${pct(bestW.m.r5)}%  MRR ${f3(bestW.m.mrr)}`)
console.log('  the same sweep on a 0.02 grid, w = 0.30 .. 0.60:')
for (let w = 0.30; w <= 0.6001; w += 0.02) {
  const m = by((r) => weighted(r.b, r.v, w))
  console.log(`    w=${w.toFixed(2)} ${pct(m.r1)}% ${pct(m.r5)}%  ${f3(m.mrr)}`)
}
console.log(`  best on the 0.02 grid: w=${fine.w.toFixed(2)}  r@1 ${pct(fine.m.r1)}%  r@5 ${pct(fine.m.r5)}%  MRR ${f3(fine.m.mrr)}`)

console.log('\n=== by type at the tuned settings ===')
const TUNED = { 'RRF k=60': (r) => rrf([r.b, r.v], 60), [`weighted w=${fine.w.toFixed(2)}`]: (r) => weighted(r.b, r.v, fine.w) }
for (const t of TYPES) {
  const cells = Object.entries(TUNED).map(([, f]) => { const m = by(f, (r) => r.type === t); return `${pct(m.r1)} ${pct(m.r5)} ${f3(m.mrr)}`.padEnd(22) })
  console.log(`  ${t.padEnd(12)}` + cells.join(''))
}

console.log('\n=== does the typo cause the miss? same queries, spelled correctly ===')
console.log('  set          n   BM25 typo -> clean      vector typo -> clean     trigram BM25 (typo)')
for (const t of ['typo-word', 'typo-id']) {
  const s2 = R.filter((r) => r.type === t)
  const m = (f) => metrics(s2.map((r) => rankOf(f(r), r.gold)))
  const a = m((r) => r.b.map(([, i]) => i)),  ac = m((r) => r.bc.map(([, i]) => i))
  const v = m((r) => r.v.map(([, i]) => i)),  vc = m((r) => r.vc.map(([, i]) => i))
  const tg = m((r) => r.t.map(([, i]) => i))
  console.log(`  ${t.padEnd(12)}${String(s2.length).padStart(2)}   ` +
    `${pct(a.r1)}% -> ${pct(ac.r1)}%     ${pct(v.r1)}% -> ${pct(vc.r1)}%     ${pct(tg.r1)}%`)
}

console.log('\n=== trigram tokenizer as the third arm (recall@1 / recall@5 / MRR) ===')
for (const t of TYPES) {
  const m = by((r) => r.t.map(([, i]) => i), (r) => r.type === t)
  const h = by((r) => rrf([r.b, r.v, r.t], 60), (r) => r.type === t)
  console.log(`  ${t.padEnd(12)} trigram ${pct(m.r1)} ${pct(m.r5)} ${f3(m.mrr)}    RRF(bm25+vec+trigram) ${pct(h.r1)} ${pct(h.r5)} ${f3(h.mrr)}`)
}
{
  const m = by((r) => r.t.map(([, i]) => i)), h = by((r) => rrf([r.b, r.v, r.t], 60))
  console.log(`  ${'ALL'.padEnd(12)} trigram ${pct(m.r1)} ${pct(m.r5)} ${f3(m.mrr)}    RRF(bm25+vec+trigram) ${pct(h.r1)} ${pct(h.r5)} ${f3(h.mrr)}`)
}

console.log('\n=== how often does fusion make a query worse? ===')
const single = (r) => Math.min(...[rankOf(r.b.map(([, i]) => i), r.gold), rankOf(r.v.map(([, i]) => i), r.gold)]
  .map((x) => (x === 0 ? 999 : x)))
for (const [name, f] of Object.entries({ 'RRF k=60': (r) => rrf([r.b, r.v], 60), 'RRF k=1': (r) => rrf([r.b, r.v], 1),
    [`weighted w=${fine.w.toFixed(2)}`]: (r) => weighted(r.b, r.v, fine.w) })) {
  let worse = 0, better = 0
  for (const r of R) { const h = rankOf(f(r), r.gold) || 999, b = single(r)
    if (h > b) worse++; else if (h < b) better++ }
  console.log(`  ${name.padEnd(18)} worse than the better single method on ${worse}/60, better on ${better}/60`)
}

// ---------------------------------------------------------------- latency
console.log('\n=== latency per query, median of 60 queries x 5 runs ===')
const med = (a) => a.sort((x, y) => x - y)[Math.floor(a.length / 2)]
const time = async (fn) => { const out = []
  for (let rep = 0; rep < 5; rep++) for (const q of queries) { const t = process.hrtime.bigint(); await fn(q); out.push(Number(process.hrtime.bigint() - t) / 1e6) }
  return med(out) }
const tb = await time(async (q) => bm25(q.q))
const te = await time(async (q) => embed([q.q]))
const QV = new Map(R.map((r) => [r.q, r.qv]))
const tv = await time(async (q) => topK(dvecs.map((d) => cos(d, QV.get(q.q))), K))
const th = await time(async (q) => { const b = bm25(q.q); const [qv] = await embed([q.q])
  return rrf([b, topK(dvecs.map((d) => cos(d, qv)), K)], 60) })
console.log(`  BM25 (FTS5 MATCH + bm25 rank)   ${tb.toFixed(3)} ms`)
console.log(`  embed the query (MiniLM)        ${te.toFixed(3)} ms`)
console.log(`  cosine scan over ${docs.length} docs      ${tv.toFixed(3)} ms`)
console.log(`  hybrid end to end (RRF)         ${th.toFixed(3)} ms`)

// ---------------------------------------------------------------- examples
console.log('\n=== the cliff at w = 0.50, one query in detail ===')
{
  const cand = R.filter((r) => r.type === 'exact-id')
    .find((r) => rankOf(weighted(r.b, r.v, 0.48), r.gold) === 1 && rankOf(weighted(r.b, r.v, 0.50), r.gold) !== 1)
  if (cand) {
    const B = normalise(cand.b), V = normalise(cand.v)
    for (const w of [0.48, 0.50]) {
      const top = [...new Set([...B.keys(), ...V.keys()])]
        .map((id) => [id, w * (V.get(id) || 0) + (1 - w) * (B.get(id) || 0)])
        .sort((a, b) => b[1] - a[1]).slice(0, 2)
      console.log(`  "${cand.q}" w=${w.toFixed(2)}  ` + top.map(([id, sc]) =>
        `#${id}${id === cand.gold ? '(gold)' : ''} ${sc.toFixed(4)} [bm25 ${(B.get(id) || 0).toFixed(2)} vec ${(V.get(id) || 0).toFixed(2)}]`).join('   '))
    }
  }
}

console.log('\n=== what each method returns when it misses ===')
const title = (id) => docs.find((d) => d.id === id).title
const rk = (l, g) => rankOf(l.map(([, i]) => i), g) || '>50'
for (const t of ['exact-id', 'paraphrase', 'typo-id']) {
  const r = R.filter((x) => x.type === t)
    .find((x) => rankOf(x.b.map(([, i]) => i), x.gold) !== 1 || rankOf(x.v.map(([, i]) => i), x.gold) !== 1)
  if (!r) continue
  console.log(`  [${t}] "${r.q}"   gold = ${title(r.gold)}`)
  console.log(`     BM25 rank ${rk(r.b, r.gold)}, vector rank ${rk(r.v, r.gold)}, RRF k=60 rank ${rankOf(rrf([r.b, r.v], 60), r.gold) || '>50'}`)
  console.log(`     BM25 top 3   : ` + r.b.slice(0, 3).map(([, i]) => title(i)).join(' | '))
  console.log(`     vector top 3 : ` + r.v.slice(0, 3).map(([, i]) => title(i)).join(' | '))
}
db.close()

The probe behind the trigram result, as trigram.mjs:

import Database from 'better-sqlite3'
const db = new Database(':memory:')
db.exec(`create virtual table t using fts5(x, tokenize='trigram')`)
db.prepare('insert into t values (?)').run('ERR-8790: servo loses its network address')
const q = (m) => { try { return db.prepare('select count(*) c from t where t match ?').get(m).c } catch (e) { return 'ERR: ' + e.message } }
for (const m of ['"ERR-8790"', '"8790"', '"790: servo"', '"ERR-879O"', '"ERR-8709"', '"ERR-879"', '"servo"', '"srvo"', '"ER"'])
  console.log(`  match ${m.padEnd(14)} -> ${q(m)}`)

Then do the experiment that matters, which takes one edit. In buildQueries, change the paraphrase line to SYMPTOMS[d.sIdx][0] instead of [1] — the document's own wording — and rerun. Paraphrase recall@1 goes to 100% for both methods and the whole argument for hybrid disappears. That is the test set most teams accidentally build for themselves: queries written in the vocabulary of the documents, by the person who wrote them. It measures nothing, and it always passes.

Where this goes next

The rule that survives the corpus: split your query log by shape before you choose a retriever. If most queries carry an identifier, an SKU or a version, a vector index is an expensive way to get worse answers. If almost none do, BM25 will keep returning fifteen plausible neighbours and no tuning will find the sixteenth. If it is both — what a support box actually receives — run both, union the candidates, and spend the tuning budget on the fusion weight.

None of it matters if the answer was never indexed in a usable piece, which is chunking a document without destroying its meaning. This is the retrieval layer under MedSearch, where the index ships in a SQLite file on the device and queries arrive as both drug names and half-remembered symptoms.