AI & Data
A typical regex redactor fully removed 296 of 900 labelled identifiers on a synthetic corpus — recall 0.3289. It caught 100% of US phone numbers written (555) 019-2837 and 0% of the same number written +84 904 879 900. Names it cannot catch at all. And even at recall 1.0, 100% of the records were st
AI & Data
Because something in the chain stored the text in a charset that has no byte for ầ. Measured on MySQL 8.4.11, a latin1 column turns Trần Thị Hương into Tr?n Th? H??ng — 14 bytes where UTF-8 needs 20, and the missing bytes are gone for good. The à mojibake is the recoverable one; question marks are n
AI & Data
Keyword search and vector search each fail almost completely on one predictable class of query, and the aggregate score hides it. Measured on 60 labelled queries: BM25 put the right document first for 100% of exact-identifier searches and 13.3% of paraphrases; the embedding model managed 40.0% and 7
AI & Data
Because the tokenizer was trained mostly on English. The same support notice is 163 tokens in English and 237 in Vietnamese — 1.45x for identical meaning — so a Vietnamese user pays 45% more to say the same thing. Measured with o200k_base and cl100k_base across seven languages, plus the accent test
AI & Data
You do not — not with JSON.parse. On Node 23 the largest file fs.readFileSync can hand it is exactly 536,870,887 bytes, and past that you get ERR_STRING_TOO_LONG before the parser is ever called. Raising --max-old-space-size to 12 GB changes nothing, because the limit is on the string, not the heap.
AI & Data
MinHash with 128 permutations plus LSH banding at b=42, r=3. It found 99.47% of the near-duplicate pairs in 100,000 documents in 1.09 seconds with no false positives, against 8.3 seconds for a full pairwise scan of a corpus a tenth the size. Exact hashing found 0.12% of them.
AI & Data
Yes. int8 scalar quantisation makes a vector set 3.96x smaller and returned 0.984 of the float32 top-10 at 384 dimensions — 1.000 once the top 100 candidates were re-scored with float32. What it does not do is make the scan faster. Measured storage, scan speed and rank agreement at 384 and 1,536 dim
AI & Data
Brute-force cosine similarity over vectors in SQLite stays under 100 ms to roughly 300,000 rows at 384 dimensions — if the dot product is not running in a Python loop, where the same wall arrives seventeen times earlier. Measured latency, storage and what an HNSW index actually buys.
AI & Data
Split on structure first — headings, paragraphs, tables — and fall back to character count only when a single block is genuinely too big. Fixed-size chunking cuts tables in half and hands your retriever a fragment that no embedding model can repair.
AI & Data
SQLite FTS5 gives you ranked, snippet-annotated, as-you-type search with no server and no network. The parts that catch people out are the external-content pattern, the delete trigger, and what the default tokenizer does to accented text.
AI & Data
Prompting for JSON gets you JSON most of the time, which is the problem. Constrain the output to a schema, validate it before you trust it, let the model assign among values you supplied, and send failures to a human rather than swallowing them.
AI & Data
Most mail can be triaged on metadata alone and never reaches a model. What is left gets redacted before it is sent, checked with a residual scan, and stored as a derived fact rather than a message body.