Does your PII redaction actually work?

A typical "good enough" regex redactor fully removed 296 of 900 labelled identifiers on our synthetic test corpus — a recall of 0.3289 — while redacting 122 pieces of text that were not personal data at all. The failures are not evenly spread. Emails and card numbers do well; names are never caught, because no regex can; and a Vietnamese mobile number written +84 904 879 900 was missed 44 times out of 44.

Nobody measures this. The pass gets written in an afternoon, a few examples are eyeballed, and the output is treated as safe to send to a model. This article is the measurement.

The short answer

• A typical regex redactor caught 32.89% of identifiers — 296 of 900, on a labelled synthetic corpus of 400 short clinical and billing messages.
• Phone recall is a property of formatting, not of phone numbers. The classic \d{3}[-.\s]?\d{3}[-.\s]?\d{4} scored recall 1.0000 on (555) 019-2837 and 0.0000 on +84 904 879 900, 0904.879.900 and 090 4879 900.
• Partial redaction is worse than a miss, because it looks like a hit. The email rule matched t.hoang@mail.exam out of t.hoang@mail.example.com, and 0701562303 out of the 12-digit national ID 070156230312.
• A Luhn check took card precision from 0.6000 to 1.0000 at no cost to recall — 50 false positives on order numbers gone, all 75 real cards still caught. Four lines of code, the best return in the whole exercise.
• Redaction is not anonymisation. With every direct identifier removed, 100% of our 400 records were still unique on birth date + postcode + gender, and 79.0% on birth year + postcode + gender.

The corpus is synthetic. Every name, number and address is invented; no real personal data appears here or in the code. Placing the PII ourselves is what makes the ground truth exact — and it means the absolute numbers describe our corpus, not yours. What transfers is the relative pattern: which types collapse, which formats are invisible, which direction each fix moves precision and recall.

Apple M3, macOS 26.4.1, Node v23.5.0, no network and no dependencies. 400 messages, 46,346 characters, 900 labelled identifier spans.

How do you measure a redactor at all?

Build the corpus yourself. Ours composes each message from templates, recording the character offsets of every PII value as the string is assembled: 900 gold spans across eight types.

The definitions matter. A catch fully covers the gold span. A partial overlaps but leaves part of the identifier visible — a miss for recall, tracked separately because it is the failure that passes review. A miss is no overlap at all. A false positive touches no identifier of any type: text destroyed for nothing. Precision is the share of redacted spans that landed on real PII, recall the share of identifiers fully covered.

What does a typical regex miss?

The redactor under test is the set of patterns that actually ships: an email regex off Stack Overflow, the North American phone pattern, an SSN-shaped ID rule, a slash-or-dot date, a 13-to-16-digit card run, an IBAN pattern and a US street-address rule.

Four rows are worth staring at.

Email recall is 0.4067 and every miss is a partial. No email escaped untouched; none was fully removed either. [A-Za-z0-9._-]+@[A-Za-z0-9-]+\.[A-Za-z]{2,4} cannot express plus-addressing, cannot cross a second dot in the domain, and stops at four TLD characters, so t.hoang@mail.example.com comes back as [REDACTED]ple.com — mailbox gone, routing hint intact. Plus-addressing (0.0000 over 32 cases), subdomains (0.0000 over 31) and long TLDs (0.0000 over 26) all fail that way; plain user@example.com scores 1.0000 over 61.

The address rule scored precision 1.0000 and recall 0.0000 at once. Every span sat on a real address and none covered a whole one: 9242 Kestrel Road, Springfield, IL 31885 lost only 9242 Kestrel Road. All 56 Vietnamese addresses were missed outright — Quận 7, Cần Thơ contains no Street, Road or Drive. National IDs fail the same way: the 12-digit Vietnamese CCCD 070156230312 was clipped by the phone rule, which ate the first ten digits and left 12 in place — 41 times out of 75, with no rule in the set that knew what a CCCD was.

Dates are where the false positives live. \d{1,2}[\/.-]\d{1,2}[\/.-]\d{4} matched 50 calendar-versioned build tags (build 6.12.2025 of the portal), dropping DOB precision to 0.4048, while missing every ISO date (27 cases) and every textual one (June 11, 1985, 39 cases). Too eager and too narrow at once: the normal condition of a hand-written date regex.

Names: 175 gold spans, zero caught. There is no rule because there is no rule to write. A name is not a pattern, it is a membership fact.

Do Vietnamese phone numbers get caught?

Mostly they do not, and the split is clean enough to be its own table. Same regex, same corpus, broken down by how the number happened to be typed:

The bare 10-digit Vietnamese form scores 1.0000 by accident: it falls as three, three, four digits, so the North American pattern eats it. Group the same digits the way a Vietnamese person writes them — 3-3-3 after a +84, 4-3-3 with dots, 3-4-3 with spaces — and the pattern has nowhere to anchor.

The +1 row is that bug aimed at a US number: with no optional country code the match starts at the first digit group and leaves the +1 outside. A partial, and exactly what an eyeball review calls a pass.

One template splits a number across a newline, and nothing survives that either. Real ticket text is full of it — numbers wrapped by a mail client, IDs split by a PDF extractor. Our document deduplication work hit the same normalisation problem from the other side.

Does a Luhn check make card redaction safer?

Yes, and it is the cheapest win in the whole exercise. The card rule \b(?:\d[ -]?){13,16}\b catches every real card — recall 1.0000 over 75 — and also catches every 16-digit order number in the corpus, because they are identical in shape. Adding a Luhn checksum test before accepting the match:

Precision 0.6000 to 1.0000, recall unchanged, 50 false positives removed. It costs no recall because Luhn is a property of real card numbers rather than a heuristic about them — every issued card passes — and it buys precision because an arbitrary 16-digit number passes only about one time in ten.

That is the fix worth hunting for: a validator rather than a pattern. Checksums exist for IBANs (mod-97), many national IDs and many tax numbers, and each turns a shape match into an existence claim.

Can a regex find names?

No. But a dictionary can, and the gap between the two is large:

The ASCII row is the trap. [A-Z][a-z]+ matches Sarah Whitlock and fails on Nguyễn Thị Hồng Vân, because ễ is not in [a-z]. The 0.6171 of recall it loses is not a random sample — it is every name carrying a diacritic, which in a Vietnamese medical system is most of them. \p{Lu} and \p{L} with the u flag lift recall to 1.0000 at the cost of 25 false positives, since they also black out Marrow Falls and Purchase Order.

The gazetteer's 1.0000 is not a real number: the corpus draws names from a 66-token list and the gazetteer is that list, so it measures a closed vocabulary against itself. Treat it as an upper bound. What transfers is the trade — a name list is precise on the names you know and blind to the rest, so it works where you control the population (a patient roster, a staff directory) and degrades to guesswork on open text. We deliberately did not download an NER model; that is the right answer for open text, and a different article.

What does fixing all of it cost?

We rewrote the pass: locale-aware phones, a real email grammar, ISO and textual dates, a 12-digit ID rule, Luhn on cards, spaced IBANs, Unicode-aware addresses, gazetteer names.

Recall 0.3289 to 0.9644. Overall precision also rose, 0.8214 to 0.9007, but only because the baseline's card rule was so bad that fixing it dominated the average. The honest place to look is the PHONE row: 0.8878 down to 0.8248, 75 false positives, because a permissive digit-run pattern now eats order numbers and a sensor reading of 198.62.2005.

Over-redaction destroys the answer, and it measures too. Counting whitespace tokens that contain no PII and are redacted anyway:

The last row is what people reach for after a leak scares them, and it buys nothing: recall 0.9644, identical to the careful pass, at more than twice the collateral damage. Redact 8.41% of the non-PII tokens and the model no longer knows the order was for four boxes rather than forty. If you already count what a token costs, count the destroyed ones too.

Is redaction enough to anonymise?

No. Assume a perfect redactor: every name, number, email and address gone, recall 1.0000. The message still carries facts. Our corpus attaches a small profile to each record — birth date, postcode, gender, job title — the sort of thing that survives redaction because it is the content. Counting how many of the 400 records are alone in their combination:

Every record is unique on birth date, postcode and gender — three facts a redactor has no reason to touch, because none of them is an identifier. Coarsening the birth date to the year still leaves 79.0% unique. Only at the decade plus a two-character postcode prefix does uniqueness fall to 3.5%, and by then the clinical usefulness of the age is gone.

Rare attributes do it alone. Across 36 job titles, 3 were held by two people or fewer and craniofacial prosthetist appeared once. "The craniofacial prosthetist in district 2 asked about her results" identifies someone with no name, number or address in it.

That is the k-anonymity result, and it is why "we redact PII before sending to the model" is a claim about hygiene, not anonymisation. Anonymisation needs controls of a different kind: generalisation, suppression of rare values, aggregation thresholds, or not sending the record at all.

This is not legal advice. Whether data counts as anonymised under GDPR or de-identified under HIPAA is a legal determination with defined methods — Safe Harbor, expert determination, the identifiability test — and none is satisfied by a recall number. A measured redactor is evidence to put in front of a lawyer or an auditor, not a substitute for one.

So what should you actually do?

1. Build the labelled set first. Two hundred messages with PII you placed yourself takes an afternoon, and turns every later argument into a number.
2. Report recall per type. Our 0.3289 hides a 1.0000 and a 0.0000 sitting next to each other.
3. Count partials separately. They are the failures that pass review.
4. Prefer validators to patterns — Luhn, mod-97, check digits.
5. Include the formats your users actually type. Four Vietnamese phone forms cost us 0.3771 recall across the whole type.
6. Treat over-redaction as a first-class metric. Recall 1.0000 with a third of the text blacked out has moved the problem, not solved it.
7. Then ask whether you needed to send the text at all. The strongest control is still the one in scanning an inbox without leaking it: decide on metadata, keep the body local, send a derived fact instead of a document — how our workflow tooling is built, because in medical software the cheapest redaction failure is the one that never got the chance to happen.

Check it yourself

Save this as redaction-recall.js and run node redaction-recall.js. No dependencies, no network, nothing to install. It builds a smaller corpus — six message templates instead of sixteen — so its numbers differ from the tables above; the pattern does not.

// redaction-recall.js — node 23, no dependencies, no network.
// Builds a small labelled synthetic corpus (every value invented), runs a
// typical regex redactor over it, and prints precision/recall per PII type.
'use strict';
let seed = 20260830;
const rnd = () => { seed = (seed + 0x6D2B79F5) | 0; let t = Math.imul(seed ^ (seed >>> 15), 1 | seed); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; };
const int = (a, b) => a + Math.floor(rnd() * (b - a + 1));
const pick = a => a[int(0, a.length - 1)];
const pad = (n, w) => String(n).padStart(w, '0');

const luhn = s => { const d = s.replace(/\D/g, ''); if (d.length < 13) return false; let x = 0; for (let i = 0; i < d.length; i++) { let v = +d[d.length - 1 - i]; if (i % 2) { v *= 2; if (v > 9) v -= 9; } x += v; } return x % 10 === 0; };
const luhnFix = p => { let x = 0; for (let i = 0; i < 15; i++) { let d = +p[14 - i]; if (i % 2 === 0) { d *= 2; if (d > 9) d -= 9; } x += d; } return p + ((10 - x % 10) % 10); };

// ---- invented values -------------------------------------------------------
const SUR = ['Nguyễn', 'Trần', 'Lê', 'Phạm', 'Hoàng', 'Đặng', 'Vũ', 'Ngô'], GIV = ['Vân', 'Quân', 'Ngọc', 'Tuấn', 'Hạnh', 'Phúc'];
const EN = ['Sarah Whitlock', 'Marcus Marbury', 'Priya Ashcombe', 'Owen Slavin'];
const name = () => rnd() < 0.6 ? `${pick(SUR)} ${pick(GIV)}` : pick(EN);
const email = () => pick(['t.hoang@example.com', 't.hoang+billing@example.com', 'vanlinh@mail.example.com', 'm.quan@example.clinic']);
const phone = () => { const d9 = pick(['90', '91', '96', '78']) + pad(int(0, 9999999), 7);
  return pick(['0' + d9, `+84 ${d9.slice(0, 3)} ${d9.slice(3, 6)} ${d9.slice(6)}`, `0${d9.slice(0, 3)}.${d9.slice(3, 6)}.${d9.slice(6)}`,
    `0${d9.slice(0, 2)} ${d9.slice(2, 6)} ${d9.slice(6)}`, `(${pad(int(200, 989), 3)}) ${pad(int(100, 999), 3)}-${pad(int(0, 9999), 4)}`, `+1 ${pad(int(200, 989), 3)} ${pad(int(100, 999), 3)} ${pad(int(0, 9999), 4)}`]); };
const card = () => { const n = luhnFix((pick(['4539', '5310']) + pad(int(0, 99999999999), 11)).slice(0, 15)); return rnd() < 0.5 ? n.replace(/(\d{4})(?=\d)/g, '$1 ') : n; };
const order = () => { let n; do { n = pad(int(0, 99999999), 8) + pad(int(0, 99999999), 8); } while (luhn(n)); return rnd() < 0.5 ? n.replace(/(\d{4})(?=\d)/g, '$1 ') : n; };
const nid = () => rnd() < 0.6 ? pad(int(1, 99), 3) + pad(int(0, 999999999), 9) : `${pad(int(100, 899), 3)}-${pad(int(1, 99), 2)}-${pad(int(1, 9999), 4)}`;
const MON = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'];
const dobT = p => pick([`${pad(p.d, 2)}/${pad(p.m, 2)}/${p.y}`, `${p.y}-${pad(p.m, 2)}-${pad(p.d, 2)}`, `${pad(p.d, 2)}.${pad(p.m, 2)}.${p.y}`, `${MON[p.m - 1]} ${p.d}, ${p.y}`, `${p.d} ${MON[p.m - 1]} ${p.y}`]);
const addr = () => rnd() < 0.6 ? `${int(1, 400)} ${pick(['Nguyễn Huệ', 'Lê Lợi', 'Pasteur'])}, Quận ${int(1, 12)}, ${pick(['TP. Hồ Chí Minh', 'Hà Nội'])}`
  : `${int(100, 9999)} ${pick(['Maple Street', 'Alder Lane'])}, ${pick(['Springfield, IL', 'Kingsport, TX'])} ${pad(int(10000, 99999), 5)}`;
const iban = () => { const s = pick(['GB', 'NL', 'VN']) + pad(int(0, 99), 2) + pick(['NWBK', 'VCBK']) + pad(int(0, 999999999999999), 14); return rnd() < 0.5 ? s.replace(/(.{4})(?=.)/g, '$1 ') : s; };
const build = () => { const parts = [], spans = []; return { t: s => parts.push(s), p(ty, s) { const b = parts.join('').length; parts.push(s); spans.push({ type: ty, start: b, end: b + s.length, text: s }); }, done: () => ({ text: parts.join(''), spans }) }; };

const T = [
  (b, f) => { b.t('Patient '); b.p('NAME', f.name); b.t(', born '); b.p('DOB', f.dob); b.t(', called. Reach them on '); b.p('PHONE', f.phone); b.t('.'); },
  (b, f) => { b.t('Send the discharge summary to '); b.p('EMAIL', f.email); b.t('. Build '); b.t(f.ver); b.t(' of the portal.'); },
  (b, f) => { b.t('Card on file '); b.p('CARD', f.card); b.t('; order reference '); b.t(f.order); b.t(' is not a card.'); },
  (b, f) => { b.t('ID check: '); b.p('NAME', f.name); b.t(' presented '); b.p('NATIONAL_ID', f.nid); b.t(' at reception, address '); b.p('ADDRESS', f.addr); b.t('.'); },
  (b, f) => { b.t('Refund to '); b.p('IBAN', f.iban); b.t('. Ring '); b.p('PHONE', f.phone); b.t(' or email '); b.p('EMAIL', f.email); b.t('.'); },
  (b, f) => { b.t('Home visit '); b.p('ADDRESS', f.addr); b.t(' for DOB '); b.p('DOB', f.dob); b.t('. Driver mobile\n'); b.p('PHONE', f.phone); b.t('.'); },
];
const N = +(process.env.N || 400), recs = [];
for (let i = 0; i < N; i++) {
  const dp = { d: int(1, 28), m: int(1, 12), y: int(1946, 2006) };
  const f = { name: name(), email: email(), phone: phone(), card: card(), order: order(), nid: nid(), addr: addr(), iban: iban(), dob: dobT(dp), ver: `${int(1, 9)}.${int(1, 24)}.${int(2019, 2026)}` };
  const b = build(); T[i % T.length](b, f); const { text, spans } = b.done();
  recs.push({ text, spans, person: { dob: dp, postcode: pick(['70000', '10000', '55112', '90210', '62704', '43004']), gender: rnd() < 0.5 ? 'F' : 'M' } });
}

// ---- the "good enough" redactor teams actually ship -------------------------
const RULES = [
  ['EMAIL', /[A-Za-z0-9._-]+@[A-Za-z0-9-]+\.[A-Za-z]{2,4}/g],
  ['PHONE', /\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}/g],
  ['NATIONAL_ID', /\b\d{3}-\d{2}-\d{4}\b/g],
  ['DOB', /\b\d{1,2}[\/.-]\d{1,2}[\/.-]\d{4}\b/g],
  ['CARD', /\b(?:\d[ -]?){13,16}\b/g],
  ['IBAN', /\b[A-Z]{2}\d{2}[A-Z0-9]{10,30}\b/g],
  ['ADDRESS', /\b\d{1,5}\s+[A-Z][a-z]+\s+(?:Street|St|Road|Rd|Avenue|Ave|Lane|Ln|Drive|Dr)\b/g],
];
const run = (text, luhnCheck) => { const o = []; for (const [type, re] of RULES) { re.lastIndex = 0; let m; while ((m = re.exec(text))) { if (!m[0].length) { re.lastIndex++; continue; } if (type === 'CARD' && luhnCheck && !luhn(m[0])) continue; o.push({ type, start: m.index, end: m.index + m[0].length, text: m[0] }); } } return o; };
const ov = (a, b) => a.start < b.end && b.start < a.end, cov = (p, g) => p.start <= g.start && p.end >= g.end;

function report(label, luhnCheck) {
  const TY = ['EMAIL', 'PHONE', 'NAME', 'NATIONAL_ID', 'DOB', 'ADDRESS', 'CARD', 'IBAN'], st = {};
  for (const t of TY) st[t] = { gold: 0, full: 0, part: 0, miss: 0, pred: 0, fp: 0 };
  let junk = 0, junkHit = 0;
  for (const r of recs) {
    const preds = run(r.text, luhnCheck);
    for (const g of r.spans) { const s = st[g.type]; s.gold++; const h = preds.filter(p => ov(p, g)); if (h.some(p => cov(p, g))) s.full++; else if (h.length) s.part++; else s.miss++; }
    for (const p of preds) { st[p.type].pred++; if (!r.spans.some(g => ov(p, g))) st[p.type].fp++; }
    const re = /\S+/g; let m; while ((m = re.exec(r.text))) { const tk = { start: m.index, end: m.index + m[0].length }; if (r.spans.some(g => ov(tk, g))) continue; junk++; if (preds.some(p => ov(tk, p))) junkHit++; }
  }
  console.log(`\n${label}\ntype          gold  caught  partial  missed   FP  precision  recall     F1`);
  let G = 0, C = 0, P = 0, F = 0;
  for (const t of TY) { const s = st[t]; G += s.gold; C += s.full; P += s.pred; F += s.fp;
    const pr = s.pred ? (s.pred - s.fp) / s.pred : 1, rc = s.gold ? s.full / s.gold : 0;
    console.log(t.padEnd(13) + String(s.gold).padStart(4) + String(s.full).padStart(8) + String(s.part).padStart(9) + String(s.miss).padStart(8) + String(s.fp).padStart(5) + pr.toFixed(4).padStart(11) + rc.toFixed(4).padStart(8) + ((pr + rc) ? 2 * pr * rc / (pr + rc) : 0).toFixed(4).padStart(7)); }
  const pr = P ? (P - F) / P : 1, rc = C / G;
  console.log('-'.repeat(74));
  console.log('OVERALL'.padEnd(13) + String(G).padStart(4) + String(C).padStart(8) + ' '.repeat(17) + String(F).padStart(5) + pr.toFixed(4).padStart(11) + rc.toFixed(4).padStart(8) + (2 * pr * rc / (pr + rc)).toFixed(4).padStart(7));
  console.log(`over-redaction: ${junkHit}/${junk} non-PII tokens redacted = ${(junkHit / junk * 100).toFixed(2)}%`);
}
console.log(`node ${process.versions.node}  ${N} synthetic messages, ${recs.reduce((a, r) => a + r.spans.length, 0)} labelled identifiers. All values invented.`);
report('=== typical regex redactor ===', false);
report('=== same rules + Luhn check on card matches ===', true);

// ---- re-identification: direct identifiers gone, records still unique ------
console.log('\n=== unique on quasi-identifiers alone ===');
const u = (fn, lab) => { const m = new Map(); for (const r of recs) m.set(fn(r.person), (m.get(fn(r.person)) || 0) + 1);
  let one = 0; for (const r of recs) if (m.get(fn(r.person)) === 1) one++;
  console.log(lab.padEnd(42) + String(m.size).padStart(6) + String(one).padStart(8) + (one / recs.length * 100).toFixed(1).padStart(8) + '%'); };
console.log('quasi-identifier set                      classes  unique  unique%');
u(p => `${p.dob.y}-${p.dob.m}-${p.dob.d}|${p.postcode}|${p.gender}`, 'full DOB + postcode + gender');
u(p => `${p.dob.y}|${p.postcode}|${p.gender}`, 'birth year + postcode + gender');
u(p => `${Math.floor(p.dob.y / 10)}|${p.postcode.slice(0, 2)}|${p.gender}`, 'birth decade + postcode prefix + gender');

On the M3 laptop above that prints:

node 23.5.0  400 synthetic messages, 932 labelled identifiers. All values invented.

=== typical regex redactor ===
type          gold  caught  partial  missed   FP  precision  recall     F1
EMAIL         133      33      100       0    0     1.0000  0.2481 0.3976
PHONE         199      53       33     113   27     0.8811  0.2663 0.4090
NAME          134       0        0     134    0     1.0000  0.0000 0.0000
NATIONAL_ID    67      20       47       0    0     1.0000  0.2985 0.4598
DOB           133      53        0      80   67     0.4417  0.3985 0.4190
ADDRESS       133       0       66      67    0     1.0000  0.0000 0.0000
CARD           67      67        0       0   67     0.6105  1.0000 0.7581
IBAN           66      28       38       0    0     1.0000  0.4242 0.5957
--------------------------------------------------------------------------
OVERALL       932     254                   161     0.7898  0.2725 0.4052
over-redaction: 254/2990 non-PII tokens redacted = 8.49%

=== same rules + Luhn check on card matches ===
type          gold  caught  partial  missed   FP  precision  recall     F1
EMAIL         133      33      100       0    0     1.0000  0.2481 0.3976
PHONE         199      53       33     113   27     0.8811  0.2663 0.4090
NAME          134       0        0     134    0     1.0000  0.0000 0.0000
NATIONAL_ID    67      20       47       0    0     1.0000  0.2985 0.4598
DOB           133      53        0      80   67     0.4417  0.3985 0.4190
ADDRESS       133       0       66      67    0     1.0000  0.0000 0.0000
CARD           67      67        0       0    0     1.0000  1.0000 1.0000
IBAN           66      28        6      32    0     1.0000  0.4242 0.5957
--------------------------------------------------------------------------
OVERALL       932     254                    94     0.8591  0.2725 0.4138
over-redaction: 94/2990 non-PII tokens redacted = 3.14%

=== unique on quasi-identifiers alone ===
quasi-identifier set                      classes  unique  unique%
full DOB + postcode + gender                 400     400   100.0%
birth year + postcode + gender               296     214    53.5%
birth decade + postcode prefix + gender       81       4     1.0%

Change the RULES array to your own patterns and the templates to your own message shapes, and it measures your redactor instead of ours. That is the only version of this exercise that tells you anything about your system.