How do you find out an API changed before your users do?

Runtime validation against the documented schema caught 7 of 12 realistic breaking changes; a recorded golden response caught 9; a handful of property assertions caught 9; TypeScript caught 0 — and one change was caught by nothing. Validating every response cost about 20 microseconds on a 4 KB payload, which disappeared entirely once real network latency was in the loop.

Everyone answers this question with "write contract tests". That is advice, not a measurement. Here is the measurement.

The short answer

• No single method catches everything. Golden diffs and property checks each caught 9 of 12; strict schema validation caught 7. They miss different changes, so the union of two cheap methods beats one thorough one.
• TypeScript catches none of them. Interfaces are erased before the code runs. The 225-byte Order interface in this article compiles to 225 bytes of whitespace and zero bytes of runtime checking.
• Doing nothing catches 5 of 12, and only ever as a thrown exception. The other 7 travel silently into storage. A renamed field wrote a row with the amount column absent and rendered an invoice of NaN.
• Validating every response is not expensive. 19,889 ns for a 25-order, 4,293-byte payload. At a realistic 20 ms upstream latency the difference in throughput was inside the noise: 22.94 ms per response unvalidated versus 22.42 ms validated.
• The change nothing catches is a change in meaning. When total_cents started returning dollars as whole numbers, every schema still passed, the golden diff still passed, and the invoice was off by a factor of 100.

What was measured, and on what

Apple M3, macOS (Darwin 25.4.0), Node 23.5.0, everything on loopback. A local HTTP server plays the upstream API and serves an orders endpoint:

{ "data": [ { "id": 1, "status": "shipped", "total_cents": 2999,
              "currency": "USD", "created_at": "2026-09-01T00:00:01.000Z",
              "shipped_at": "2026-09-02T00:00:01.000Z",
              "items": [ { "sku": "SKU-101", "qty": 1 } ] } ],
  "page": 1, "page_size": 25, "has_more": true }

status takes three values. shipped_at is null for two thirds of orders. An optional coupon field appears on about 10% of them — that field turns out to matter a great deal later. A /mutate endpoint switches on one breaking change at a time, and the same client runs against each.

The five detection methods:

1. Nothing — parse, map to a row, insert, sum, render an invoice. Caught means something threw.
2. Runtime validation — every response checked against a schema written from the API's documentation. Strict: unknown keys are an error.
3. Golden response — 20 healthy responses recorded once. A path whose value was constant across all 20 is compared by value; a path whose value varied is compared by type. Re-run on a schedule.
4. Type-only — a TypeScript interface and a cast, nothing else.
5. Property-based sampling — eight assertions about live responses that a human who understands the domain would actually write. total_cents is a non-negative integer, currency matches /^[A-Z]{3}$/, a shipped order has a shipped_at, page_size is what you asked for, and so on.

Which method catches which change?

This is the deliverable. Every cell is the observed result of running that detector against that change.

Four results are worth pausing on.

Validation missed the null. The published docs say currency is string | null, so a schema written from the docs accepts null. Across the 20 recorded healthy responses it was "USD" every time. The golden recording, which learns from what actually arrives rather than what is documented, caught it immediately: $.data[].currency type null never recorded. Documentation describes what the vendor is allowed to send. Traffic describes what they do send, and your code is written against the second one.

The golden diff missed the new enum value. status legitimately varies across three values, so the recorder marked it type-only and a fourth value slipped through as just another string. Value comparison and change tolerance are the same dial.

Validation missed the page-size change, and it is the quiet data-loss bug. The response is still perfectly valid — page_size is still a number. You just silently process 10 rows where you used to get 25, and if your loop trusts the page it asked for, 15 orders a page never get imported.

Only one method caught a change in meaning, and only by accident. When cents became dollars as 29.99, the property Number.isInteger(total_cents) failed. When the same change produced whole dollars — 29 instead of 2999 — every detector passed. A number in the right place with the wrong units is indistinguishable from correct data. Nothing on this list will save you; only a reconciliation against a second source will.

What does a renamed field actually do?

The expensive failure is not the one that throws. It is the one that does not. Tracing a renamed field through a five-stage pipeline with no validation:

--- change: rename ---
  1 fetch + JSON.parse         ok      {"data":[{"id":1,"status":"pending", ...
  2 map to a row               ok      {"id":1,"cur":"USD"}
  3 INSERT (JSON round-trip)   ok      {"id":1,"cur":"USD"}
  4 SUM(cents) for the day     ok      null
  5 render the invoice         ok      "NaN"
  column "cents" survived?     false

Nothing threw. o.total_cents is undefined, JSON.stringify drops undefined keys, so the row reaches storage with the amount column missing — not zero, not null, gone. The daily total is NaN, which serialises as JSON null. The invoice says NaN. Every stage reported success.

The type change is worse, because it produces a plausible-looking number:

--- change: type ---
  4 SUM(cents) for the day     ok      "0689699622554"
  5 render the invoice         ok      "6896996225.54"

0 + "6896" is string concatenation. Three orders became an invoice for $6,896,996,225.54. And the meaning change produced "1.94" where the correct invoice was $194.12 — an error small enough that nobody notices until a reconciliation, months later.

Do TypeScript types catch any of this?

No, and it is worth demonstrating rather than asserting, because "we're type-safe" is the most common reason teams skip runtime checks.

Node 23 can strip TypeScript types and run the file. It can also show you what it hands to the engine:

const { stripTypeScriptTypes } = require('node:module')
const out = stripTypeScriptTypes(readFileSync('typed.ts', 'utf8'))

source lines 1-8 (the Order interface): 225 bytes
same lines after Node strips types    : 225 bytes, "" (all whitespace)
bytes of runtime checking emitted     : 0
the cast line: "const page = await res.json()"

The interface becomes whitespace. as Page disappears without a trace. Running the same typed client against three of the changes:

### upstream change: type
declared: total_cents: number, status: a 3-value union, currency: string
actual  : total_cents = "6896" (typeof string)
o.total_cents * 2     = 13792
runtime errors so far : 0

### upstream change: enum
actual  : status      = "on_hold"
runtime errors so far : 0

### upstream change: nulled
actual  : currency    = null
runtime errors so far : 0

o.total_cents * 2 even returned 13792, because JavaScript coerced the string. The type checker verified your source, which is correct. The source never changed. The API did. This is exactly the gap that makes validating a model's JSON output a runtime job too, and for the same reason.

What does validating every response cost?

People skip runtime validation because they assume it is expensive. Measured against JSON.parse alone, nanoseconds per response:

Validation costs roughly 2.5x a bare parse, which sounds alarming and is the number people half-remember. In absolute terms it is 20 microseconds for a typical page, scaling linearly at about 650–800 ns per record.

Against a loopback server with no latency, that overhead is visible — 8,938 responses/second unvalidated against 6,082 validated, down 32%. That is the worst case anyone will ever measure, and it is not your case. Put 20 ms of upstream latency in front of it, which is optimistic for a third-party API:

upstream latency 20ms, validate=false  218 responses in 5s  22.94 ms each
upstream latency 20ms, validate=true   223 responses in 5s  22.42 ms each

The validating client was marginally faster, which means the difference is inside the run-to-run noise. Twenty microseconds against twenty milliseconds is one part in a thousand. If you are calling an API over a network, response validation is free, and the reason to skip it is not performance.

How many samples before an inferred schema stops crying wolf?

Recording a golden response, or inferring a schema from live traffic, has one real cost: false alarms. It is the reason people abandon these systems.

A naive byte-for-byte golden comparison is useless — ids and timestamps vary, so it fired on 50 of 50 unchanged responses. The constant-or-type recording described above fired on 0 of 50.

Inference from a stream of records is harder, because rare shapes have not happened yet. Building a schema from the first N records and then checking the next 1,000, over 40 trials each:

Below 20 records the schema is worse than nothing: it flags four out of five records. The tail from 30 to 50 is entirely one field — coupon, present on 10.3% of records. Miss it in the sample window and every tenth record afterwards reports unknown field coupon.

The rule that falls out: you need roughly 3/p records, where p is the frequency of the rarest optional field you care about. For a field on 10% of traffic that is 30 records; for one on 1% it is 300. And it stays probabilistic — one trial at N=50 still misfired.

There is a second pathology visible in the raw output. At N=10, the inferrer decided shipped_at was an enum, because only three shipped orders had appeared and three distinct timestamps look exactly like a three-value enum. Low cardinality in a small sample is not the same as low cardinality.

What should you actually run?

Two cheap layers, not one thorough one:

1. Validate every response against a schema derived from traffic, not from the docs. Traffic-derived is what caught the null. Record for at least 3/p records before you switch alarms on.
2. Add five or six property assertions that encode intent — units, ranges, closed enumerations, "this page has the size I asked for". They are what caught the meaning change and the pagination change, and they are the only part of this that has to be written by someone who understands the domain.
3. Treat a validation failure as a retryable-class decision, not a crash. Quarantine the record, alert, keep processing. The reasoning is the same as deciding which errors to retry: a malformed payload will be malformed on every attempt, so it belongs in a dead-letter queue, not a retry loop.
4. Validate what arrives at your webhook endpoint too. An inbound payload is an upstream contract you do not control either — the same schema check belongs on the webhook receiver, before the 200.

And accept that the units bug is not covered. No structural method catches it. A daily reconciliation against a second source does.

Check it yourself

One file, no dependencies, Node 23. It starts the fake upstream in-process, applies all 12 changes, runs four of the five detectors and prints the table above.

// contract-check.mjs — node 23, no dependencies.  node contract-check.mjs
import { createServer } from 'node:http'

const S = ['pending', 'shipped', 'cancelled']
let change = 'none', seq = 0
const order = i => {
  const st = S[i % 3]
  const o = { id: i, status: st, total_cents: 1000 + (i * 37) % 9000, currency: 'USD',
    created_at: new Date(Date.UTC(2026, 8, 1, 0, 0, i % 60)).toISOString(),
    shipped_at: st === 'shipped' ? new Date(Date.UTC(2026, 8, 2)).toISOString() : null,
    items: [{ sku: 'SKU-' + (100 + i % 40), qty: 1 }] }
  if (i % 10 === 0) o.coupon = 'SAVE10'
  return o
}
const C = {
  none: p => p,
  rename: p => { for (const o of p.data) { o.totalCents = o.total_cents; delete o.total_cents } return p },
  remove: p => { for (const o of p.data) delete o.currency; return p },
  add: p => { for (const o of p.data) o.risk_score = 0.12; return p },
  type: p => { for (const o of p.data) o.total_cents = String(o.total_cents); return p },
  nulled: p => { for (const o of p.data) o.currency = null; return p },
  enum: p => { for (const o of p.data) o.status = 'on_hold'; return p },
  array: p => { for (const o of p.data) o.items = [o.items[0], { sku: 'SKU-9', qty: 1 }, { sku: 'SKU-8', qty: 2 }]; return p },
  date: p => { for (const o of p.data) { o.created_at = Date.parse(o.created_at); if (o.shipped_at) o.shipped_at = Date.parse(o.shipped_at) } return p },
  pagesize: p => { p.page_size = 10; p.data = p.data.slice(0, 10); return p },
  errshape: p => p,
  meaning: p => { for (const o of p.data) o.total_cents = o.total_cents / 100; return p },
  meaning_int: p => { for (const o of p.data) o.total_cents = Math.round(o.total_cents / 100); return p }
}
const srv = createServer((q, r) => {
  const u = new URL(q.url, 'http://x')
  if (u.pathname === '/mutate') { change = u.searchParams.get('change'); seq = 0; return r.end('{}') }
  if (u.pathname === '/orders/missing') {
    r.writeHead(404, { 'content-type': 'application/json' })
    return r.end(JSON.stringify(change === 'errshape'
      ? { message: 'no such order', code: 'not_found' }
      : { error: { code: 'not_found', message: 'no such order' } }))
  }
  const data = []; for (let i = 0; i < 25; i++) data.push(order(++seq))
  r.writeHead(200, { 'content-type': 'application/json' })
  r.end(JSON.stringify(C[change]({ data, page: 1, page_size: 25, has_more: true })))
}).listen(8787)

const API = 'http://127.0.0.1:8787'
const get = async p => (await fetch(API + p)).json()
const mut = c => fetch(`${API}/mutate?change=${c}`)

// ---- 1. do nothing -------------------------------------------------------
const L = { pending: 'Pending', shipped: 'Shipped', cancelled: 'Cancelled' }
const nothing = (p, e) => {
  const rows = p.data.map(o => ({ day: o.created_at.slice(0, 10), cents: o.total_cents,
    cur: o.currency, label: L[o.status].toUpperCase() }))
  const rev = rows.reduce((a, x) => a + x.cents, 0)
  new Intl.NumberFormat('en-US', { style: 'currency', currency: rows[0].cur }).format(rev / 100)
  return e.error.code
}
// ---- 2. runtime validation against the documented schema -----------------
const iso = v => typeof v === 'string' && /^\d{4}-\d{2}-\d{2}T/.test(v)
const SCHEMA = { data: { array: { id: 'number', status: S, total_cents: 'number',
  currency: 'string|null', created_at: 'iso', shipped_at: 'iso|null',
  items: { array: { sku: 'string', qty: 'number' } }, 'coupon?': 'string' } },
page: 'number', page_size: 'number', has_more: 'boolean' }
const ERRS = { error: { object: { code: 'string', message: 'string' } } }
function validate (v, spec, path, out) {
  for (const rk of Object.keys(spec)) {
    const opt = rk.endsWith('?'); const k = opt ? rk.slice(0, -1) : rk; const s = spec[rk]; const p = path + '.' + k
    if (!(k in v)) { if (!opt) out.push(`missing ${p}`); continue }
    const x = v[k]
    if (Array.isArray(s)) { if (!s.includes(x)) out.push(`${p} not in enum: ${JSON.stringify(x)}`); continue }
    if (s.array) { Array.isArray(x) ? x.forEach(el => validate(el, s.array, p + '[]', out)) : out.push(`${p} not array`); continue }
    if (s.object) { validate(x, s.object, p, out); continue }
    if (!s.split('|').some(t => t === 'null' ? x === null : t === 'iso' ? iso(x) : typeof x === t))
      out.push(`${p} is ${x === null ? 'null' : typeof x}, want ${s}`)
  }
  for (const k of Object.keys(v)) if (!(k in spec) && !(k + '?' in spec)) out.push(`unexpected key ${path}.${k}`)
  return out
}
// ---- 3. golden response, constant-or-type ---------------------------------
const walk = (v, p, f) => Array.isArray(v)
  ? (f(p + '[]', '#len', v.length), v.forEach(e => walk(e, p + '[]', f)))
  : v && typeof v === 'object' ? Object.keys(v).forEach(k => walk(v[k], p + '.' + k, f))
    : f(p, v === null ? 'null' : typeof v, v)
function record (ss) {
  const m = new Map()
  for (const s of ss) { const here = new Set()
    walk(s, '$', (p, t, v) => { const e = m.get(p) || { types: new Set(), vals: new Set(), seen: 0 }
      e.types.add(t); if (e.vals.size < 8) e.vals.add(JSON.stringify(v)); m.set(p, e); here.add(p) })
    for (const p of here) m.get(p).seen++ }
  for (const e of m.values()) { e.const = e.vals.size === 1; e.optional = e.seen < ss.length }
  return m
}
function diff (g, s) {
  const seen = new Map()
  walk(s, '$', (p, t, v) => { const e = seen.get(p) || { types: new Set(), vals: new Set() }
    e.types.add(t); e.vals.add(JSON.stringify(v)); seen.set(p, e) })
  const out = []
  for (const [p, e] of g) {
    if (!seen.has(p)) { if (!e.optional) out.push(`path gone ${p}`); continue }
    for (const t of seen.get(p).types) if (!e.types.has(t)) out.push(`${p}: new type ${t}`)
    if (e.const) for (const v of seen.get(p).vals) if (!e.vals.has(v)) out.push(`${p}: was ${[...e.vals][0]}, now ${v}`)
  }
  for (const p of seen.keys()) if (!g.has(p)) out.push(`new path ${p}`)
  return out
}
// ---- 5. properties you actually believe ----------------------------------
function props (p, e) {
  const o = []
  for (const x of p.data) {
    if (!Number.isInteger(x.total_cents) || x.total_cents < 0) o.push(`P1 total_cents=${JSON.stringify(x.total_cents)}`)
    if (!S.includes(x.status)) o.push(`P2 status=${JSON.stringify(x.status)}`)
    if (!(typeof x.created_at === 'string' && new Date(x.created_at).toISOString() === x.created_at)) o.push('P3 created_at')
    if (!/^[A-Z]{3}$/.test(x.currency)) o.push(`P4 currency=${JSON.stringify(x.currency)}`)
    if (x.status === 'shipped' && x.shipped_at === null) o.push('P5 shipped without shipped_at')
    if (!x.items?.length) o.push('P6 items')
    if (o.length) break
  }
  if (p.page_size !== 25) o.push(`P7 page_size=${p.page_size}`)
  if (typeof e?.error?.code !== 'string') o.push('P8 error.code')
  return o
}

// ---- run -----------------------------------------------------------------
await mut('none')
const ok = [], er = []
for (let i = 0; i < 20; i++) { ok.push(await get('/orders')); er.push(await get('/orders/missing')) }
const G = record(ok), GE = record(er)

const NAMES = { rename: 'field renamed', remove: 'field removed', add: 'field added',
  type: 'type number->string', nulled: 'non-null field returns null', enum: 'enum gains a value',
  array: 'array 1 -> 3 elements', date: 'date ISO -> epoch', pagesize: 'page size 25 -> 10',
  errshape: 'error shape changed', meaning: 'cents -> dollars (29.99)', meaning_int: 'cents -> dollars (29)' }
const tot = { nothing: 0, validate: 0, golden: 0, types: 0, property: 0 }
console.log('change'.padEnd(30) + 'nothing'.padEnd(14) + 'validate'.padEnd(10) + 'golden'.padEnd(10) + 'types'.padEnd(9) + 'property')
for (const c of Object.keys(NAMES)) {
  await mut(c)
  const p = await get('/orders'); const e = await get('/orders/missing')
  let n = 'missed'; try { nothing(p, e) } catch (x) { n = x.constructor.name; tot.nothing++ }
  const v = validate(p, SCHEMA, '$', []).concat(validate(e, ERRS, '$err', []))
  const g = diff(G, p).concat(diff(GE, e))
  const r = props(p, e)
  if (v.length) tot.validate++; if (g.length) tot.golden++; if (r.length) tot.property++
  const y = a => (a.length ? 'CAUGHT' : 'missed')
  console.log(NAMES[c].padEnd(30) + n.padEnd(14) + y(v).padEnd(10) + y(g).padEnd(10) + 'missed'.padEnd(9) + y(r))
}
console.log(`\ncaught / 12   nothing=${tot.nothing}  validate=${tot.validate}  golden=${tot.golden}  types=${tot.types}  property=${tot.property}`)
srv.close()

node contract-check.mjs

change                        nothing       validate  golden    types    property
field renamed                 missed        CAUGHT    CAUGHT    missed   CAUGHT
field removed                 TypeError     CAUGHT    CAUGHT    missed   CAUGHT
field added                   missed        CAUGHT    CAUGHT    missed   missed
type number->string           missed        CAUGHT    CAUGHT    missed   CAUGHT
non-null field returns null   RangeError    missed    CAUGHT    missed   CAUGHT
enum gains a value            TypeError     CAUGHT    missed    missed   CAUGHT
array 1 -> 3 elements         missed        missed    CAUGHT    missed   missed
date ISO -> epoch             TypeError     CAUGHT    CAUGHT    missed   CAUGHT
page size 25 -> 10            missed        missed    CAUGHT    missed   CAUGHT
error shape changed           TypeError     CAUGHT    CAUGHT    missed   CAUGHT
cents -> dollars (29.99)      missed        missed    missed    missed   CAUGHT
cents -> dollars (29)         missed        missed    missed    missed   missed

caught / 12   nothing=5  validate=7  golden=9  types=0  property=9

Add a change of your own to the C object at the top and re-run. If your schema catches it, you have a detector. If it does not, you have found the change that will reach your database.

Workflow Builder records a schema per connector from live responses and quarantines records that stop matching it, so a vendor renaming a field surfaces as a run that failed loudly rather than a column that quietly filled with nulls.