Run history is a product feature, not logging

Store every step of every run — its input, its output, its error and its duration — as rows your user is allowed to read. Not log lines. Rows, with a schema, queried by the same app that ran the flow.

The test is simple: when a customer says "it didn't work this morning", can they find out why without you? If the answer is no, you have logging. If the answer is yes, you have run history, and a whole class of support ticket stops being a ticket.

The ticket you cannot answer

A customer has a five-step flow: webhook in, look up the customer, create the invoice, send the email, write back to the CRM. At 09:14 it stopped working. They open a ticket that says "invoices aren't going out".

You have logs. You go and find them. And then the trouble starts, because you need to know which of the thousands of runs was theirs, which step failed, and what that step was actually holding when it failed. Your log line says:

09:14:22 ERROR invoice step failed: request failed with status code 422

That tells you the step and the status. It does not tell you whose run it was, what was in the body, or whether the two hundred runs after it failed for the same reason. So you go back to the customer and ask for the time, the record id, and a screenshot — which is the moment the ticket becomes a two-day thread.

Why your logs cannot answer it

The instinct is to fix this by logging more: add the payload to the log line, add the flow id, add the customer id. That fails for four separate reasons, and they compound.

The access control line is the one that kills it. You cannot show a customer a log stream that contains other customers' payloads, and you cannot build per-tenant filtering into a text stream you did not design. Run history is tenant-scoped because it is a table with a foreign key, like everything else in your product.

The second reason is that logs are lossy on purpose. Sampled, rate-limited, dropped under load — exactly the conditions where a run is most likely to fail. The third is retention: log retention is one number for the whole system, and "keep the payloads a user might ask about for 30 days" is not the same policy as "keep the HTTP access log for 7 days".

The fourth is subtler. A log line is written by whichever code was executing. A run history row is written by the runner, in a known shape, whether the step succeeded, failed, retried or timed out. It exists for successful runs too — which matters, because half the questions are not "why did it fail" but "did it run at all, and what did it send?"

The schema

Two tables. Runs, and the steps inside them.

CREATE TABLE runs (
  id           BIGSERIAL   PRIMARY KEY,
  tenant_id    BIGINT      NOT NULL,
  flow_id      TEXT        NOT NULL,
  flow_version INT         NOT NULL,      -- which version this run executed
  status       TEXT        NOT NULL,      -- running | success | error | cancelled
  trigger_kind TEXT        NOT NULL,      -- webhook | schedule | manual
  started_at   TIMESTAMPTZ NOT NULL,
  finished_at  TIMESTAMPTZ
);

CREATE TABLE run_steps (
  run_id       BIGINT      NOT NULL REFERENCES runs(id) ON DELETE CASCADE,
  seq          INT         NOT NULL,      -- execution order, not node order
  node_id      TEXT        NOT NULL,
  node_kind    TEXT        NOT NULL,      -- http.request | branch | delay
  status       TEXT        NOT NULL,
  attempt      INT         NOT NULL DEFAULT 1,
  input        JSONB,
  output       JSONB,
  error        JSONB,
  input_bytes  INT,                       -- size before redaction and trimming
  output_bytes INT,
  duration_ms  INT,
  trimmed_at   TIMESTAMPTZ,               -- payloads dropped, row kept
  PRIMARY KEY (run_id, seq)
);

CREATE INDEX runs_recent ON runs (tenant_id, flow_id, started_at DESC);
CREATE INDEX runs_failed ON runs (tenant_id, started_at DESC) WHERE status = 'error';

Five of those decisions are doing more work than they look like they are.

flow_version. A run belongs to the version of the flow that was live when it started, not the one on the canvas now. Without it, a user edits a flow and last week's history becomes unreadable — the step that failed no longer exists. This is the same argument as pinning a run to a flow version in the data model itself; the history is where you notice if you got it wrong.

seq, not node order. A loop visits the same node several times. A branch skips nodes entirely. The primary key is (run_id, seq) because execution order is the only order the user can follow.

attempt. Retries are normal, and a user seeing "attempt 3 of 5, waiting 40 seconds" understands the system far better than a user seeing nothing until it gives up. If you are retrying with backoff — and you should be, with jitter — then the attempt number is part of the story.

error as JSONB, not text. {"kind":"http","status":422,"message":"..."} can be grouped, counted and filtered. "request failed with status 422" can be grepped. You want the first one, because the useful support query is "how many runs failed with the same error in the last hour", and that is a GROUP BY.

input_bytes / output_bytes. Recorded before anything is trimmed. Once a payload is gone, the size is the only thing that still answers "was the response empty?"

Write the row when the step starts and update it when it ends — a step that never finished should leave a row with status = 'running' and no finished_at. That row is how you find runs killed by a deploy, and it is invisible if you only write history on completion.

What you must not write down

Run history is a copy of production traffic sitting in a table your support team can read. Redact at the point of writing, never at the point of display — a filter in the UI still leaves the secret in the database, in your backups, and in the next database dump someone takes to a laptop.

// save as redact.mjs — run with: node redact.mjs
const SECRET_KEY = /^(authorization|cookie|set-cookie|x-api-key|api[-_]?key|secret|password|token|access[-_]?token|refresh[-_]?token|client[-_]?secret|private[-_]?key|card[-_]?number|cvv|ssn)$/i

const MAX_STRING = 2_000     // characters kept per string
const MAX_ARRAY  = 50        // elements kept per array
const MAX_BYTES  = 32_000    // budget for the whole value

export function redact(value) {
  const rawBytes = JSON.stringify(value ?? null).length
  const out = walk(value, 0)
  const json = JSON.stringify(out)
  if (json.length <= MAX_BYTES) {
    return { value: out, truncated: false, rawBytes, storedBytes: json.length }
  }
  return {
    value: { _truncated: true, _raw_bytes: rawBytes, _preview: json.slice(0, 500) },
    truncated: true, rawBytes, storedBytes: 560,
  }
}

function walk(v, depth) {
  if (depth > 12) return '[depth limit]'
  if (v === null || typeof v !== 'object') return clamp(v)
  if (Array.isArray(v)) {
    const kept = v.slice(0, MAX_ARRAY).map((x) => walk(x, depth + 1))
    if (v.length > MAX_ARRAY) kept.push(`[+${v.length - MAX_ARRAY} more]`)
    return kept
  }
  const o = {}
  for (const [k, val] of Object.entries(v)) {
    o[k] = SECRET_KEY.test(k) ? mask(val) : walk(val, depth + 1)
  }
  return o
}

const clamp = (v) =>
  typeof v === 'string' && v.length > MAX_STRING
    ? v.slice(0, MAX_STRING) + `…[+${v.length - MAX_STRING} chars]`
    : v

function mask(v) {
  if (typeof v !== 'string') return '[redacted]'
  return `[redacted:${v.length}${v.length >= 4 ? ':…' + v.slice(-4) : ''}]`
}

Run it against a realistic step input — an HTTP request with a bearer token, a card number, a 5,000-character note and 120 line items:

raw 8313 stored 3566 truncated false
{"content-type":"application/json","authorization":"[redacted:35:…6c0e]","x-api-key":"[redacted:9:…8b9d]"}
"[redacted:16:…4242]" {"id":"cus_9812","email":"ada@example.com"}
note ends: "xxxxxxxxxxxxxxxx…[+3000 chars]"
line_items last: "[+70 more]"
wide: raw 45781 stored 560 truncated true

Four decisions worth stealing:

• Mask by key name, not by value shape. Detecting "things that look like a token" produces both false positives and a false sense of safety. A deny list of header and field names is boring and it works.
• Keep the length and last four characters. [redacted:35:…6c0e] lets a user confirm they pasted the right key without exposing it. A bare [redacted] turns "wrong API key" into another support ticket.
• Clamp arrays before you clamp bytes. The 8,313-byte input above came down to 3,566 mostly by keeping 50 of 120 line items. The whole-value budget is a backstop for the pathological case — the wide object with 3,000 keys, which is the only one of these that actually hit MAX_BYTES and got replaced by a preview.
• Redaction is not anonymisation. ada@example.com is still in there, because a run history with the email removed cannot answer "did it send to the right address?". That is a deliberate choice, and it means run history is personal data: it belongs in your privacy policy, your deletion flow and your data-processing agreement.

The customer's own secrets — the API keys they configured on a step — should never reach the runner as plain values in the first place. Store a reference, resolve it inside the HTTP client, and the redactor becomes a second line of defence rather than the only one.

Retention: trim the payloads, keep the row

Run history grows with usage, which is exactly when you cannot afford to delete it. The way out is that the two questions have different lifetimes:

• "Did it run, and did it work?" — cheap, and interesting for a year.
• "What exactly did it send?" — expensive, and interesting for about a week.

So trim in stages. Payloads go first; the row stays.

UPDATE run_steps s
   SET input = NULL, output = NULL, trimmed_at = now()
  FROM runs r
 WHERE r.id = s.run_id
   AND r.started_at < now() - interval '30 days'
   AND s.trimmed_at IS NULL;

Note what survives: status, duration_ms, attempt, the byte counts and error. Errors are the smallest column and the most valuable, so they are the last thing you should drop. A three-month-old failed run that still says 422 customer has no payment method is worth keeping; the 400 KB response body that came with it is not.

Then, later, delete the runs themselves. Both stages are one statement each, which makes them look free — and that is where the measurement below stops agreeing with the intuition.

Check it yourself

Two thousand runs, four steps each, and a trim. Ports and container names here are arbitrary; change them if they clash.

docker run --rm -d --name integ-runs -e POSTGRES_PASSWORD=demo -p 55511:5432 postgres:16
until docker exec integ-runs pg_isready -U postgres >/dev/null 2>&1; do sleep 1; done
DB=postgresql://postgres:demo@localhost:55511/postgres

psql $DB -q <<'SQL'
CREATE TABLE runs (
  id BIGSERIAL PRIMARY KEY, flow_id TEXT NOT NULL, flow_version INT NOT NULL,
  status TEXT NOT NULL, started_at TIMESTAMPTZ NOT NULL, finished_at TIMESTAMPTZ);
CREATE TABLE run_steps (
  run_id BIGINT NOT NULL REFERENCES runs(id) ON DELETE CASCADE, seq INT NOT NULL,
  node_id TEXT NOT NULL, status TEXT NOT NULL,
  input JSONB, output JSONB, error JSONB,
  input_bytes INT, output_bytes INT, duration_ms INT, trimmed_at TIMESTAMPTZ,
  PRIMARY KEY (run_id, seq));

-- 2,000 runs, four steps each; every twentieth run fails at step 3
INSERT INTO runs (flow_id, flow_version, status, started_at, finished_at)
SELECT 'flow_invoice', 3,
       CASE WHEN g % 20 = 0 THEN 'error' ELSE 'success' END,
       now() - (g || ' minutes')::interval,
       now() - (g || ' minutes')::interval + interval '900 ms'
FROM generate_series(1, 2000) g;

INSERT INTO run_steps (run_id, seq, node_id, status, input, output, error,
                       input_bytes, output_bytes, duration_ms)
SELECT r.id, s.seq, 'n' || s.seq,
       CASE WHEN r.status = 'error' AND s.seq = 3 THEN 'error' ELSE 'success' END,
       jsonb_build_object('url', 'https://api.acme.test/v1/invoices',
         'body', jsonb_build_object('customer', 'cus_' || r.id,
                                    'amount', 4200 + r.id, 'note', repeat('x', 400))),
       jsonb_build_object('status', 200,
         'body', jsonb_build_object('id', 'inv_' || r.id, 'lines', repeat('y', 400))),
       CASE WHEN r.status = 'error' AND s.seq = 3
            THEN '{"kind":"http","status":422,"message":"customer has no payment method"}'::jsonb
       END,
       520, 480, 120
FROM runs r, generate_series(1, 4) AS s(seq);
SQL

# the support question, answered without opening a log file
psql $DB -c "
SELECT s.run_id, s.seq, s.node_id, s.error->>'status' AS http, s.error->>'message' AS message
  FROM run_steps s JOIN runs r ON r.id = s.run_id
 WHERE r.status = 'error' AND s.status = 'error'
 ORDER BY s.run_id DESC LIMIT 3;"

psql $DB -t -c "SELECT 'seeded             ' || pg_size_pretty(pg_total_relation_size('run_steps'));"

# trim payloads past the retention window; keep the row, keep the error
psql $DB -q -c "
UPDATE run_steps s SET input = NULL, output = NULL, trimmed_at = now()
  FROM runs r
 WHERE r.id = s.run_id
   AND r.started_at < now() - interval '12 hours'
   AND s.trimmed_at IS NULL;"

psql $DB -t -c "SELECT 'after UPDATE       ' || pg_size_pretty(pg_total_relation_size('run_steps'));"
psql $DB -q -c "VACUUM FULL run_steps;"
psql $DB -t -c "SELECT 'after VACUUM FULL  ' || pg_size_pretty(pg_total_relation_size('run_steps'));"

psql $DB -c "
SELECT count(*) FILTER (WHERE trimmed_at IS NULL)     AS payloads_kept,
       count(*) FILTER (WHERE trimmed_at IS NOT NULL) AS metadata_only,
       count(*) FILTER (WHERE error IS NOT NULL)      AS errors_kept
  FROM run_steps;"

docker rm -f integ-runs

Output, on Postgres 16.15:

 run_id | seq | node_id | http |            message
--------+-----+---------+------+--------------------------------
   2000 |   3 | n3      | 422  | customer has no payment method
   1980 |   3 | n3      | 422  | customer has no payment method
   1960 |   3 | n3      | 422  | customer has no payment method

 seeded             9560 kB
 after UPDATE       9600 kB
 after VACUUM FULL  3920 kB

 payloads_kept | metadata_only | errors_kept
---------------+---------------+-------------
          2876 |          5124 |         100

The trim made the table bigger. 9560 kB before, 9600 kB after nulling the payloads on 5,124 of 8,000 steps. That is Postgres doing what Postgres does: an UPDATE writes a new row version and leaves the old one behind, so "deleting" data costs space until something reclaims it. Only after VACUUM FULL does the table drop to 3,920 kB — 59% smaller.

That is worth stating plainly, because the obvious reading of the trim query — "we removed the payloads, so the table is smaller now" — is wrong, and it is wrong in the direction that makes you think the retention job is working.

VACUUM FULL takes an ACCESS EXCLUSIVE lock, so it is not something to run on a live history table. Two ways out, both better than trimming in place:

• Partition by month and DROP TABLE the old partition. Dropping a partition is a file unlink — instant, no lock on the rest, space returned to the operating system.
• Accept the reuse. Plain VACUUM — which autovacuum runs for you — only hands pages back to the operating system when the empty ones sit at the end of the file. Deleting the 5,124 trimmed rows above and running VACUUM run_steps moved the table from 3,920 kB to 3,952 kB: nothing returned, and slightly larger for the churn. What you do get is space marked reusable, so the table stops growing rather than shrinking. On a table that is always gaining rows, that is usually the right outcome, and the only cost is a disk-usage graph that never goes down.

Whichever you pick, the trim query itself is the same. What changes is who reclaims the space, and when.

Where this goes next

Run history is also the thing that makes replay possible. If you stored the input of step 3, you can re-run the flow from step 3 with the same input — and "retry from here" is the single feature that turns a failed run from a support ticket into a button. That needs the input to be the resolved input, after templates were rendered, which is the version worth storing anyway.

The runner that fills these tables is a worker pulling jobs from a queue — a Postgres table and SKIP LOCKED is enough — and every step it writes is one attempt of an at-least-once delivery. This is what Workflow Builder shows on every run: the steps, their inputs, their outputs, and the error, in the account that owns them.