Self-hosting without the support hell

Make the install diagnose itself. A customer running your software on their own hardware cannot be SSH'd into, so every question you would have answered by looking has to be answered by something you shipped: a preflight that names the missing variable, migrations that run themselves however many releases were skipped, a /health they can curl, and one command that produces a file safe to email you. Plus a support window written down before anyone asks.

Building the image is the afternoon. This is what decides whether self-hosting scales.

The problem, stated exactly

The ticket says "it doesn't work". You do not know which version they run, what their config says, whether their schema migrated, or what the error was. That is three round-trips of email before the diagnosis starts, and each is a day, because they are in another timezone and this is not their job. Every item below removes one round-trip.

Bound the support window in writing

Decide this before a customer asks. After they ask, any answer sounds like it was invented for them.

The last row keeps the other three honest. "We support two minors" is only reasonable if a customer three years behind can reach current in one step. If upgrading from 1.2 means stopping at 1.3 and 1.4 first, you have widened your support surface rather than narrowing it, because now you maintain the intermediate hops too.

Two rules make that promise cheap. Migrations are forward-only and additive, so none needs the next to have run. And the image runs every pending migration on start, so there is no upgrade command a customer can run in the wrong order.

Migrations that a skipped release cannot break

export const MIGRATIONS = [
  { id: 1, name: 'users', sql: `
    CREATE TABLE IF NOT EXISTS users (id bigserial PRIMARY KEY, email text UNIQUE NOT NULL);` },
  { id: 3, name: 'users_timezone', sql: `
    ALTER TABLE users ADD COLUMN IF NOT EXISTS timezone text NOT NULL DEFAULT 'UTC';` },
];
const LEDGER = `CREATE TABLE IF NOT EXISTS schema_migrations (id int PRIMARY KEY,
  name text NOT NULL, applied_at timestamptz NOT NULL DEFAULT now())`;
const LOCK = 0x5c570001;   // any constant; it just has to be ours

export async function migrate(url, log = console.log) {
  const c = new pg.Client({ connectionString: url });
  await c.connect();
  try {
    // One container per customer is the happy path. Two is the case that
    // corrupts a schema, so hold a lock the whole way through.
    await c.query('SELECT pg_advisory_lock($1)', [LOCK]);
    await c.query(LEDGER);
    const q = await c.query('SELECT id FROM schema_migrations');
    const done = new Set(q.rows.map((r) => r.id));
    const at = Math.max(0, ...done);
    const known = Math.max(0, ...MIGRATIONS.map((m) => m.id));

    // A rollback to an older image lands here. Refuse: the old code does not
    // know the columns the new code added, and will write rows the current
    // schema rejects.
    if (at > known) throw new Error(`this database is at schema version ${at}, but ` +
      `this image only knows about ${known}. Use the image you upgraded to, or ` +
      `restore a backup taken before the upgrade.`);

    const pending = MIGRATIONS.filter((m) => !done.has(m.id));
    if (!pending.length) return log(`Schema up to date at migration ${at}.`);
    log(`Applying ${pending.length}: ${pending.map((m) => m.id).join(', ')}`);

    for (const m of pending) {
      // Each migration is its own transaction: a failure at 5 leaves 1-4
      // recorded, so the retry resumes rather than starting over.
      await c.query('BEGIN');
      try {
        await c.query(m.sql);
        await c.query('INSERT INTO schema_migrations (id,name) VALUES ($1,$2)', [m.id, m.name]);
        await c.query('COMMIT');
      } catch (e) {
        await c.query('ROLLBACK');
        throw new Error(`migration ${m.id} (${m.name}) failed: ${e.message}`);
      }
      log(`  ${m.id}  ${m.name}`);
    }
  } finally {
    await c.query('SELECT pg_advisory_unlock($1)', [LOCK]).catch(() => {});
    await c.end();
  }
}

IF NOT EXISTS on every statement is belt and braces — schema_migrations already prevents a re-run — but it turns a half-applied migration from an outage into a retry. The advisory lock matters the first time a customer scales to two replicas without telling you.

Against Postgres 16.15, a customer on 1.2.0 pulling 1.6.0:

== customer installs v1.2.0 (migrations 1-2) ==
Applying 2: 1, 2
== they skip v1.3 and v1.4, pull v1.6.0 ==
Applying 4: 3, 4, 5, 6
== the container restarts ==
Schema up to date at migration 6.
== somebody else installs v1.6.0 today, from empty ==
Applying 6: 1, 2, 3, 4, 5, 6
== they roll back to the v1.4.0 image ==
Cannot start: this database is at schema version 6, but this image only knows
about 4. Use the image you upgraded to, or restore a backup taken before the
upgrade.
refused, exit 78

And the claim that actually matters — that the upgraded database is not a slightly-different second dialect you now have to support. pg_dump --schema-only of both databases, diffed:

== is the upgraded schema the same as the fresh one? ==
IDENTICAL

Filter out the \restrict lines first: pg_dump 16.15 emits a fresh random nonce in each one, so two identical schemas differ there and nowhere else.

Preflight: say what is wrong, not where it threw

The default failure for a missing environment variable is a stack trace out of a database driver, naming a file in your source tree the customer does not have.

const RULES = [
  { name: 'DATABASE_URL', required: true,
    hint: 'postgres://user:password@host:5432/dbname',
    check: (v) => (/^postgres(ql)?:\/\/.+@.+\/.+/.test(v) ? null
      : 'does not look like a Postgres connection URI') },
  { name: 'APP_SECRET', required: true,
    hint: 'any 32+ random characters — openssl rand -hex 32',
    check: (v) => (v.length >= 32 ? null : `is ${v.length} characters, needs at least 32`) },
];

Run it before anything opens a socket, collect all the problems, and exit 78 — EX_CONFIG from sysexits.h, which tells any log aggregator "you configured this wrong" rather than "we crashed".

$ node preflight.js

Cannot start: 2 configuration problems.

  * APP_SECRET is 7 characters, needs at least 32. Expected: any 32+ random characters — openssl rand -hex 32
  * DATABASE_URL: cannot reach the database at postgres://postgres:***@127.0.0.1:5999/postgres (connection refused). Is it running?

See https://docs.example.com/self-hosting/config
exit: 78

Collecting all of them is the whole point: one-problem-per-restart is how a five-minute setup becomes an afternoon. Translate driver error codes while you are there. The customer does not know what 28P01 means:

if (e.code === 'ECONNREFUSED') return `cannot reach the database at ${redactUrl(url)} (connection refused). Is it running?`;
if (e.code === '28P01')        return `the database rejected the password in DATABASE_URL.`;
if (e.code === '3D000')        return `the database named in DATABASE_URL does not exist.`;

Note redactUrl. Preflight output goes straight into support emails and public issue trackers.

A /health they can curl before emailing you

$ curl -s localhost:8080/health
{
  "status": "ok",
  "version": "1.6.0",
  "schema_version": { "expected": 6, "actual": 6 },
  "telemetry": "off",
  "checks": { "database": { "ok": true, "ms": 12, "schema_version": 6 } }
}

With the database stopped, the same endpoint returns 503, "status": "degraded", "actual": null, and "error": "connect ECONNREFUSED ...".

expected versus actual ends the most arguments: one line tells you whether the customer is running a new image against an un-migrated database. The ms is there so "it's slow" arrives with a number.

This is the readiness half of a pair. The liveness half, the timeouts on that database check, and why a failing dependency must never trigger a restart are in health checks that tell you something.

One command, one file, no secrets

docker compose exec app node support-bundle.js

One .tar.gz holding app and Node versions, the host platform, the Postgres server version, database size, the applied-migration list, the redacted config, and the last twenty errors from an in-process ring buffer.

The redaction is where this goes wrong. Here is a bundle built with sensible key-name redaction — DATABASE_URL and APP_SECRET both handled properly:

  "config": {
    "DATABASE_URL": "postgres://postgres:[REDACTED]@127.0.0.1:55536/skipper",
    "APP_SECRET": "[REDACTED 45 chars, sha256:1d441d97e09c]"
  },
  "recent_errors": [

Two lines further down, in the same file:

"message": "connect ETIMEDOUT — while connecting with postgres://postgres:demo@127.0.0.1:55536/skipper",
"message": "signature mismatch (expected key s3cr3t-please-do-not-leak-me-0123456789abcdef)",

Both secrets, in clear, in the file the customer is about to email you. Key-name redaction cannot work: the leak is never in the config section, it is in the error messages, stack traces and logged SQL that copied the value somewhere with a harmless-looking name. Collect the literal values instead, and scrub the whole document:

function secretValues(env) {
  const out = new Set();
  for (const [k, v] of Object.entries(env)) {
    if (!v || v.length < 8) continue;
    if (!/(SECRET|TOKEN|PASSWORD|KEY|DSN|URL|CREDENTIAL)/i.test(k)) continue;
    out.add(v);
    try { const u = new URL(v); if (u.password) out.add(u.password); } catch {}
  }
  return [...out].sort((a, b) => b.length - a.length);   // longest first
}

const scrub = (text, values) =>
  values.reduce((s, v) => s.split(v).join('[REDACTED]'), String(text));

const json = scrub(JSON.stringify(report, null, 2), secretValues(env));

The URL is decomposed so the password is its own needle — otherwise a log line containing only demo still leaks. Longest first, so a secret containing another as a substring is replaced whole. The same file, scrubbed:

"message": "connect ETIMEDOUT — while connecting with [REDACTED]",
"message": "signature mismatch (expected key [REDACTED])",

Ship that as a test. It is the one thing in the bundle that has to be right every time, and it is cheap to assert: build a bundle in CI with a known password in an error message, then grep the tarball for it.

Telemetry: off, and it says what it would send

Customers choose self-hosting because data cannot leave their network. Default-on telemetry is not a growth tactic for that audience; it is the reason procurement rejects you.

if (config.TELEMETRY === 'on') {
  const payload = { install_id, version, schema_version,
                    node: process.version, platform: process.platform };
  console.log(`Telemetry ON. Every 24h this instance will POST exactly this to ` +
              `${process.env.TELEMETRY_URL}:\n  ${JSON.stringify(payload)}`);
  fetch(process.env.TELEMETRY_URL, { method: 'POST', body: JSON.stringify(payload) })
    .catch((e) => console.error(`Telemetry send failed (ignored): ${e.message}`));
} else {
  console.log('Telemetry OFF. This instance makes no outbound connections.');
}

Started with TELEMETRY unset, against a local collector, the collector logged nothing at all. With TELEMETRY=on:

Telemetry ON. Every 24h this instance will POST exactly this to http://127.0.0.1:55538/v1/ping:
  {"install_id":"acme-prod","version":"1.6.0","schema_version":6,"node":"v23.5.0","platform":"darwin"}
[collector] request #1 POST /v1/ping body={"install_id":"acme-prod",...}

Printing the exact payload at boot beats a paragraph in the docs: it is auditable by the person who signs off on it, and it survives the version where somebody adds a field. Note the .catch(), with nothing downstream of it — telemetry that can delay or fail a start is a dependency you have added to every customer's uptime in exchange for a metric.

Check it yourself

The migration claim is the one holding the support window up, so verify that one. Postgres 16, Node 23, port 55539. migrate.mjs is the migrate() above with the list filtered to id <= argv[3], so the second argument stands in for "which version of the image is this". Run it under bash — the last line needs process substitution.

docker run -d --name selfhost-demo -e POSTGRES_PASSWORD=demo -p 55539:5432 postgres:16
until docker exec selfhost-demo pg_isready -U postgres; do sleep 1; done
docker exec selfhost-demo psql -U postgres -c 'CREATE DATABASE skipper' -c 'CREATE DATABASE fresh'
npm install pg

S=postgres://postgres:demo@127.0.0.1:55539/skipper
F=postgres://postgres:demo@127.0.0.1:55539/fresh

node migrate.mjs "$S" 2     # customer installs v1.2.0
node migrate.mjs "$S" 6     # they skip two releases
node migrate.mjs "$S" 6     # the container restarts: no-op
node migrate.mjs "$F" 6     # a fresh install today
node migrate.mjs "$S" 4     # a rollback: refused, exit 78

dump() { docker exec selfhost-demo pg_dump -U postgres -d "$1" --schema-only --no-owner \
           | grep -vE '^--|^\\(un)?restrict|^$'; }
diff -u <(dump skipper) <(dump fresh) && echo IDENTICAL

docker rm -f selfhost-demo

If diff ever prints something, that is your support window breaking: a customer who upgraded now has a schema no fresh install has, and you are debugging a database shape you have never seen.

Where this goes next

This sits behind the self-hosted builds of Simple CRM and the Workflow Builder, and starts with shipping the product as a Docker image rather than an account.

The two things a support bundle cannot rescue are the customer's data and their TLS: put the database on a path they chose, tell them to restore a backup once before they need to, and ship one Caddyfile.