What happens when a 2 GB upload fails at 90%?

You send the whole thing again. A 256 MiB single POST killed at 90% wasted 242,221,056 bytes and needed 510,656,512 bytes in total to finish — 1.9 copies of the file. The same failure against a chunked endpoint re-sent 419,430 bytes: 577x less, and it completed in 306 ms against 799 ms. A plain multipart POST has exactly one failure mode, and it is start again. On a phone on mobile data that is not an edge case, it is the normal case.

Hardware: Apple M3, 16 GB, macOS 26.4.1, APFS SSD. Node v23.5.0 arm64, built-ins only — no dependencies, no Docker, no network. Everything runs over 127.0.0.1, so wall-clock figures are what a 500–2,500 MB/s link does, not what a phone does. The transferable numbers are the ratios: bytes wasted, bytes re-sent, request counts, round trips. Where time matters I paced the client to 1 MiB/s (8.4 Mbit/s, a plausible mobile uplink) and say so. The file is 256 MiB rather than 2 GB so a sweep runs in seconds; the proportions are the same.

The short answer

• A failed single POST costs everything already sent. Killed at 25 / 50 / 90% of a 256 MiB upload, the client wasted 67,108,864 / 134,217,728 / 242,221,056 bytes and had to send the whole file again.
• Chunking bounds the loss by chunk size, not by how far you got. With 5 MiB chunks the same three failures re-sent 4,194,304 / 3,145,728 / 419,430 bytes — the partial chunk in flight, nothing else.
• Chunking is nearly free when nothing fails. Extra bytes on the wire for 256 MiB: +142,913 (0.053%) at 256 KiB chunks, +7,415 (0.0028%) at 5 MiB — about 140 bytes per chunk. It was not slower either: median 205 ms with 1 MiB chunks against 234 ms for the single POST.
• Chunk size is a hard constraint, not a soft optimum: the chunk must be smaller than the distance the link survives. With a drop every 16 MiB, 256 KiB chunks finished having wasted 1,853,688 bytes (+0.69%), 5 MiB chunks wasted 47,059,174 (+17.53%), and 25 MiB chunks never finished — 250 attempts, 4,270,851,237 bytes sent, zero chunks committed.
• Without a per-chunk checksum, corruption is silent. One flipped byte in chunk 7 produced 52 HTTP 200s, a file of exactly the right size (268,435,456 bytes) and the wrong SHA-256. With the checksum the server rejected it with 422 on the 8th request; hashing cost 95 ms of client CPU for the whole 256 MiB.

What does a failed upload actually cost?

The naive endpoint is the one everybody writes first: one POST, pipe the request into a file, respond when it ends. If the connection dies the partial file is garbage and gets deleted. Here it is against a 256 MiB file, with the client destroying the socket at three points.

The chunked rows re-send only the chunk in flight. At 90% that was 419,430 bytes: the client had got 0.4 MiB into chunk 46 when the socket died, and the server already held chunks 0–45. Every chunked run reassembled to 268,435,456 bytes with SHA-256 64333548783170ab…, byte-identical to the source.

Loopback hides the part that hurts. Paced to 1 MiB/s with a 12 MiB file, a clean upload took 12.2 s and the failures cost this:

The two zeroes are honest and slightly lucky: 25% and 50% of 12 MiB land exactly on a 1 MiB chunk boundary, so nothing was in flight. At 90% the drop landed mid-chunk and cost 838,860 bytes.

Scale that to the title: 2 GiB at 8.4 Mbit/s is 2,048 seconds, and a drop at 90% throws away 30.7 minutes of somebody's tethered connection and puts the progress bar back to zero.

What does chunking cost when nothing fails?

More requests, more headers — that is the whole bill. Inbound socket bytes measured on the server for a 256 MiB upload:

That works out at about 140 bytes of request line and headers per chunk.

The wall-time column contradicted what I expected. I assumed 1,024 requests would visibly cost something; over a keep-alive connection the median for 1 MiB chunks (205 ms, range 174–258 over five runs) beat the single POST (234 ms, range 134–453), which was by far the noisiest row because its timing is dominated by one enormous disk flush. I would not carry that ordering to a real network. I would carry the conclusion: per-request overhead is not a reason to pick a big chunk. 0.05% of bandwidth is not a budget item.

What chunk size should you pick?

Small enough that a chunk fits inside the time your link survives. Same 256 MiB file, and the link now drops at a random point averaging every 16 MiB; the client resumes from the first missing chunk each time.

The 25 MiB row is not a bad score, it is a livelock. Every attempt died before a single 25 MiB chunk completed, so the server committed nothing, so the next attempt started from the same place: 250 attempts, 4 GB of traffic, zero progress. A single POST fails the same way for the same reason — with a drop every 64 MiB it never once delivered the file in 60 attempts, having sent 4,026,531,840 bytes trying.

The rule is asymmetric, and that is what makes it easy. Too small costs a bounded, tiny amount — 0.053% of bandwidth at 256 KiB, and it never gets worse. Too big costs an unbounded amount, up to and including never finishing. So pick the smallest chunk whose request overhead you can tolerate: 1 MiB is a good default, 5 MiB is fine on stable links, and anything above about a quarter of what your worst link survives is a trap. If you must go large, shrink the chunk after every failure.

How does the server know what it already has?

Two designs, and the difference is exactly one round trip.

The client asks. HEAD /status/:id returned in 0.91 ms with x-next-offset: 131072000 and x-received-count: 25, no body. The GET variant took 0.58 ms and returned 105 bytes listing every received index — worth it only if you allow gaps and fill them out of order. Resuming a 50%-failed upload took 1 discovery request + 27 chunk requests = 28.

The server tells. Every chunk response carries x-next-index and x-next-offset, so a client still running knows where to restart: 27 requests, no discovery call.

Server-tells wins while the client process survives — a dropped socket, but not an app killed in the background, the common case on iOS. Then it has nothing cached and degrades to 28 requests. Implement both. One extra round trip is not worth optimising away; a missing status endpoint means an app restart cannot resume at all.

Does the file actually arrive intact?

Not necessarily, and the failure is quiet. I flipped one byte in chunk 7 and uploaded normally: the server returned 200 for all 52 chunks, assembled a file of exactly 268,435,456 bytes, and produced SHA-256 86f276d11dc740f6… against the source's 64333548783170ab…. Right size, wrong file, no error anywhere. Content-Length cannot catch this and neither can your progress bar. With an x-chunk-sha256 header the server verified each chunk and rejected the bad one with 422 on the 8th request, before writing it.

The cost is CPU, not bandwidth: hashing all 256 MiB client-side took 95 ms, and a whole-file SHA-256 on the server at the end took 201.5 ms. On loopback that pushed the upload from 265 ms to 429 ms, which looks alarming and is a localhost artefact — the same 95 ms sits against 34 minutes of transfer on the mobile link above. Do both: per-chunk checksums so a bad chunk is rejected while re-sending it is still cheap, and a whole-file checksum so assembly bugs cannot ship silently.

Can you upload chunks in parallel?

Yes, and it is the largest speedup available — though on loopback these are memory bandwidth, so treat them as an upper bound.

Four is the knee. What breaks is assembly: with the chunk order shuffled and eight in flight, a server writing each chunk at index * chunkSize produced a byte-perfect file, while one appending in arrival order produced a file of exactly the right size and the wrong SHA-256. Concurrency turns "append" from a latent bug into a certainty.

Why does a retried chunk make the file bigger?

Because appending is not idempotent. This is a double-charged payment in another costume, and it has the same shape as idempotency keys: the retry is correct behaviour, the handler is what is wrong.

With an appending server a clean 256 MiB upload assembled correctly. Then I re-sent three chunks the client believed had timed out — an ordinary thing for a mobile client to do, and the right thing per which errors to retry. The file became 284,164,096 bytes, exactly +15,728,640 (3 × 5 MiB), and the hash no longer matched.

The fix is one line, and it is not a lock or a dedupe table: write the chunk at its offset instead of at the end.

fs.writeSync(fd, buf, 0, buf.length, index * chunkSize);   // idempotent
// fs.appendFileSync(path, buf);                           // is not

The same three retries against the positional server left the file at 268,435,456 bytes with the correct hash and 52 of 52 chunk indices recorded. The chunk index is the idempotency key; the offset is where it redeems it.

What breaks when you scale to two containers?

Upload state that lives in a process is not upload state. I uploaded 90% to instance A — 46 chunks committed, 241,591,910 bytes — then sent the resume to instance B, which had never heard of it. B reported 0 chunks received and next index 0, and the client re-sent 268,435,456 bytes in 52 requests, 111% of everything it had already uploaded.

Nothing errored. The client resumed politely, from the beginning, and B assembled a perfectly correct file at twice the cost. Behind a round-robin load balancer that is one request in two. Either put the received-chunk state and the partial file somewhere both instances can see — shared volume, object store, a row in Postgres — or pin the upload id to one instance for its lifetime. Affinity is the cheaper fix and the one that breaks quietly when that instance is redeployed mid-upload.

Check it yourself

Self-contained: it generates a 64 MiB file, starts both endpoints, kills the connection at 25 / 50 / 90%, and prints the bytes each strategy had to re-send. Node 18+, no dependencies, about 30 seconds.

#!/bin/bash
set -e
MB=${1:-64}; PORT=55721
D=$(mktemp -d); cd "$D"; mkdir store
node -e 'const fs=require("fs"),n=+process.argv[1]*1048576,w=fs.createWriteStream("big.bin");
let s=0x9e3779b9>>>0,d=0;const b=Buffer.allocUnsafe(1<<20);
(function go(){while(d<n){for(let i=0;i<b.length;i+=4){s^=s<<13;s>>>=0;s^=s>>>17;s^=s<<5;s>>>=0;b.writeUInt32LE(s,i);}
d+=b.length;if(!w.write(b))return w.once("drain",go);}w.end();})();' "$MB"

cat > srv.cjs <<'EOF'
const http=require('http'),fs=require('fs'),crypto=require('crypto');
const up=new Map();
const st=id=>{if(!up.has(id))up.set(id,{fd:fs.openSync(`store/${id}.bin`,'w+'),chunks:new Set()});return up.get(id)};
const next=s=>{let i=0;while(s.chunks.has(i))i++;return i};
const srv=http.createServer((q,r)=>{
  const u=new URL(q.url,'http://x'),p=u.pathname.split('/').filter(Boolean);
  if(p[0]==='naive'){                                  // one POST, all or nothing
    const f=`store/${p[1]}.naive`,ws=fs.createWriteStream(f);ws.on('error',()=>{});
    q.on('error',()=>{});
    q.on('aborted',()=>{q.unpipe(ws);ws.destroy();try{fs.unlinkSync(f)}catch{}});  // partial upload discarded
    q.pipe(ws);q.on('end',()=>ws.end(()=>{r.writeHead(200);r.end('{}')}));return;
  }
  if(p[0]==='c'){                                      // PUT /c/:id/:index?off=
    const s=st(p[1]),i=+p[2],off=+u.searchParams.get('off'),b=[];
    q.on('data',c=>b.push(c));
    q.on('end',()=>{const buf=Buffer.concat(b);
      fs.writeSync(s.fd,buf,0,buf.length,off);         // positional write: retry-safe
      s.chunks.add(i);r.writeHead(200,{'x-next-index':String(next(s))});r.end('{}')});
    return;
  }
  if(p[0]==='status'){const s=up.get(p[1]);const n=s?next(s):0;
    r.writeHead(200,{'x-next-index':String(n)});return r.end(JSON.stringify({nextIndex:n}))}
  if(p[0]==='sha'){const s=up.get(p[1]);const h=crypto.createHash('sha256');
    const rs=fs.createReadStream(s?`store/${p[1]}.bin`:`store/${p[1]}.naive`);
    rs.on('data',c=>h.update(c));rs.on('error',()=>{r.writeHead(404);r.end('{}')});
    rs.on('end',()=>{r.writeHead(200);r.end(JSON.stringify({sha:h.digest('hex'),size:fs.statSync(`store/${p[1]}.bin`).size}))});return}
  r.writeHead(404);r.end('{}');
});
srv.keepAliveTimeout=300000;srv.listen(55721,'127.0.0.1',()=>console.log('up'));
EOF

cat > run.cjs <<'EOF'
const http=require('http'),fs=require('fs'),crypto=require('crypto');
const F='big.bin',SIZE=fs.statSync(F).size,CS=5*1048576,PORT=55721;
const ag=new http.Agent({keepAlive:true});
const now=()=>Number(process.hrtime.bigint())/1e6;
const src=crypto.createHash('sha256').update(fs.readFileSync(F)).digest('hex');
function naive(id,killAt){return new Promise(res=>{
  const t0=now(),fd=fs.openSync(F,'r');let sent=0,open=true;
  const shut=()=>{if(open){open=false;try{fs.closeSync(fd)}catch{}}};
  const r=http.request({host:'127.0.0.1',port:PORT,method:'POST',path:`/naive/${id}`,agent:false,
    headers:{'content-length':SIZE}},x=>{x.resume();x.on('end',()=>{shut();res({sent,ms:now()-t0})})});
  r.on('error',()=>{shut();res({sent,ms:now()-t0})});
  const buf=Buffer.allocUnsafe(1<<20);let done=false;
  (function pump(){if(done)return;
    while(sent<SIZE){
      if(killAt!==undefined&&sent>=killAt){done=true;r.socket.destroy();shut();return res({sent,ms:now()-t0})}
      const n=fs.readSync(fd,buf,0,Math.min(buf.length,SIZE-sent),sent);sent+=n;
      if(!r.write(buf.subarray(0,n)))return r.once('drain',pump);
    } done=true;r.end();})();
})}
const req=(o,b)=>new Promise((ok,no)=>{const r=http.request({host:'127.0.0.1',port:PORT,agent:ag,...o},x=>{
  const c=[];x.on('data',d=>c.push(d));x.on('end',()=>ok({h:x.headers,b:Buffer.concat(c).toString()}))});
  r.on('error',no);b?r.end(b):r.end()});
async function chunked(id,from,killAt){
  const t0=now(),fd=fs.openSync(F,'r');let sent=0,reqs=0;
  for(let i=from;i*CS<SIZE;i++){
    const off=i*CS,len=Math.min(CS,SIZE-off),b=Buffer.allocUnsafe(len);fs.readSync(fd,b,0,len,off);
    if(killAt!==undefined&&sent+len>killAt){                 // die mid-chunk
      const part=Math.max(0,killAt-sent);
      await new Promise(r2=>{const r=http.request({host:'127.0.0.1',port:PORT,method:'PUT',agent:ag,
        path:`/c/${id}/${i}?off=${off}`,headers:{'content-length':len}},()=>{});
        r.on('error',()=>r2());r.write(b.subarray(0,part),()=>{sent+=part;r.socket.destroy();r2()})});
      fs.closeSync(fd);return{sent,reqs,ms:now()-t0};
    }
    await req({method:'PUT',path:`/c/${id}/${i}?off=${off}`,headers:{'content-length':len}},b);
    sent+=len;reqs++;
  } fs.closeSync(fd);return{sent,reqs,ms:now()-t0};
}
(async()=>{
  console.log(`file ${SIZE} bytes, 5 MiB chunks, sha ${src.slice(0,16)}`);
  console.log('\nstrategy  fail_at  bytes_resent  total_bytes  total_ms');
  for(const pct of [25,50,90]){
    const k=Math.floor(SIZE*pct/100);
    const a=await naive(`n${pct}`,k), b=await naive(`n${pct}b`);
    console.log(`naive     ${pct}%  ${a.sent}  ${a.sent+b.sent}  ${(a.ms+b.ms).toFixed(0)}`);
    const c=await chunked(`c${pct}`,0,k);
    const s=JSON.parse((await req({method:'GET',path:`/status/c${pct}`})).b);
    const d=await chunked(`c${pct}`,s.nextIndex);
    const fin=JSON.parse((await req({method:'GET',path:`/sha/c${pct}`})).b);
    console.log(`chunked   ${pct}%  ${c.sent-s.nextIndex*CS}  ${c.sent+d.sent}  ${(c.ms+d.ms).toFixed(0)}  assembled=${fin.size} sha_ok=${fin.sha===src}`);
  }
})();
EOF
node srv.cjs & SRV=$!
sleep 1
node run.cjs
kill $SRV 2>/dev/null || true
cd /; rm -rf "$D"

Output here, at 64 MiB:

file 67108864 bytes, 5 MiB chunks, sha c33e4c943c5f403d

strategy  fail_at  bytes_resent  total_bytes  total_ms
naive     25%  16777216  83886080  280
chunked   25%  1048576  68157440  147  assembled=67108864 sha_ok=true
naive     50%  33554432  100663296  237
chunked   50%  2097152  69206016  167  assembled=67108864 sha_ok=true
naive     90%  60817408  127926272  203
chunked   90%  2726297  69835161  112  assembled=67108864 sha_ok=true

At 90%, 60,817,408 bytes thrown away against 2,726,297 re-sent — a 22x difference on a 64 MiB file, growing linearly with file size.

Where this goes next

The download direction has a nicer answer, because HTTP already has Range: that is how do you serve a 2 GB file without using 2 GB of RAM?. Upload has no such standard, which is why every provider ships its own — and why those five decisions are yours to get wrong. Every file step in Workflow Builder chunks, checksums and writes at an offset, because the version that appends is the one that silently doubles your file.