Design Case Studies · Lesson 02

Design: File upload/download API

Naively routing file bytes through your API server is the single most expensive mistake in file handling — it saturates bandwidth, blocks request threads, and scales poorly. This case study shows how presigned URLs to object storage let you handle large, resumable uploads without your API touching a single byte of payload.

Requirements

Functional requirements

• Clients can upload files of any type up to 5 GB.
• Uploads are resumable: an interrupted transfer can continue from the last successfully uploaded chunk rather than starting over.
• File integrity is verified end-to-end: the client provides a checksum; the server rejects uploads that do not match.
• Clients can download files via a time-limited authenticated URL.
• File metadata (name, size, content type, upload status) is queryable via the API.

Non-functional requirements

• The API service itself must not become a bottleneck as file traffic grows — the solution must scale bandwidth independently of the API fleet.
• Partial uploads must not leave permanent orphaned storage; incomplete uploads expire after 24 hours.
• Download URLs expire after a configurable TTL (default 1 hour) to prevent link sharing of private files.
• Chunk uploads must be safe to retry: re-uploading the same chunk must not corrupt the file.

Scale assumption

10 000 active uploads per day, average file size 200 MB (so ~10 MB chunks, ~20 chunks per file). Peak concurrent uploads: ~500. Total daily data volume: ~2 TB. At this scale, a naive proxy architecture would require roughly 2 TB of bandwidth through the API fleet every day — which is the problem the presigned-URL design avoids entirely.

Design decisions

Presigned URLs to object storage — the core move

The central decision is: the API server never touches the file bytes. Instead, the API server exchanges metadata and issues short-lived signed URLs that let the client upload directly to object storage (e.g., S3, GCS, Azure Blob). The storage service checks the signature, accepts the bytes, and stores them. The API server's job is to orchestrate — create the upload record, generate the URLs, verify completion — not to proxy gigabytes of data.

This matters because object storage is built to ingest large byte streams at high concurrency; a general-purpose application server is not. A single 500 MB upload proxied through an API server ties up a connection and thread for potentially minutes. With presigned URLs, a 500 MB upload adds exactly two lightweight API calls (initialise and complete) regardless of file size; the heavy bytes travel on a separate path that scales independently.

Chunked / multipart upload

Splitting a 5 GB file into 10 MB chunks makes uploads resumable at a fine granularity. Each chunk gets its own presigned URL; the client uploads chunks in any order (or in parallel for speed) and tracks which have succeeded. On network failure, only the failed chunk needs to be retried — not the entire file. The storage service assembles the final file only after the client signals completion.

Idempotent chunk PUT

Each chunk PUT is addressed by its position (e.g., chunk number 4 of upload upl_abc). Re-uploading the same chunk to the same position is idempotent: the storage layer either acknowledges the existing chunk or overwrites it with the same bytes. The client never needs to know whether a chunk was received; it retries on any non-2xx response. See idempotency for the full treatment of why this matters for retries.

Checksums for integrity

The client computes an MD5 (or stronger: SHA-256) checksum of each chunk before uploading and includes it in the presigned URL or as a header. The storage service rejects chunks whose bytes do not match the declared checksum. The final "complete" call includes the overall file checksum; the API verifies the assembled file against it before marking the upload as ready. This catches bit-flip corruption and partial writes without requiring the API to inspect the bytes itself.

Rejected alternative: server-side streaming proxy

A streaming proxy (where the API streams bytes from the client to storage without buffering them all in memory) avoids the memory problem but not the connection-holding problem. During a slow upload, the API server holds a TCP connection open for minutes, preventing that connection from serving other requests. At 500 concurrent uploads, that is 500 connections tied up doing no compute work. Presigned URLs avoid this entirely.

The API model

===================================================================
 PHASE 1 — Initialise an upload
===================================================================
POST /v1/files
Authorization: Bearer <token>
Content-Type: application/json

{
  "filename": "quarterly-report.pdf",
  "content_type": "application/pdf",
  "size_bytes": 204857600,         // 200 MB — used to plan chunk count
  "sha256": "e3b0c44298fc1c149..."  // whole-file checksum declared upfront
}

→ 201 Created
{
  "upload_id": "upl_7g3kf",
  "status": "pending",
  "chunk_size_bytes": 10485760,   // 10 MB server-recommended chunk size
  "chunk_count": 20,
  "expires_at": "2026-06-21T00:00:00Z"  // incomplete uploads expire in 24 h
}

===================================================================
 PHASE 2a — Get a presigned URL for a chunk
===================================================================
GET /v1/files/upl_7g3kf/chunks/4/url
Authorization: Bearer <token>

→ 200 OK
{
  "presigned_url": "https://storage.example.com/uploads/upl_7g3kf/4?X-Amz-Signature=...",
  "url_expires_at": "2026-06-20T12:15:00Z",  // 15-minute URL lifetime
  "chunk_number": 4,
  "expected_md5": "7215ee9c7d9dc229d..."     // client verifies locally before PUT
}

===================================================================
 PHASE 2b — Client PUTs chunk directly to storage (no API involved)
===================================================================
PUT https://storage.example.com/uploads/upl_7g3kf/4?X-Amz-Signature=...
Content-Length: 10485760
Content-MD5: "7215ee9c7d9dc229d..."

<10 MB of binary data>

→ 200 OK  (from storage service, not from the API)
ETag: "7215ee9c7d9dc229d..."   // returned by storage for the complete call

===================================================================
 PHASE 3 — Signal upload complete
===================================================================
POST /v1/files/upl_7g3kf/complete
Authorization: Bearer <token>
Content-Type: application/json

{
  "parts": [
    { "chunk_number": 1, "etag": "a87ff679a2f3e71d9181..." },
    { "chunk_number": 2, "etag": "e4da3b7fbbce2345d77..." },
    /* … all 20 parts … */
  ]
}

→ 200 OK
{
  "upload_id": "upl_7g3kf",
  "file_id": "file_q9r2p",
  "status": "ready",
  "size_bytes": 204857600,
  "sha256_verified": true,
  "created_at": "2026-06-20T12:00:05Z"
}

===================================================================
 DOWNLOAD — Get a time-limited download URL
===================================================================
GET /v1/files/file_q9r2p/download-url?ttl_seconds=3600
Authorization: Bearer <token>

→ 200 OK
{
  "url": "https://storage.example.com/files/file_q9r2p?X-Amz-Signature=...",
  "expires_at": "2026-06-20T13:00:05Z"
}

===================================================================
 METADATA — Query file status
===================================================================
GET /v1/files/file_q9r2p
Authorization: Bearer <token>

→ 200 OK
{
  "id": "file_q9r2p",
  "filename": "quarterly-report.pdf",
  "content_type": "application/pdf",
  "size_bytes": 204857600,
  "status": "ready",    // pending | ready | failed | expired
  "sha256": "e3b0c44298fc1c149...",
  "created_at": "2026-06-20T12:00:05Z"
}

Evaluation & latency budget

Why direct-to-storage offloads the API

Consider a 200 MB upload proxied through the API server vs. via presigned URL:

Resumability walkthrough

Suppose a client is uploading chunk 12 of 20 when the network drops. When it reconnects:

1. Call GET /v1/files/upl_7g3kf. The response includes a completed_parts array listing chunks 1–11 with their ETags.
2. Identify the gap: chunks 12–20 are missing.
3. Request presigned URLs for chunks 12–20. Chunk 12 may have been partially received; the idempotent PUT will handle the retry safely.
4. Upload chunks 12–20 directly to storage.
5. Call POST /v1/files/upl_7g3kf/complete with all 20 ETags. The API assembles and verifies the file.

Total extra work on resume: only the missing chunks. For a 200 MB file with 10 MB chunks, a failure at chunk 12 means re-uploading at most ~90 MB, not the full 200 MB. For a 5 GB file, this is the difference between a failed upload and a tolerable retry.

Under the hood: the core mechanism

Presigned URLs look like magic — a URL the client uses to talk directly to storage without any credentials of its own. The mechanism is a conventional HMAC signature over a canonical string that encodes the intent: who authorized it, for which object, doing what, until when. Tamper with any component and the HMAC no longer matches.

How a presigned URL is constructed and verified

-- SERVER-SIDE: construct the presigned URL for chunk 4 of upload upl_7g3kf --

Step 1: build the canonical string-to-sign
string_to_sign =
  "PUT\n"                              // HTTP method
  + "7215ee9c7d9dc229d2921a40e899ec5f\n"  // Content-MD5 of this chunk (hex)
  + "application/octet-stream\n"         // Content-Type
  + "1750003600\n"                        // expiry as Unix timestamp (now + 900 s)
  + "/uploads/upl_7g3kf/4"               // object path (bucket + key)

Step 2: sign with HMAC-SHA256 using the storage service's secret key
raw_sig     = HMAC_SHA256(string_to_sign, secret_access_key)
signature   = base64url(raw_sig)
→ "SflKxwRJSMeKKF2QT4fwpMeJf36P..."

Step 3: assemble the presigned URL
presigned_url =
  "https://storage.example.com/uploads/upl_7g3kf/4"
  + "?X-Amz-Expires=900"
  + "&X-Amz-Date=20260620T120000Z"
  + "&X-Amz-Signature=SflKxwRJSMeKKF2QT4fwpMeJf36P..."

-- CLIENT-SIDE: PUT the chunk bytes to the presigned URL --

PUT https://storage.example.com/uploads/upl_7g3kf/4?X-Amz-Expires=900&X-Amz-Signature=...
Content-Length: 10485760
Content-MD5: 7215ee9c7d9dc229d2921a40e899ec5f
Content-Type: application/octet-stream

<10 MB binary data>

-- STORAGE-SIDE: verify the presigned URL --

1. Recompute string-to-sign from the incoming request (method + content-md5 + content-type + expiry + path)
2. Recompute HMAC-SHA256 with the stored secret key
3. Compare: constant_time_equals(recomputed_sig, X-Amz-Signature query param)
   → mismatch → 403 SignatureDoesNotMatch
4. Check expiry: now() > X-Amz-Date + X-Amz-Expires
   → expired  → 403 RequestExpired
5. Receive bytes; compute MD5 of received data; compare with Content-MD5 header
   → mismatch → 400 BadDigest (chunk corrupted in transit)
6. Store chunk; return 200 OK with ETag = MD5 of chunk (for the complete call)
   ETag: "7215ee9c7d9dc229d2921a40e899ec5f"

-- PHASE 3: multipart assembly (what POST /complete triggers) --

The client sends all 20 {chunk_number, etag} pairs to POST /v1/files/upl_7g3kf/complete.
Storage assembles the file:
  1. Sort parts by chunk_number
  2. Verify each part's stored ETag matches the submitted ETag
     → mismatch on part 7 → 422 (ETag in complete call doesn't match uploaded chunk)
  3. Concatenate all part byte-streams in order
  4. Compute final ETag = MD5( concat(etag_1, etag_2, ..., etag_20) ) + "-20"
     (the "-N" suffix is an S3 convention for multi-part ETags)
  5. Compute SHA-256 of assembled file; API compares with declared sha256 from Phase 1
     → mismatch → 422, status: failed
  6. Mark file as status: ready; issue file_id

Operating & debugging it

Upload failures almost always fall into three buckets: a signature problem, an expiry problem, or a checksum mismatch. Each produces a distinct error code that points straight at the cause.

Debug checklist:

1. Test the presigned URL directly with curl (no client app) — if curl succeeds and the browser fails, the issue is CORS on the storage bucket, not the signature.
2. Check the error code in the storage response body (XML or JSON): SignatureDoesNotMatch, RequestExpired, and BadDigest each point to a different root cause.
3. Call GET /v1/files/:upload_id to see the completed_parts list — confirm which chunks the API believes were successfully received before attempting to resume.
4. For checksum failures at completion: recompute the SHA-256 of the original file locally and compare with what was sent in Phase 1; a mismatch there means the client computed the checksum incorrectly before declaring it.
5. Verify that each chunk is read from the correct byte offset: chunk N starts at (N-1) × chunk_size_bytes and ends at min(N × chunk_size_bytes, file_size) — off-by-one errors here produce files that are byte-correct per chunk but in the wrong order or with gaps.

GET /v1/files?status=ready&cursor=eyJjcmVhdGVkIjoiMjAyN..."&limit=25
Authorization: Bearer <token>

→ 200 OK
{
  "data": [
    {
      "id": "file_q9r2p",
      "filename": "quarterly-report.pdf",
      "size_bytes": 204857600,
      "status": "ready",
      "created_at": "2026-06-20T12:00:05Z"
    }
  ],
  "next_cursor": "eyJjcmVhdGVkIjoiMjAy...",
  "total_estimate": 147
}

Sources & further reading

• AWS S3 — Multipart Upload Overview (the production implementation of this exact pattern)
• Google Cloud Storage — Resumable Uploads (GCS implementation; compare with S3 to see where protocols converge)
• AWS S3 — Presigned URLs (how signatures work, URL lifetime, security considerations)
• RFC 9110 §9.3.4 — PUT method semantics (idempotency defined by the spec)
• API Design course — Idempotency
• tus — Open Protocol for Resumable File Uploads (open standard for client-agnostic resumable uploads)