Design Case Studies · Lesson 11

Design: Online Judge API

Running user-submitted code is one of the most dangerous things a server can do. Designing it correctly means separating the act of accepting a submission from the act of running it — and then running it inside a cage it cannot escape.

1 — Requirements

An online judge does one thing: accept code, run it against test cases, and return a verdict. The operational constraints make that simple statement complicated.

Functional requirements

• Submit code: accept a problem ID, a language, and source code. Return an identifier the client can use to check status.
• Run in isolation: execute the code against a private set of judge test cases. User code must not be able to read the test cases, communicate with the network, or affect other submissions.
• Return a verdict: Accepted, Wrong Answer, Time Limit Exceeded, Memory Limit Exceeded, Runtime Error, or Compile Error — plus which test case failed and why.
• Async by default: execution can take up to a few seconds. The API must not hold an HTTP connection open waiting for it.
• Webhook or poll: clients need a way to learn when the verdict is ready, either by polling GET or by registering a webhook URL.

Non-functional requirements

• Security isolation: untrusted code runs in a sandboxed container with a read-only filesystem, no network, no host IPC, strict seccomp syscall filtering, and CPU/memory/time limits enforced by the kernel.
• Scale: submissions burst heavily around contest start times. The queue absorbs spikes; worker count scales independently of the API tier.
• Idempotency: a client retrying a submission (after a network timeout) should not produce two judge runs for the same code.
• Rate limiting: per-user and per-IP submission limits prevent compute abuse. See Lesson rel-03.
• Resource limits: each run has a wall-clock time limit (typically 1–3 s), a memory ceiling (typically 256 MB), and an output size cap to prevent fork bombs and output floods.

2 — Design decisions

Decision 1: Accept synchronously, execute asynchronously

A naive design executes code inline during the HTTP request and returns the verdict in the response body. This breaks under load: a single slow Python solution (3 s time limit) holds an HTTP worker thread for 3 seconds. At 100 concurrent slow submissions, 100 threads are blocked doing nothing. HTTP servers are designed for millisecond-range request handling — holding threads for seconds is an antipattern that collapses throughput.

The correct split is a 202 Accepted pattern: the API validates the submission, writes it to a queue, and returns an ID immediately. Workers pull from the queue at their own pace. This decoupling appears in Lesson rel-10 (Event-driven pub/sub): the API is the producer, judge workers are consumers.

Decision 2: Idempotency on submission

Network timeouts happen. A client that submits and then never receives a response will retry. Without idempotency, two identical retries produce two separate judge runs — burning compute and potentially producing two entries on a leaderboard. The fix is a client-generated idempotency key (see Lesson rel-02): if the key was seen before, the server returns the original submission's ID without creating a new one.

Decision 3: Verdict delivery — poll vs webhook

Clients need to know when execution finishes. Two options:

• Poll: client calls GET /v1/submissions/:id repeatedly. Simple, stateless, slightly wasteful. Good for browser clients that can't receive incoming connections.
• Webhook: client registers a callback URL when submitting; the judge POSTs the verdict to that URL when done. Efficient for server-to-server integrations (CI pipelines, contest backends). Requires the client to run an HTTPS endpoint. See Lesson dbg-04 for reliability patterns.

Expose both. Browser clients poll; backend systems use webhooks.

Decision 4: Sandbox architecture

User code is hostile code until proven otherwise. The sandbox enforces isolation at multiple layers:

1. Container isolation (Linux namespaces + cgroups): each run gets a fresh container. Network namespace has no external routes. PID namespace prevents processes from seeing or signaling the host or other containers.
2. Read-only filesystem + mounted test cases: source code is injected into an ephemeral tmpfs. Test case input is mounted read-only. The judge binary itself is read-only. The user process cannot write to anything persistent.
3. Seccomp syscall filter: only a minimal whitelist of syscalls is permitted — read, write, mmap, clock_gettime, exit. execve to a different binary, socket, ptrace, and fork-into-fork are all blocked. Attempting a forbidden syscall kills the process immediately with a Runtime Error verdict.

Decision 5: Queue depth as the primary scaling lever

Contest start events create submission spikes that can be 20× the average rate. The queue absorbs that burst: the API tier scales with web traffic (fast, cheap), while the worker tier scales with execution throughput (slow, CPU-bound). Operators can pre-warm workers before a contest starts without touching the API layer.

3 — The API model

Submit a solution

POST /v1/submissions
Authorization: Bearer <user-token>
Idempotency-Key: client-generated-uuid
Content-Type: application/json

{
  "problem_id": "prob_twosum",
  "language":   "python3",
  "source_code": "class Solution:\n    def twoSum...",
  "webhook_url": "https://app.example.com/hooks/judge"  // optional
}

# Accepted — execution has NOT started yet
HTTP/1.1 202 Accepted
{
  "id":         "sub_4mNpQ7",
  "status":     "queued",
  "problem_id": "prob_twosum",
  "language":   "python3",
  "submitted_at": "2025-09-01T14:22:11Z"
}

# If the Idempotency-Key was seen before, same 202 with the original submission id
# No duplicate judge run is created

Poll for the verdict

GET /v1/submissions/sub_4mNpQ7
Authorization: Bearer <user-token>

# While running
HTTP/1.1 200 OK
{
  "id":     "sub_4mNpQ7",
  "status": "running",
  "verdict": null
}

# Completed — Accepted
HTTP/1.1 200 OK
{
  "id":       "sub_4mNpQ7",
  "status":   "finished",
  "verdict":  "Accepted",
  "runtime_ms": 48,
  "memory_kb":  14320,
  "passed_cases": 72,
  "total_cases":  72
}

# Completed — Wrong Answer
{
  "verdict":   "Wrong Answer",
  "failed_case": 5,           // 1-indexed
  "expected":  "[0,1]",
  "got":       "[1,0]"
}

# Completed — Time Limit Exceeded
{
  "verdict":    "Time Limit Exceeded",
  "failed_case": 61,
  "limit_ms":   2000
}

Webhook delivery (when webhook_url was provided)

# Judge POSTs to the registered webhook URL when execution finishes
POST https://app.example.com/hooks/judge
X-Judge-Signature: sha256=...   // HMAC — verify before trusting
Content-Type: application/json

{
  "event":         "submission.finished",
  "submission_id": "sub_4mNpQ7",
  "verdict":       "Accepted",
  "runtime_ms":    48
}

# Client must respond 200; non-200 triggers exponential retry up to 3×

4 — Evaluation & latency budget

The async pattern under load

Why 202 instead of 200? The POST contract is: I have accepted your request and will process it. If we returned 200, the client would expect the response body to contain the final verdict — which we cannot provide until execution finishes. 202 lets us return immediately while being honest about what has happened: the submission is in the queue, not yet run.

This is the same pattern as image transcoding, email sending, or any job that is measured in seconds rather than milliseconds. The API tier and the worker tier scale completely independently: if a contest attracts 10× the usual submissions, operators spin up more workers without touching the API servers — because the queue absorbs the burst.

Queue depth as a health signal

Security isolation analysis

Why three layers and not one? Because each layer has a different failure mode:

• Container/namespace isolation prevents a running process from seeing the host filesystem, other containers, or the network. But if the container runtime itself has a kernel exploit (rare but possible), the process could escape.
• Seccomp syscall filtering is enforced by the kernel BPF engine — not by the container runtime. Even if the container boundary were breached, a process that cannot call socket() or ptrace() has severely limited escape paths.
• Resource limits via cgroups prevent denial-of-service against the worker host — a solution that loops forever hits the wall-clock limit; a solution that allocates unbounded memory hits the cgroup ceiling and is killed, not the host.

Under the hood: the core mechanism

Two mechanisms make an online judge simultaneously safe and fair: the multi-layer sandbox that prevents user code from escaping, and the submit→queue→worker→verdict pipeline that ensures execution never blocks the API server. Understanding both at the implementation level lets you reason about what each verdict actually means and how to debug unexpected ones.

Sandbox isolation: how it actually works

When a worker dequeues a submission, it does not simply run the code on the host machine. It builds a tight cage using three independent kernel mechanisms, each of which would alone be insufficient:

The seccomp whitelist for a typical judge contains roughly 30–50 syscalls out of the ~400+ available on Linux x86-64. The filter is a compiled BPF bytecode program loaded into the kernel with prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &bpf_prog) before the user binary starts. From that point, every syscall instruction in the user process first passes through the BPF VM; if the syscall number is not on the allow-list, the kernel delivers SIGKILL synchronously — there is no way for user code to catch or defer it.

The submit → queue → worker → verdict pipeline

The four stages are deliberately decoupled so that each one can scale and fail independently:

Stage 1 — API server (milliseconds)
  Client:  POST /v1/submissions  { problem_id, language, source_code }
           Idempotency-Key: <client-uuid>
  Server:
    1a. Check idempotency key in Redis/DB — if found, return stored submission
    1b. Validate request (language supported? code within size limit? rate limit OK?)
    1c. Write submission row to DB:  { id, status:"queued", code, problem_id, user_id }
    1d. Publish message to queue:   { submission_id: "sub_4mNpQ7" }
    1e. Return 202:  { id: "sub_4mNpQ7", status: "queued" }

Stage 2 — Message queue (milliseconds to seconds)
  Queue holds: [ "sub_4mNpQ7", "sub_...", ... ]
  Workers compete for messages; each message is leased (invisible) for 30 s.
  If no worker ACKs within 30 s, message becomes visible again → retry.

Stage 3 — Judge worker (1–5 seconds per submission)
  Worker:
    3a. Dequeue submission_id; fetch code + problem from DB
    3b. Update DB: status → "running"
    3c. Create sandbox container (namespace + cgroups + seccomp + ro-fs)
    3d. For each test case (sequentially):
          Inject test case stdin into tmpfs
          Run compiled binary (or interpreter) inside sandbox
          Capture stdout, stderr, exit code, wall time, peak RSS
          Compare stdout to expected output
          If any check fails → record failed_case, stop iteration
    3e. Compile all results → verdict
    3f. Write verdict to DB:  { status:"finished", verdict:"Accepted", runtime_ms, memory_kb }
    3g. ACK queue message (delete lease)
    3h. If webhook_url present → enqueue webhook delivery job

Stage 4 — Verdict delivery (poll or webhook)
  Poll:    Client GET /v1/submissions/sub_4mNpQ7 → reads DB row
  Webhook: Delivery worker POSTs result to client URL with HMAC signature

Worked trace: one submission from POST to verdict

T=0 ms    Client sends POST /v1/submissions
           Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
           { problem_id:"prob_twosum", language:"python3", source_code:"class Solution:..." }

T=8 ms    API writes row to DB (status: queued), publishes to queue
           Returns: 202 { id:"sub_4mNpQ7", status:"queued" }

T=12 ms   Worker dequeues sub_4mNpQ7 (queue depth was 3, wait ≈ 0)
           Fetches submission + problem metadata from DB
           Updates DB: status → "running"

T=15 ms   Worker spawns sandbox container
           - unshare(CLONE_NEWNET | CLONE_NEWPID | CLONE_NEWNS)
           - cgroup limits: cpu.max=1000000 1000000, memory.max=268435456, pids.max=64
           - seccomp BPF loaded
           - source_code written to tmpfs /sandbox/solution.py

T=18 ms   Compile step (Python: no-op; C++: g++ -O2 -o /sandbox/a.out /sandbox/sol.cpp)
T=70 ms   Test case 1/72: stdin → [2,7,11,15], 9
           stdout: [0,1]   expected: [0,1]   ✓   runtime: 48 ms, rss: 14 MB

           ... test cases 2–72 run in 48–96 ms each ...
T=3700 ms All 72 cases passed
           Verdict: Accepted   total_runtime: 3682 ms   peak_memory: 14320 KB

T=3705 ms DB update: { status:"finished", verdict:"Accepted", runtime_ms:3682, memory_kb:14320 }
           Queue message ACKed (deleted)

T=3710 ms Webhook job enqueued (if webhook_url was provided)

T=3800 ms Client polls GET /v1/submissions/sub_4mNpQ7
           ← 200 { status:"finished", verdict:"Accepted", runtime_ms:48, memory_kb:14320,
                   passed_cases:72, total_cases:72 }
           (runtime_ms here is the median/best-case per test case, not total wall time)

Key timing insight: the 3682 ms total wall time is the sum of 72 test case runs. The runtime_ms: 48 returned to the client is the per-case execution time (the slowest case), which is what's compared against the problem's time limit — not the full wall-clock duration of the judge run.

Operating & debugging it

Judge debugging splits into two categories: unexpected verdicts (code that should pass but doesn't, or passes when it shouldn't) and operational problems (slow verdicts, stuck queue, workers crashing). The tools and investigation paths are different for each.

Inspecting the pipeline

Symptom → cause → fix

{ "status": "queued", "queue_position": 312, "estimated_wait_s": 94 }

Sources & further reading

• Linux man page — seccomp(2)
• Linux kernel docs — Control Group v2
• IOI/isolate — the sandbox used by major judges (Codeforces, DMOJ)
• Codeforces — How the Judge Works (blog post)
• MDN — HTTP 202 Accepted
• Stripe — Idempotent Requests