# Router Front Door — implementation record

**Completed:** 2026-08-17 · **Status:** implemented, E2E-verified, in service ✅

Supersedes the Sleep Mode + nginx plan (v2, 2026-08-14) — that stack worked
but did not wake models on request; the front door's real job. Design record:
`.claude/memory/router-front-door-plan.md` (v3.3).

## What was built

| Piece | What it does |
|---|---|
| `router/` (FastAPI) | Only public ingress (:8000). Model routing, wake-on-request (single-flight), streaming proxy, tiered idle sleep/offload, depth-aware 503s, dev-endpoint hiding, persisted wake-intent recovery. Admin API on 127.0.0.1:8010. |
| `compose.yml` | 4 services: `vllm-text` (TP=2 GPU0+1, util 0.85, 262K ctx), `vllm-ocr` (GPU2, 0.10, `--max-num-seqs 256`), `vllm-embed` (GPU2, 0.25), `router`. Named volume `router-state` for wake-intent. nginx removed. |
| `vllmctl` | `status/up/down/sleep/list/logs/restart/pull`. Routine control is pure HTTP to the admin API — no docker, no credentials. |
| `CALIBRATION.md` | Measured GPU footprints, sleep residuals, wake latencies (L1 fast path ~2.5–4 s text; L2 ~23 s), host RAM, backend quirks (`/health` lies while asleep; L1 wake needs `wake_up` only; L1→L2 re-sleep is a no-op → wake-then-sleep). |

## Verification trail

- Plan review: 2 independent rounds, all critical/major issues fixed.
- Router unit tests: 91 passed (no GPU needed) — `router/tests/`.
- E2E: 15-test matrix on real hardware — 14 PASS, 1 FAIL (router restart in
  the wake_up→reload_weights window served garbage).
- Fix: persisted wake-intent + startup recovery; re-verified PASS on the
  real stack (503 `router_shutting_down` for in-flight, clean content after
  recovery, backend logs show reload precedes any proxy).
- Deployment fix: state volume switched from bind mount (uid mismatch,
  silently unwritable) to a named volume seeded from image ownership.

## Known limitations (accepted)

- Host RAM grows by the weights size while a model naps at level 1
  (~63 GB if all three nap; host has 1082 GB).
- An NFS outage makes level-2-offloaded models unwakeable until NFS returns
  (router 503s meanwhile). Optional mitigation: cap text at level 1.
- A stream already in flight when the router shuts down is truncated
  (client sees EOF, not a 503) — unavoidable once bytes are sent.
- vLLM's chat endpoint is JSON-only: multipart is routed correctly but real
  OCR calls use JSON + base64 images.
- Host reboot: all models cold-boot awake (vLLM cannot boot asleep);
  requests get `"restarting"` 503s for the load minutes.

## Housekeeping

- [ ] Delete `Qwen3.6-35B-A3B-FP8.corrupt-20260817` (43 GB, in MODEL_ROOT)
      once the repaired FP8 copy is trusted.
- [x] Old-stack leftovers removed 2026-08-17 before `git init`: nginx.conf,
      `*.v2.bak`, `.runas.py`, `.user.env`, idle-watcher files. The v2 stack
      survives only as design docs (`sleep-mode-implementation-plan.md`).
- [ ] Confirm calling services' client timeouts ≥ ~60 s (offload wake).
- [ ] Ask the admin to run the GPU fix ladder (`iommu=pt` decisive test) —
      see `NOTES-2026-08-17-gpu2-recheck.md`. If P2P is ever fixed,
      `NCCL_P2P_DISABLE` can be revisited (perf only; not required).