Initial commit: router front-door vLLM stack

Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2,
sleep mode) behind a FastAPI router that auto-wakes models on request.
Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with
Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010.
Routine control via vllmctl is pure HTTP — no docker on the request path.

Verified: 91 router unit tests + 15-test E2E on real hardware
(measurements in CALIBRATION.md; design record in
.claude/memory/router-front-door-plan.md).

Old nginx stack files removed before git init; design survives in
.claude/memory/sleep-mode-implementation-plan.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

TODO.md

@@ -0,0 +1,53 @@
+# Router Front Door — implementation record
+
+**Completed:** 2026-08-17 · **Status:** implemented, E2E-verified, in service ✅
+
+Supersedes the Sleep Mode + nginx plan (v2, 2026-08-14) — that stack worked
+but did not wake models on request; the front door's real job. Design record:
+`.claude/memory/router-front-door-plan.md` (v3.3).
+
+## What was built
+
+| Piece | What it does |
+|---|---|
+| `router/` (FastAPI) | Only public ingress (:8000). Model routing, wake-on-request (single-flight), streaming proxy, tiered idle sleep/offload, depth-aware 503s, dev-endpoint hiding, persisted wake-intent recovery. Admin API on 127.0.0.1:8010. |
+| `compose.yml` | 4 services: `vllm-text` (TP=2 GPU0+1, util 0.85, 262K ctx), `vllm-ocr` (GPU2, 0.10, `--max-num-seqs 256`), `vllm-embed` (GPU2, 0.25), `router`. Named volume `router-state` for wake-intent. nginx removed. |
+| `vllmctl` | `status/up/down/sleep/list/logs/restart/pull`. Routine control is pure HTTP to the admin API — no docker, no credentials. |
+| `CALIBRATION.md` | Measured GPU footprints, sleep residuals, wake latencies (L1 fast path ~2.5–4 s text; L2 ~23 s), host RAM, backend quirks (`/health` lies while asleep; L1 wake needs `wake_up` only; L1→L2 re-sleep is a no-op → wake-then-sleep). |
+
+## Verification trail
+
+- Plan review: 2 independent rounds, all critical/major issues fixed.
+- Router unit tests: 91 passed (no GPU needed) — `router/tests/`.
+- E2E: 15-test matrix on real hardware — 14 PASS, 1 FAIL (router restart in
+  the wake_up→reload_weights window served garbage).
+- Fix: persisted wake-intent + startup recovery; re-verified PASS on the
+  real stack (503 `router_shutting_down` for in-flight, clean content after
+  recovery, backend logs show reload precedes any proxy).
+- Deployment fix: state volume switched from bind mount (uid mismatch,
+  silently unwritable) to a named volume seeded from image ownership.
+
+## Known limitations (accepted)
+
+- Host RAM grows by the weights size while a model naps at level 1
+  (~63 GB if all three nap; host has 1082 GB).
+- An NFS outage makes level-2-offloaded models unwakeable until NFS returns
+  (router 503s meanwhile). Optional mitigation: cap text at level 1.
+- A stream already in flight when the router shuts down is truncated
+  (client sees EOF, not a 503) — unavoidable once bytes are sent.
+- vLLM's chat endpoint is JSON-only: multipart is routed correctly but real
+  OCR calls use JSON + base64 images.
+- Host reboot: all models cold-boot awake (vLLM cannot boot asleep);
+  requests get `"restarting"` 503s for the load minutes.
+
+## Housekeeping
+
+- [ ] Delete `Qwen3.6-35B-A3B-FP8.corrupt-20260817` (43 GB, in MODEL_ROOT)
+      once the repaired FP8 copy is trusted.
+- [x] Old-stack leftovers removed 2026-08-17 before `git init`: nginx.conf,
+      `*.v2.bak`, `.runas.py`, `.user.env`, idle-watcher files. The v2 stack
+      survives only as design docs (`sleep-mode-implementation-plan.md`).
+- [ ] Confirm calling services' client timeouts ≥ ~60 s (offload wake).
+- [ ] Ask the admin to run the GPU fix ladder (`iommu=pt` decisive test) —
+      see `NOTES-2026-08-17-gpu2-recheck.md`. If P2P is ever fixed,
+      `NCCL_P2P_DISABLE` can be revisited (perf only; not required).