Files

bing 80eef4ce6a Initial commit: router front-door vLLM stack

Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2,
sleep mode) behind a FastAPI router that auto-wakes models on request.
Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with
Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010.
Routine control via vllmctl is pure HTTP — no docker on the request path.

Verified: 91 router unit tests + 15-test E2E on real hardware
(measurements in CALIBRATION.md; design record in
.claude/memory/router-front-door-plan.md).

Old nginx stack files removed before git init; design survives in
.claude/memory/sleep-mode-implementation-plan.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-17 10:17:42 +00:00

3.2 KiB

Raw Permalink Blame History

Router Front Door — implementation record

Completed: 2026-08-17 · Status: implemented, E2E-verified, in service ✅

Supersedes the Sleep Mode + nginx plan (v2, 2026-08-14) — that stack worked but did not wake models on request; the front door's real job. Design record: .claude/memory/router-front-door-plan.md (v3.3).

What was built

Piece	What it does
`router/` (FastAPI)	Only public ingress (:8000). Model routing, wake-on-request (single-flight), streaming proxy, tiered idle sleep/offload, depth-aware 503s, dev-endpoint hiding, persisted wake-intent recovery. Admin API on 127.0.0.1:8010.
`compose.yml`	4 services: `vllm-text` (TP=2 GPU0+1, util 0.85, 262K ctx), `vllm-ocr` (GPU2, 0.10, `--max-num-seqs 256`), `vllm-embed` (GPU2, 0.25), `router`. Named volume `router-state` for wake-intent. nginx removed.
`vllmctl`	`status/up/down/sleep/list/logs/restart/pull`. Routine control is pure HTTP to the admin API — no docker, no credentials.
`CALIBRATION.md`	Measured GPU footprints, sleep residuals, wake latencies (L1 fast path ~2.5–4 s text; L2 ~23 s), host RAM, backend quirks (`/health` lies while asleep; L1 wake needs `wake_up` only; L1→L2 re-sleep is a no-op → wake-then-sleep).

Verification trail

Plan review: 2 independent rounds, all critical/major issues fixed.
Router unit tests: 91 passed (no GPU needed) — router/tests/.
E2E: 15-test matrix on real hardware — 14 PASS, 1 FAIL (router restart in the wake_up→reload_weights window served garbage).
Fix: persisted wake-intent + startup recovery; re-verified PASS on the real stack (503 router_shutting_down for in-flight, clean content after recovery, backend logs show reload precedes any proxy).
Deployment fix: state volume switched from bind mount (uid mismatch, silently unwritable) to a named volume seeded from image ownership.

Known limitations (accepted)

Host RAM grows by the weights size while a model naps at level 1 (~63 GB if all three nap; host has 1082 GB).
An NFS outage makes level-2-offloaded models unwakeable until NFS returns (router 503s meanwhile). Optional mitigation: cap text at level 1.
A stream already in flight when the router shuts down is truncated (client sees EOF, not a 503) — unavoidable once bytes are sent.
vLLM's chat endpoint is JSON-only: multipart is routed correctly but real OCR calls use JSON + base64 images.
Host reboot: all models cold-boot awake (vLLM cannot boot asleep); requests get "restarting" 503s for the load minutes.

Housekeeping

Delete Qwen3.6-35B-A3B-FP8.corrupt-20260817 (43 GB, in MODEL_ROOT) once the repaired FP8 copy is trusted.
Old-stack leftovers removed 2026-08-17 before git init: nginx.conf, *.v2.bak, .runas.py, .user.env, idle-watcher files. The v2 stack survives only as design docs (sleep-mode-implementation-plan.md).
Confirm calling services' client timeouts ≥ ~60 s (offload wake).
Ask the admin to run the GPU fix ladder (iommu=pt decisive test) — see NOTES-2026-08-17-gpu2-recheck.md. If P2P is ever fixed, NCCL_P2P_DISABLE can be revisited (perf only; not required).