bing
80eef4ce6a
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
54 lines
3.2 KiB
Markdown
54 lines
3.2 KiB
Markdown
# Router Front Door — implementation record
|
||
|
||
**Completed:** 2026-08-17 · **Status:** implemented, E2E-verified, in service ✅
|
||
|
||
Supersedes the Sleep Mode + nginx plan (v2, 2026-08-14) — that stack worked
|
||
but did not wake models on request; the front door's real job. Design record:
|
||
`.claude/memory/router-front-door-plan.md` (v3.3).
|
||
|
||
## What was built
|
||
|
||
| Piece | What it does |
|
||
|---|---|
|
||
| `router/` (FastAPI) | Only public ingress (:8000). Model routing, wake-on-request (single-flight), streaming proxy, tiered idle sleep/offload, depth-aware 503s, dev-endpoint hiding, persisted wake-intent recovery. Admin API on 127.0.0.1:8010. |
|
||
| `compose.yml` | 4 services: `vllm-text` (TP=2 GPU0+1, util 0.85, 262K ctx), `vllm-ocr` (GPU2, 0.10, `--max-num-seqs 256`), `vllm-embed` (GPU2, 0.25), `router`. Named volume `router-state` for wake-intent. nginx removed. |
|
||
| `vllmctl` | `status/up/down/sleep/list/logs/restart/pull`. Routine control is pure HTTP to the admin API — no docker, no credentials. |
|
||
| `CALIBRATION.md` | Measured GPU footprints, sleep residuals, wake latencies (L1 fast path ~2.5–4 s text; L2 ~23 s), host RAM, backend quirks (`/health` lies while asleep; L1 wake needs `wake_up` only; L1→L2 re-sleep is a no-op → wake-then-sleep). |
|
||
|
||
## Verification trail
|
||
|
||
- Plan review: 2 independent rounds, all critical/major issues fixed.
|
||
- Router unit tests: 91 passed (no GPU needed) — `router/tests/`.
|
||
- E2E: 15-test matrix on real hardware — 14 PASS, 1 FAIL (router restart in
|
||
the wake_up→reload_weights window served garbage).
|
||
- Fix: persisted wake-intent + startup recovery; re-verified PASS on the
|
||
real stack (503 `router_shutting_down` for in-flight, clean content after
|
||
recovery, backend logs show reload precedes any proxy).
|
||
- Deployment fix: state volume switched from bind mount (uid mismatch,
|
||
silently unwritable) to a named volume seeded from image ownership.
|
||
|
||
## Known limitations (accepted)
|
||
|
||
- Host RAM grows by the weights size while a model naps at level 1
|
||
(~63 GB if all three nap; host has 1082 GB).
|
||
- An NFS outage makes level-2-offloaded models unwakeable until NFS returns
|
||
(router 503s meanwhile). Optional mitigation: cap text at level 1.
|
||
- A stream already in flight when the router shuts down is truncated
|
||
(client sees EOF, not a 503) — unavoidable once bytes are sent.
|
||
- vLLM's chat endpoint is JSON-only: multipart is routed correctly but real
|
||
OCR calls use JSON + base64 images.
|
||
- Host reboot: all models cold-boot awake (vLLM cannot boot asleep);
|
||
requests get `"restarting"` 503s for the load minutes.
|
||
|
||
## Housekeeping
|
||
|
||
- [ ] Delete `Qwen3.6-35B-A3B-FP8.corrupt-20260817` (43 GB, in MODEL_ROOT)
|
||
once the repaired FP8 copy is trusted.
|
||
- [x] Old-stack leftovers removed 2026-08-17 before `git init`: nginx.conf,
|
||
`*.v2.bak`, `.runas.py`, `.user.env`, idle-watcher files. The v2 stack
|
||
survives only as design docs (`sleep-mode-implementation-plan.md`).
|
||
- [ ] Confirm calling services' client timeouts ≥ ~60 s (offload wake).
|
||
- [ ] Ask the admin to run the GPU fix ladder (`iommu=pt` decisive test) —
|
||
see `NOTES-2026-08-17-gpu2-recheck.md`. If P2P is ever fixed,
|
||
`NCCL_P2P_DISABLE` can be revisited (perf only; not required).
|