Initial commit: router front-door vLLM stack

Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2,
sleep mode) behind a FastAPI router that auto-wakes models on request.
Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with
Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010.
Routine control via vllmctl is pure HTTP — no docker on the request path.

Verified: 91 router unit tests + 15-test E2E on real hardware
(measurements in CALIBRATION.md; design record in
.claude/memory/router-front-door-plan.md).

Old nginx stack files removed before git init; design survives in
.claude/memory/sleep-mode-implementation-plan.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.claude/memory/router-front-door-plan.md

@@ -0,0 +1,339 @@
+# Plan v3.3 — Router Front Door (auto wake-on-request)
+
+**Date:** 2026-08-14, revised 2026-08-17 (v3.3)
+**Status:** **IMPLEMENTED & VERIFIED 2026-08-17** — v3.2 passed two review rounds (5 critical + 6 major fixed and confirmed; round-2 minors applied). v3.3 = hardware-driven placement change (GPU2 cleared for single-GPU services → OCR+embed on GPU2, text exclusive on GPU0/1). Implementation record and verification trail: `../../TODO.md`; measured numbers: `../../CALIBRATION.md`. One post-plan correctness fix: persisted wake-intent + startup recovery (E2E case 12), see TODO "Verification trail".
+**Decisions made with user:** multiple models may be awake concurrently; router = Python FastAPI; tiered idle (level-1 sleep after 15 min, level-2 offload after 3 hr); depth-aware 503 errors.
+
+## 1. Problem & constraints
+
+Calling services must be able to hit `http://<host>:8000/v1/...` with any of our
+three models at any time, with no knowledge of load state. Hard constraints:
+
+1. **Native to the Docker stack** — the entire request path runs inside
+   containers; no host-side cron, watchers, or helpers.
+2. **No docker permissions on the request path** — model load/unload/switch is
+   pure HTTP (vLLM Sleep Mode). Docker is only needed for rare manual ops
+   (image upgrade, pulling new models).
+3. **Highly reliable** — failures degrade to depth-aware 503 + `Retry-After`,
+   never a hang, never a proxy to a half-awake backend.
+
+The Plan v2 nginx front door hid dev endpoints but did **not** wake models on
+request; that inverted priority is what this plan fixes.
+
+## 2. Hardware reality (re-verified 2026-08-17 — see NOTES-2026-08-17-gpu2-recheck.md)
+
+- **3× A800-SXM4-80GB**. P2P is **still broken host-wide, 4/6 paths corrupt**,
+  but the fault MOVED: now every transfer **sourced from GPU0 or GPU1**
+  corrupts; GPU2-sourced transfers are the only clean ones (was the inverse
+  on 2026-08-13). Per-GPU integrity (H2D/D2H, matmul, stress) passes on all
+  three, twice.
+- Consequences: `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce` stay
+  mandatory for any TP>1 workload (host-staged copies verify clean — which is
+  why TP=2 on GPU0+1 with P2P off works). **GPU2 is usable for single-GPU
+  services** (no P2P involved; H2D/D2H verified clean).
+- On-disk models (verified): `Qwen3.6-35B-A3B-FP8` (43GB dir), `OvisOCR2`
+  (1.8GB), `Qwen3-Embedding-8B` (12GB). **User confirmed 2026-08-14: the 8B
+  embedding model is the intended one** (earlier docs mentioning a 0.6B were
+  wrong; slices are sized for the 8B).
+
+## 3. Architecture
+
+```
+                        ┌──────────────────── Docker network ────────────────────┐
+callers ─► :8000 ─► ┌─────────┐   ┌──────────────────────┐ ┌───────────────────┐
+                    │ router  │ ─►│ vllm-text (TP=2)     │ │ vllm-ocr  GPU2    │
+                    │ FastAPI │   │  GPU0 + GPU1 only    │ │ vllm-embed GPU2   │
+                    └─────────┘   └──────────────────────┘ └───────────────────┘
+           127.0.0.1:8010 (admin)
+```
+
+- **One always-running vLLM service per model**, each with
+  `--enable-sleep-mode` + `VLLM_SERVER_DEV_MODE=1`, each pinned to a GPU
+  memory slice so all three can be awake simultaneously.
+- **Router** is the only public ingress (port 8000). It owns: model-name →
+  service mapping, wake-on-request, streaming proxy, tiered idle management,
+  and dev-endpoint hiding (404 for anything not an allowed path, matched on
+  the **normalized** path to defeat `/v1/../sleep` traversal).
+- **Admin API is a separate listener** published only on `127.0.0.1:8010` —
+  NOT paths on the public port (review C5: a public `/admin/sleep` would be a
+  trivial remote DoS).
+- **nginx is removed.** The router replaces it.
+- **Placement (v3.3, user-approved 2026-08-17):** text TP=2 gets GPU0+GPU1
+  **exclusively** (0.85 slice, full 262K context); OCR and embed run TP=1
+  **on GPU2** — single-GPU services never issue P2P transfers, and GPU2's
+  H2D/D2H paths verify clean (§2). `CUDA_VISIBLE_DEVICES` is set explicitly
+  per service. Rollback if GPU2 ever misbehaves: change one env var to move
+  a service back to GPU0/1 (slices would need re-tightening).
+- All services keep `restart: unless-stopped` and `ipc: host` (NCCL shm).
+  Each service's TP TCP store is isolated by its own network namespace — no
+  distributed-init port collision (would only become a risk if someone later
+  sets `network_mode: host`; do not).
+
+## 4. Port layout
+
+| Port | Binding | Purpose |
+|---|---|---|
+| 8000 | 0.0.0.0 (host) → router | Public API (only public ingress) |
+| 8010 | 127.0.0.1 (host) → router admin listener | `vllmctl` / debugging |
+| 8001 | 127.0.0.1 (host) → vllm-text :8000 | Direct debug access |
+| 8002 | 127.0.0.1 (host) → vllm-ocr :8000 | Direct debug access |
+| 8003 | 127.0.0.1 (host) → vllm-embed :8000 | Direct debug access |
+| — | Docker net only | router → vllm-* |
+
+**Security boundary, stated explicitly:** the loopback debug ports 8001–8003
+expose each service's dev endpoints (`/sleep`, `/wake_up`, …) to **any local
+user** on this multi-user host, bypassing the router's 404s. Same exposure as
+the current stack's port 8001; accepted, but "dev endpoints unreachable"
+means "unreachable from the network," not "unreachable locally."
+
+## 5. GPU memory budget (3× 80GB in use; calibration gates final numbers)
+
+`gpu_memory_utilization` is a fraction of **total** GPU memory, and each
+process accounts only for itself. Budget per GPU:
+
+    Σ(slices on that GPU) + (CUDA ctx per process ~1–2GB) + boot/wake transients ≤ 0.88
+
+Initial slices (recalibrated in Step 1):
+
+| Service | Weights on disk | TP / GPU | util slice | Notes |
+|---|---|---|---|---|
+| vllm-text | 43GB (FP8) | 2 / GPU0+1 (exclusive) | 0.85 | full 262K context; KV ~25GB — no 131K fallback needed |
+| vllm-ocr | 1.8GB | 1 / GPU2 | 0.10 | small vision model |
+| vllm-embed | 12GB (8B) | 1 / GPU2 | 0.25 | `--max-model-len 8192`; raisable at calibration |
+
+Per-GPU: GPU0 = GPU1 = 0.85 + 1 ctx ≈ 0.87; GPU2 = 0.10 + 0.25 = 0.35 + 2 ctx.
+Disjoint pools — text never competes with the small services, and the small
+services have an entire idle GPU of headroom.
+
+Sleeping (level 2) residual is ~2.5GB per **process** (E2E measured ~5GB
+total for one TP=2 instance = 1 proc/GPU). Under v3.3 placement: GPU0/GPU1
+host 1 process each (text) → ~2.5GB residual; GPU2 hosts 2 (ocr + embed)
+→ ~5GB. Measured in Step 1.
+
+## 6. Router design
+
+Python 3.12-slim container, **pinned** fastapi/httpx/uvicorn versions,
+`restart: unless-stopped`. ~300 lines.
+
+**Single process, single event loop, two listening sockets** (public :8000 +
+admin :8010): run two `uvicorn.Server` instances as coroutines on **one**
+asyncio loop (uvicorn can't bind two ports in one worker, and two separate
+processes would break the in-process wake locks — the admin listener MUST
+share the same locks and depth tracking as the request path).
+
+### 6.1 Model registry (env or `models.json`)
+
+```
+text : { service: "vllm-text",  model: "Qwen3.6-35B-A3B-FP8", aliases: ["qwen3.6-35b-a3b-fp8", ...] }
+ocr  : { service: "vllm-ocr",   model: "OvisOCR2" }
+embed: { service: "vllm-embed", model: "Qwen3-Embedding-8B" }
+```
+
+(No 0.6B alias — the 8B is the confirmed model (2026-08-14); an alias would
+mislead clients into thinking they got a different model than they did.)
+
+Matching is case-insensitive. Unknown `model` → OpenAI-style 404
+(`model_not_found`), **no wake triggered**.
+
+### 6.2 Request path
+
+1. Resolve target service:
+   - Content-Type `application/json` → parse body, use `model` field;
+     missing `model` defaults to text **only for** `/v1/chat/completions`
+     and `/v1/completions`; other endpoints require an explicit model.
+   - Non-JSON bodies (e.g. `multipart/form-data` image uploads for OCR —
+     review C2) → **never fully parse**; extract only the small `model`
+     text field from the multipart parts (without reading file parts) and
+     forward the raw buffered body unchanged. A multipart request on
+     `/v1/chat/completions` with no resolvable `model` **defaults to OCR**
+     (that's the image-bearing path).
+   - `/v1/embeddings` → embed service regardless of body.
+2. Check service state (cached briefly — don't add a backend round trip to
+   every request):
+   - **awake** → proxy immediately (httpx streaming passthrough, no buffering).
+   - **sleeping/offloading** → single-flight per service under an asyncio
+     lock; concurrent callers await the same event. Re-check `is_sleeping`
+     after acquiring the lock (another path may have just woken or slept it).
+3. Wake sequence (proven in E2E on 2026-08-14):
+   `POST /wake_up` → `POST /collective_rpc {"method":"reload_weights"}`
+   → `POST /reset_prefix_cache` → poll `GET /health` until 200.
+4. Proxy the held request.
+
+**Proxy timeouts (review M4):** httpx connect timeout 5s; read timeout
+**900s** (matching today's nginx — capped, never disabled, so a wedged
+backend stream can't hang forever) for both normal and streaming paths;
+connection pool sized for ≥16 concurrent requests. A 5s default read timeout
+would kill nearly every generation.
+
+**Other public endpoints:**
+- `/v1/models` → aggregated list of all three registered models (clients
+  commonly list before calling).
+- `/health` → router liveness + per-service state summary (uptime checks).
+- `/metrics` → pass through from all services, or omit; decide at
+  implementation (was unauthenticated under nginx anyway).
+
+### 6.2.1 Depth-aware error semantics
+
+The router tracks each service's sleep depth from its own actions.
+**Depth loss after router restart or direct `vllmctl` admin calls:** if
+sleeping but depth unknown → treat as `offloaded` (conservative — worst case
+the client waits slightly longer than needed, never gets a too-early retry).
+
+Hold-first policy: requests queue during wake and only error if the
+depth-dependent hold deadline is exceeded.
+
+| Situation | Hold deadline | then Status | `Retry-After` | body `sleep_depth` |
+|---|---|---|---|---|
+| Wake from level 1 (sleep, ~1–6s) | 30s | 503 | `10` | `"sleeping"` |
+| Wake from level 2 (offload, ~15–60s+) | 180s | 503 | `60` | `"offloaded"` |
+| Container restarting (cold start = 2–10 min NFS load) | 300s | 503 | `600` | `"restarting"` |
+| Unknown model | — | 404 | — | `model_not_found` |
+
+503 body (OpenAI-style, parseable):
+
+```json
+{"error": {"type": "model_waking", "code": "model_waking",
+           "message": "Model 'OvisOCR2' is waking from offload; retry shortly",
+           "sleep_depth": "offloaded", "estimated_wake_seconds": 45}}
+```
+
+**Wake-sequence failure:** retry the sequence once, then the depth-aware 503.
+Never proxy to a half-awake backend. All deadlines/estimates get re-tuned
+from latencies logged during calibration (Step 1).
+
+### 6.3 Tiered idle management (replaces `vllmctl idle-watch`)
+
+| Tier | Trigger (idle) | Action | State after | Wake cost |
+|---|---|---|---|---|
+| Sleep | 15 min (`IDLE_SLEEP_MIN`) | `POST /sleep?level=1` | weights → host RAM | ~1–6s |
+| Offload | 3 hr (`IDLE_OFFLOAD_MIN`) | `POST /sleep?level=2` | RAM freed; ~5GB/GPU ctx | ~15–60s (NFS) |
+
+Race-safety (review M1): the idle manager takes the **same per-service
+asyncio lock** as the wake path and re-checks under the lock that
+(a) active-request count == 0 and (b) last-activity is still past the
+threshold. Last-activity is refreshed at request **completion** as well as
+arrival, and an active-request counter guards long streaming generations —
+a 20-minute stream must not trigger sleep mid-flight.
+
+**Verify at implementation:** `/sleep?level=2` on a service already sleeping
+at level 1 must discard the offloaded weights (not no-op/error). Fallback:
+wake → re-sleep at level 2.
+
+**Verify at implementation:** host RAM — level 1 holds a full weights copy
+(text 43GB + ocr + embed ≈ 57GB total if all three nap simultaneously).
+Check `free -g` headroom.
+
+**NFS risk (accepted, documented):** level-2 wake requires a healthy NFS
+mount. If NFS is down, the model is unwakeable and the router 503s (with
+`restarting`-grade Retry-After) until NFS returns. Mitigation option: cap the
+text model at level 1 (weights stay in RAM). Decide after calibration.
+
+### 6.4 Router restart mid-wake (review M3)
+
+The wake lock is in-process and dies with the router. The new instance must
+not fire a second concurrent wake at a backend already waking. Mitigation:
+the wake sequence is effectively idempotent from the router's view (re-POST
+`/wake_up` on an already-waking backend is safe; vLLM serializes it), and the
+request path re-checks `is_sleeping` under the lock before acting.
+
+### 6.5 Admin API — separate listener, `127.0.0.1:8010` only
+
+- `GET  /admin/status` — per-service: running?, sleeping?, depth, last activity, wake in progress
+- `POST /admin/wake/{key}` / `POST /admin/sleep/{key}[?level=1|2]` — manual control (updates the router's depth tracking)
+
+### 6.6 Failure modes
+
+| Failure | Router behavior |
+|---|---|
+| vLLM container down/crashing | Detect unhealthy; hold up to 300s (`restart: unless-stopped` recovers it — but note cold start is 2–10 min for text); then 503 `Retry-After: 600` |
+| Wake sequence fails | Retry once, then depth-aware 503 (6.2.1); never proxy half-awake |
+| Router crash | `restart: unless-stopped`; vLLM services unaffected; at most in-flight requests lost; depth resets to conservative `offloaded` |
+| Host reboot | **All three vLLM containers cold-start AWAKE** (vLLM has no boot-asleep) — see §7; text on GPU0/1 and small services on GPU2 initialize in parallel without racing; recovery takes minutes, router serves `restarting` 503s meanwhile |
+| NFS outage | Level-2 unwakeable → 503s until NFS returns (§6.3) |
+| Request without `model` field | Defaults to text **only** on chat/completions; elsewhere 400 |
+
+## 7. Compose changes
+
+- Add `router` service (build `./router/`): public `8000:8000`, admin
+  `127.0.0.1:8010:8010`, `depends_on` all vllm services
+  (`condition: service_started`).
+- **Boot ordering (simplified in v3.3):** no staggering needed — text
+  initializes on GPU0/1 while OCR and embed initialize on GPU2, disjoint
+  memory pools, no race. All services use plain `depends_on:
+  service_started` (sleep makes healthchecks flappy, so never
+  `service_healthy`). The earlier wait-for-healthy entrypoint choreography
+  (review-2 issue #1) is obsolete under this placement.
+- Split `vllm` into `vllm-text` / `vllm-ocr` / `vllm-embed` with per-model
+  args baked in; `MODEL_NAME`/`EXTRA_ARGS`/`TP_SIZE`/`MAX_MODEL_LEN` in
+  `.env` become obsolete for serving (kept only for rollback).
+- Remove `nginx` service and `nginx.conf`.
+- All services: `restart: unless-stopped`, `ipc: host`,
+  `NCCL_P2P_DISABLE=1`, explicit `CUDA_VISIBLE_DEVICES`.
+
+Per-model serving args (all include `--disable-custom-all-reduce
+--enable-sleep-mode`, **explicit `--max-model-len` each** — review M2):
+
+- **text:** `--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.85`
+- **ocr:** `--max-model-len 32768 --gpu-memory-utilization 0.10` (+ vision args from OvisOCR2 model card, e.g. `--limit-mm-per-prompt`)
+- **embed:** `--max-model-len 8192 --gpu-memory-utilization 0.25`
+
+## 8. `vllmctl` changes
+
+- `up [MODEL]` / `down [MODEL]` → HTTP to `127.0.0.1:8010` admin API (no
+  docker needed for routine control).
+- `status` → `GET 127.0.0.1:8010/admin/status` pretty-printed.
+- `pull` unchanged (docker, rare, manual).
+- `idle-watch` removed (router owns idle management).
+
+## 9. Implementation order
+
+0. **Backups + cutover plan (review M5):**
+   `cp compose.yml compose.yml.v2.bak; cp vllmctl vllmctl.v2.bak`.
+   **Status 2026-08-14: `renbaibing` is now in the docker group** (works
+   directly in fresh login shells; use `sg docker -c "…"` in sessions
+   started before the change — `.runas.py`/`.user.env` are obsolete).
+   Old stack already brought down by user; GPUs idle. Rollback: restore
+   `.v2.bak` files, `docker compose up -d`.
+1. **Calibration (gates everything):** bring up the three services with
+   initial slices (old stack stopped); measure awake footprints, sleeping
+   residuals, level-1 and level-2 wake latencies, host RAM cost; adjust
+   slices so each GPU ≤ 0.88 including contexts. Log latencies → tune the
+   §6.2.1 deadlines from data.
+2. `router/` app (registry, wake single-flight, raw-body-safe routing,
+   streaming proxy with proper timeouts, tiered idle with lock, admin
+   listener) + compose rewiring; nginx removed.
+3. `vllmctl` rewrite of up/down/status.
+4. E2E verification (below).
+5. Docs: README, TODO, memory.
+
+## 10. E2E test matrix
+
+1. All asleep → `POST /v1/chat/completions` (text) → 200 after wake; latency logged.
+2. While text awake → `POST /v1/embeddings` → embed wakes independently; both serve.
+3. OCR request with **`multipart/form-data`** body (not just JSON base64) → routes correctly, no body parsing error.
+4. Idle tiers (shortened timers): 1 min → level-1 sleep (host RAM grows by weights); 3 min → level-2 offload (RAM freed); auto-wake from both.
+5. Streaming request during wake → first chunk after wake, no buffering.
+6. Unknown model name → 404 `model_not_found`, no wake triggered.
+7. Dev endpoints (`/sleep`, `/wake_up`, `/collective_rpc`, `/reset_prefix_cache`) **and** `/admin/*` on public :8000 → 404; admin reachable only via 127.0.0.1:8010.
+8. Kill vllm-text container → request → `restarting` 503 (`Retry-After: 600`) during recovery → 200 after restart completes.
+9. Concurrent 10× requests while sleeping → exactly one wake sequence (router logs).
+10. Depth-aware errors (hold deadline forced to 1s): level-1 wake → 503 `Retry-After: 10`, `"sleeping"`; level-2 → `Retry-After: 60`, `"offloaded"`; body parses with `estimated_wake_seconds`.
+11. Idle-race: fire a request at the moment the idle timer expires → request served, no sleep mid-request (lock + active-counter verified).
+12. Router restart mid-wake → no double-wake crash; request eventually served.
+13. Two services waking concurrently (text + embed requested simultaneously) → both complete, no lock cross-talk.
+14. Path traversal: `/v1/../sleep`, `/v1%2f..%2fsleep` → 404.
+15. Host reboot (if feasible to test): parallel boot (text on GPU0/1, small services on GPU2 — no race), router serves `restarting` 503s, all three become available.
+
+## 11. Risks / open items
+
+- **GPU slice calibration** gates final numbers (§5, Step 1). v3.3 placement
+  (text exclusive on GPU0/1 at 0.85) removes the KV-starvation concern;
+  calibration confirms.
+- **GPU2 trust:** single-GPU tests pass (2026-08-17, twice), but it has a
+  fault history (Aug 11 driver re-probe). Mitigation: E2E test 3+4 exercise
+  it before rollout; if GPU2 degrades in production, move OCR/embed back to
+  GPU0/1 with one env-var change each (slices re-tighten to v3.2 values).
+- ~~Which embedding model~~ — **resolved 2026-08-14: Qwen3-Embedding-8B.**
+- **Level 1→2 escalation** and **host RAM** (~57GB if all three nap) need verification (§6.3).
+- **NFS outage strands level-2 models** — accepted risk or cap text at level 1 (§6.3).
+- Wake latency must fit calling services' client timeouts — confirm their settings before rollout.