vllm-frontdoor/.claude/memory/router-front-door-plan.md

Plan v3.3 — Router Front Door (auto wake-on-request)

Date: 2026-08-14, revised 2026-08-17 (v3.3) Status: IMPLEMENTED & VERIFIED 2026-08-17 — v3.2 passed two review rounds (5 critical + 6 major fixed and confirmed; round-2 minors applied). v3.3 = hardware-driven placement change (GPU2 cleared for single-GPU services → OCR+embed on GPU2, text exclusive on GPU0/1). Implementation record and verification trail: ../../TODO.md; measured numbers: ../../CALIBRATION.md. One post-plan correctness fix: persisted wake-intent + startup recovery (E2E case 12), see TODO "Verification trail". Decisions made with user: multiple models may be awake concurrently; router = Python FastAPI; tiered idle (level-1 sleep after 15 min, level-2 offload after 3 hr); depth-aware 503 errors.

1. Problem & constraints

Calling services must be able to hit http://<host>:8000/v1/... with any of our three models at any time, with no knowledge of load state. Hard constraints:

1. Native to the Docker stack — the entire request path runs inside containers; no host-side cron, watchers, or helpers.
2. No docker permissions on the request path — model load/unload/switch is pure HTTP (vLLM Sleep Mode). Docker is only needed for rare manual ops (image upgrade, pulling new models).
3. Highly reliable — failures degrade to depth-aware 503 + Retry-After, never a hang, never a proxy to a half-awake backend.

The Plan v2 nginx front door hid dev endpoints but did not wake models on request; that inverted priority is what this plan fixes.

2. Hardware reality (re-verified 2026-08-17 — see NOTES-2026-08-17-gpu2-recheck.md)

• 3× A800-SXM4-80GB. P2P is still broken host-wide, 4/6 paths corrupt, but the fault MOVED: now every transfer sourced from GPU0 or GPU1 corrupts; GPU2-sourced transfers are the only clean ones (was the inverse on 2026-08-13). Per-GPU integrity (H2D/D2H, matmul, stress) passes on all three, twice.
• Consequences: NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce stay mandatory for any TP>1 workload (host-staged copies verify clean — which is why TP=2 on GPU0+1 with P2P off works). GPU2 is usable for single-GPU services (no P2P involved; H2D/D2H verified clean).
• On-disk models (verified): Qwen3.6-35B-A3B-FP8 (43GB dir), OvisOCR2 (1.8GB), Qwen3-Embedding-8B (12GB). User confirmed 2026-08-14: the 8B embedding model is the intended one (earlier docs mentioning a 0.6B were wrong; slices are sized for the 8B).

3. Architecture

                        ┌──────────────────── Docker network ────────────────────┐
callers ─► :8000 ─► ┌─────────┐   ┌──────────────────────┐ ┌───────────────────┐
                    │ router  │ ─►│ vllm-text (TP=2)     │ │ vllm-ocr  GPU2    │
                    │ FastAPI │   │  GPU0 + GPU1 only    │ │ vllm-embed GPU2   │
                    └─────────┘   └──────────────────────┘ └───────────────────┘
           127.0.0.1:8010 (admin)

• One always-running vLLM service per model, each with --enable-sleep-mode + VLLM_SERVER_DEV_MODE=1, each pinned to a GPU memory slice so all three can be awake simultaneously.
• Router is the only public ingress (port 8000). It owns: model-name → service mapping, wake-on-request, streaming proxy, tiered idle management, and dev-endpoint hiding (404 for anything not an allowed path, matched on the normalized path to defeat /v1/../sleep traversal).
• Admin API is a separate listener published only on 127.0.0.1:8010 — NOT paths on the public port (review C5: a public /admin/sleep would be a trivial remote DoS).
• nginx is removed. The router replaces it.
• Placement (v3.3, user-approved 2026-08-17): text TP=2 gets GPU0+GPU1 exclusively (0.85 slice, full 262K context); OCR and embed run TP=1 on GPU2 — single-GPU services never issue P2P transfers, and GPU2's H2D/D2H paths verify clean (§2). CUDA_VISIBLE_DEVICES is set explicitly per service. Rollback if GPU2 ever misbehaves: change one env var to move a service back to GPU0/1 (slices would need re-tightening).
• All services keep restart: unless-stopped and ipc: host (NCCL shm). Each service's TP TCP store is isolated by its own network namespace — no distributed-init port collision (would only become a risk if someone later sets network_mode: host; do not).

4. Port layout

Port	Binding	Purpose
8000	0.0.0.0 (host) → router	Public API (only public ingress)
8010	127.0.0.1 (host) → router admin listener	vllmctl / debugging
8001	127.0.0.1 (host) → vllm-text :8000	Direct debug access
8002	127.0.0.1 (host) → vllm-ocr :8000	Direct debug access
8003	127.0.0.1 (host) → vllm-embed :8000	Direct debug access
—	Docker net only	router → vllm-*

Security boundary, stated explicitly: the loopback debug ports 8001–8003 expose each service's dev endpoints (/sleep, /wake_up, …) to any local user on this multi-user host, bypassing the router's 404s. Same exposure as the current stack's port 8001; accepted, but "dev endpoints unreachable" means "unreachable from the network," not "unreachable locally."

5. GPU memory budget (3× 80GB in use; calibration gates final numbers)

gpu_memory_utilization is a fraction of total GPU memory, and each process accounts only for itself. Budget per GPU:

Σ(slices on that GPU) + (CUDA ctx per process ~1–2GB) + boot/wake transients ≤ 0.88

Initial slices (recalibrated in Step 1):

Service	Weights on disk	TP / GPU	util slice	Notes
vllm-text	43GB (FP8)	2 / GPU0+1 (exclusive)	0.85	full 262K context; KV ~25GB — no 131K fallback needed
vllm-ocr	1.8GB	1 / GPU2	0.10	small vision model
vllm-embed	12GB (8B)	1 / GPU2	0.25	--max-model-len 8192; raisable at calibration

Per-GPU: GPU0 = GPU1 = 0.85 + 1 ctx ≈ 0.87; GPU2 = 0.10 + 0.25 = 0.35 + 2 ctx. Disjoint pools — text never competes with the small services, and the small services have an entire idle GPU of headroom.

Sleeping (level 2) residual is ~2.5GB per process (E2E measured ~5GB total for one TP=2 instance = 1 proc/GPU). Under v3.3 placement: GPU0/GPU1 host 1 process each (text) → ~2.5GB residual; GPU2 hosts 2 (ocr + embed) → ~5GB. Measured in Step 1.

6. Router design

Python 3.12-slim container, pinned fastapi/httpx/uvicorn versions, restart: unless-stopped. ~300 lines.

Single process, single event loop, two listening sockets (public :8000 + admin :8010): run two uvicorn.Server instances as coroutines on one asyncio loop (uvicorn can't bind two ports in one worker, and two separate processes would break the in-process wake locks — the admin listener MUST share the same locks and depth tracking as the request path).

6.1 Model registry (env or models.json)

text : { service: "vllm-text",  model: "Qwen3.6-35B-A3B-FP8", aliases: ["qwen3.6-35b-a3b-fp8", ...] }
ocr  : { service: "vllm-ocr",   model: "OvisOCR2" }
embed: { service: "vllm-embed", model: "Qwen3-Embedding-8B" }

(No 0.6B alias — the 8B is the confirmed model (2026-08-14); an alias would mislead clients into thinking they got a different model than they did.)

Matching is case-insensitive. Unknown model → OpenAI-style 404 (model_not_found), no wake triggered.

6.2 Request path

1. Resolve target service:
   • Content-Type application/json → parse body, use model field; missing model defaults to text only for /v1/chat/completions and /v1/completions; other endpoints require an explicit model.
   • Non-JSON bodies (e.g. multipart/form-data image uploads for OCR — review C2) → never fully parse; extract only the small model text field from the multipart parts (without reading file parts) and forward the raw buffered body unchanged. A multipart request on /v1/chat/completions with no resolvable model defaults to OCR (that's the image-bearing path).
   • /v1/embeddings → embed service regardless of body.
2. Check service state (cached briefly — don't add a backend round trip to every request):
   • awake → proxy immediately (httpx streaming passthrough, no buffering).
   • sleeping/offloading → single-flight per service under an asyncio lock; concurrent callers await the same event. Re-check is_sleeping after acquiring the lock (another path may have just woken or slept it).
3. Wake sequence (proven in E2E on 2026-08-14): POST /wake_up → POST /collective_rpc {"method":"reload_weights"} → POST /reset_prefix_cache → poll GET /health until 200.
4. Proxy the held request.

Proxy timeouts (review M4): httpx connect timeout 5s; read timeout 900s (matching today's nginx — capped, never disabled, so a wedged backend stream can't hang forever) for both normal and streaming paths; connection pool sized for ≥16 concurrent requests. A 5s default read timeout would kill nearly every generation.

Other public endpoints:

• /v1/models → aggregated list of all three registered models (clients commonly list before calling).
• /health → router liveness + per-service state summary (uptime checks).
• /metrics → pass through from all services, or omit; decide at implementation (was unauthenticated under nginx anyway).

6.2.1 Depth-aware error semantics

The router tracks each service's sleep depth from its own actions. Depth loss after router restart or direct vllmctl admin calls: if sleeping but depth unknown → treat as offloaded (conservative — worst case the client waits slightly longer than needed, never gets a too-early retry).

Hold-first policy: requests queue during wake and only error if the depth-dependent hold deadline is exceeded.

Situation	Hold deadline	then Status	Retry-After	body sleep_depth
Wake from level 1 (sleep, ~1–6s)	30s	503	10	"sleeping"
Wake from level 2 (offload, ~15–60s+)	180s	503	60	"offloaded"
Container restarting (cold start = 2–10 min NFS load)	300s	503	600	"restarting"
Unknown model	—	404	—	model_not_found

503 body (OpenAI-style, parseable):

{"error": {"type": "model_waking", "code": "model_waking",
           "message": "Model 'OvisOCR2' is waking from offload; retry shortly",
           "sleep_depth": "offloaded", "estimated_wake_seconds": 45}}

Wake-sequence failure: retry the sequence once, then the depth-aware 503. Never proxy to a half-awake backend. All deadlines/estimates get re-tuned from latencies logged during calibration (Step 1).

6.3 Tiered idle management (replaces vllmctl idle-watch)

Tier	Trigger (idle)	Action	State after	Wake cost
Sleep	15 min (IDLE_SLEEP_MIN)	POST /sleep?level=1	weights → host RAM	~1–6s
Offload	3 hr (IDLE_OFFLOAD_MIN)	POST /sleep?level=2	RAM freed; ~5GB/GPU ctx	~15–60s (NFS)

Race-safety (review M1): the idle manager takes the same per-service asyncio lock as the wake path and re-checks under the lock that (a) active-request count == 0 and (b) last-activity is still past the threshold. Last-activity is refreshed at request completion as well as arrival, and an active-request counter guards long streaming generations — a 20-minute stream must not trigger sleep mid-flight.

Verify at implementation: /sleep?level=2 on a service already sleeping at level 1 must discard the offloaded weights (not no-op/error). Fallback: wake → re-sleep at level 2.

Verify at implementation: host RAM — level 1 holds a full weights copy (text 43GB + ocr + embed ≈ 57GB total if all three nap simultaneously). Check free -g headroom.

NFS risk (accepted, documented): level-2 wake requires a healthy NFS mount. If NFS is down, the model is unwakeable and the router 503s (with restarting-grade Retry-After) until NFS returns. Mitigation option: cap the text model at level 1 (weights stay in RAM). Decide after calibration.

6.4 Router restart mid-wake (review M3)

The wake lock is in-process and dies with the router. The new instance must not fire a second concurrent wake at a backend already waking. Mitigation: the wake sequence is effectively idempotent from the router's view (re-POST /wake_up on an already-waking backend is safe; vLLM serializes it), and the request path re-checks is_sleeping under the lock before acting.

6.5 Admin API — separate listener, 127.0.0.1:8010 only

• GET /admin/status — per-service: running?, sleeping?, depth, last activity, wake in progress
• POST /admin/wake/{key} / POST /admin/sleep/{key}[?level=1|2] — manual control (updates the router's depth tracking)

6.6 Failure modes

Failure	Router behavior
vLLM container down/crashing	Detect unhealthy; hold up to 300s (restart: unless-stopped recovers it — but note cold start is 2–10 min for text); then 503 Retry-After: 600
Wake sequence fails	Retry once, then depth-aware 503 (6.2.1); never proxy half-awake
Router crash	restart: unless-stopped; vLLM services unaffected; at most in-flight requests lost; depth resets to conservative offloaded
Host reboot	All three vLLM containers cold-start AWAKE (vLLM has no boot-asleep) — see §7; text on GPU0/1 and small services on GPU2 initialize in parallel without racing; recovery takes minutes, router serves restarting 503s meanwhile
NFS outage	Level-2 unwakeable → 503s until NFS returns (§6.3)
Request without model field	Defaults to text only on chat/completions; elsewhere 400

7. Compose changes

• Add router service (build ./router/): public 8000:8000, admin 127.0.0.1:8010:8010, depends_on all vllm services (condition: service_started).
• Boot ordering (simplified in v3.3): no staggering needed — text initializes on GPU0/1 while OCR and embed initialize on GPU2, disjoint memory pools, no race. All services use plain depends_on: service_started (sleep makes healthchecks flappy, so never service_healthy). The earlier wait-for-healthy entrypoint choreography (review-2 issue #1) is obsolete under this placement.
• Split vllm into vllm-text / vllm-ocr / vllm-embed with per-model args baked in; MODEL_NAME/EXTRA_ARGS/TP_SIZE/MAX_MODEL_LEN in .env become obsolete for serving (kept only for rollback).
• Remove nginx service and nginx.conf.
• All services: restart: unless-stopped, ipc: host, NCCL_P2P_DISABLE=1, explicit CUDA_VISIBLE_DEVICES.

Per-model serving args (all include --disable-custom-all-reduce --enable-sleep-mode, explicit --max-model-len each — review M2):

• text: --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.85
• ocr: --max-model-len 32768 --gpu-memory-utilization 0.10 (+ vision args from OvisOCR2 model card, e.g. --limit-mm-per-prompt)
• embed: --max-model-len 8192 --gpu-memory-utilization 0.25

8. vllmctl changes

• up [MODEL] / down [MODEL] → HTTP to 127.0.0.1:8010 admin API (no docker needed for routine control).
• status → GET 127.0.0.1:8010/admin/status pretty-printed.
• pull unchanged (docker, rare, manual).
• idle-watch removed (router owns idle management).

9. Implementation order

0. Backups + cutover plan (review M5): cp compose.yml compose.yml.v2.bak; cp vllmctl vllmctl.v2.bak. Status 2026-08-14: renbaibing is now in the docker group (works directly in fresh login shells; use sg docker -c "…" in sessions started before the change — .runas.py/.user.env are obsolete). Old stack already brought down by user; GPUs idle. Rollback: restore .v2.bak files, docker compose up -d.
1. Calibration (gates everything): bring up the three services with initial slices (old stack stopped); measure awake footprints, sleeping residuals, level-1 and level-2 wake latencies, host RAM cost; adjust slices so each GPU ≤ 0.88 including contexts. Log latencies → tune the §6.2.1 deadlines from data.
2. router/ app (registry, wake single-flight, raw-body-safe routing, streaming proxy with proper timeouts, tiered idle with lock, admin listener) + compose rewiring; nginx removed.
3. vllmctl rewrite of up/down/status.
4. E2E verification (below).
5. Docs: README, TODO, memory.

10. E2E test matrix

1. All asleep → POST /v1/chat/completions (text) → 200 after wake; latency logged.
2. While text awake → POST /v1/embeddings → embed wakes independently; both serve.
3. OCR request with multipart/form-data body (not just JSON base64) → routes correctly, no body parsing error.
4. Idle tiers (shortened timers): 1 min → level-1 sleep (host RAM grows by weights); 3 min → level-2 offload (RAM freed); auto-wake from both.
5. Streaming request during wake → first chunk after wake, no buffering.
6. Unknown model name → 404 model_not_found, no wake triggered.
7. Dev endpoints (/sleep, /wake_up, /collective_rpc, /reset_prefix_cache) and /admin/* on public :8000 → 404; admin reachable only via 127.0.0.1:8010.
8. Kill vllm-text container → request → restarting 503 (Retry-After: 600) during recovery → 200 after restart completes.
9. Concurrent 10× requests while sleeping → exactly one wake sequence (router logs).
10. Depth-aware errors (hold deadline forced to 1s): level-1 wake → 503 Retry-After: 10, "sleeping"; level-2 → Retry-After: 60, "offloaded"; body parses with estimated_wake_seconds.
11. Idle-race: fire a request at the moment the idle timer expires → request served, no sleep mid-request (lock + active-counter verified).
12. Router restart mid-wake → no double-wake crash; request eventually served.
13. Two services waking concurrently (text + embed requested simultaneously) → both complete, no lock cross-talk.
14. Path traversal: /v1/../sleep, /v1%2f..%2fsleep → 404.
15. Host reboot (if feasible to test): parallel boot (text on GPU0/1, small services on GPU2 — no race), router serves restarting 503s, all three become available.

11. Risks / open items

• GPU slice calibration gates final numbers (§5, Step 1). v3.3 placement (text exclusive on GPU0/1 at 0.85) removes the KV-starvation concern; calibration confirms.
• GPU2 trust: single-GPU tests pass (2026-08-17, twice), but it has a fault history (Aug 11 driver re-probe). Mitigation: E2E test 3+4 exercise it before rollout; if GPU2 degrades in production, move OCR/embed back to GPU0/1 with one env-var change each (slices re-tighten to v3.2 values).
• Which embedding model — resolved 2026-08-14: Qwen3-Embedding-8B.
• Level 1→2 escalation and host RAM (~57GB if all three nap) need verification (§6.3).
• NFS outage strands level-2 models — accepted risk or cap text at level 1 (§6.3).
• Wake latency must fit calling services' client timeouts — confirm their settings before rollout.