vllm-frontdoor/.claude/memory/router-front-door-plan.md

# Plan v3.3 — Router Front Door (auto wake-on-request)

**Date:** 2026-08-14, revised 2026-08-17 (v3.3)
**Status:** **IMPLEMENTED & VERIFIED 2026-08-17** — v3.2 passed two review rounds (5 critical + 6 major fixed and confirmed; round-2 minors applied). v3.3 = hardware-driven placement change (GPU2 cleared for single-GPU services → OCR+embed on GPU2, text exclusive on GPU0/1). Implementation record and verification trail: `../../TODO.md`; measured numbers: `../../CALIBRATION.md`. One post-plan correctness fix: persisted wake-intent + startup recovery (E2E case 12), see TODO "Verification trail".
**Decisions made with user:** multiple models may be awake concurrently; router = Python FastAPI; tiered idle (level-1 sleep after 15 min, level-2 offload after 3 hr); depth-aware 503 errors.

## 1. Problem & constraints

Calling services must be able to hit `http://<host>:8000/v1/...` with any of our
three models at any time, with no knowledge of load state. Hard constraints:

1. **Native to the Docker stack** — the entire request path runs inside
   containers; no host-side cron, watchers, or helpers.
2. **No docker permissions on the request path** — model load/unload/switch is
   pure HTTP (vLLM Sleep Mode). Docker is only needed for rare manual ops
   (image upgrade, pulling new models).
3. **Highly reliable** — failures degrade to depth-aware 503 + `Retry-After`,
   never a hang, never a proxy to a half-awake backend.

The Plan v2 nginx front door hid dev endpoints but did **not** wake models on
request; that inverted priority is what this plan fixes.

## 2. Hardware reality (re-verified 2026-08-17 — see NOTES-2026-08-17-gpu2-recheck.md)

- **3× A800-SXM4-80GB**. P2P is **still broken host-wide, 4/6 paths corrupt**,
  but the fault MOVED: now every transfer **sourced from GPU0 or GPU1**
  corrupts; GPU2-sourced transfers are the only clean ones (was the inverse
  on 2026-08-13). Per-GPU integrity (H2D/D2H, matmul, stress) passes on all
  three, twice.
- Consequences: `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce` stay
  mandatory for any TP>1 workload (host-staged copies verify clean — which is
  why TP=2 on GPU0+1 with P2P off works). **GPU2 is usable for single-GPU
  services** (no P2P involved; H2D/D2H verified clean).
- On-disk models (verified): `Qwen3.6-35B-A3B-FP8` (43GB dir), `OvisOCR2`
  (1.8GB), `Qwen3-Embedding-8B` (12GB). **User confirmed 2026-08-14: the 8B
  embedding model is the intended one** (earlier docs mentioning a 0.6B were
  wrong; slices are sized for the 8B).

## 3. Architecture

```
                        ┌──────────────────── Docker network ────────────────────┐
callers ─► :8000 ─► ┌─────────┐   ┌──────────────────────┐ ┌───────────────────┐
                    │ router  │ ─►│ vllm-text (TP=2)     │ │ vllm-ocr  GPU2    │
                    │ FastAPI │   │  GPU0 + GPU1 only    │ │ vllm-embed GPU2   │
                    └─────────┘   └──────────────────────┘ └───────────────────┘
           127.0.0.1:8010 (admin)
```

- **One always-running vLLM service per model**, each with
  `--enable-sleep-mode` + `VLLM_SERVER_DEV_MODE=1`, each pinned to a GPU
  memory slice so all three can be awake simultaneously.
- **Router** is the only public ingress (port 8000). It owns: model-name →
  service mapping, wake-on-request, streaming proxy, tiered idle management,
  and dev-endpoint hiding (404 for anything not an allowed path, matched on
  the **normalized** path to defeat `/v1/../sleep` traversal).
- **Admin API is a separate listener** published only on `127.0.0.1:8010` —
  NOT paths on the public port (review C5: a public `/admin/sleep` would be a
  trivial remote DoS).
- **nginx is removed.** The router replaces it.
- **Placement (v3.3, user-approved 2026-08-17):** text TP=2 gets GPU0+GPU1
  **exclusively** (0.85 slice, full 262K context); OCR and embed run TP=1
  **on GPU2** — single-GPU services never issue P2P transfers, and GPU2's
  H2D/D2H paths verify clean (§2). `CUDA_VISIBLE_DEVICES` is set explicitly
  per service. Rollback if GPU2 ever misbehaves: change one env var to move
  a service back to GPU0/1 (slices would need re-tightening).
- All services keep `restart: unless-stopped` and `ipc: host` (NCCL shm).
  Each service's TP TCP store is isolated by its own network namespace — no
  distributed-init port collision (would only become a risk if someone later
  sets `network_mode: host`; do not).

## 4. Port layout

| Port | Binding | Purpose |
|---|---|---|
| 8000 | 0.0.0.0 (host) → router | Public API (only public ingress) |
| 8010 | 127.0.0.1 (host) → router admin listener | `vllmctl` / debugging |
| 8001 | 127.0.0.1 (host) → vllm-text :8000 | Direct debug access |
| 8002 | 127.0.0.1 (host) → vllm-ocr :8000 | Direct debug access |
| 8003 | 127.0.0.1 (host) → vllm-embed :8000 | Direct debug access |
| — | Docker net only | router → vllm-* |

**Security boundary, stated explicitly:** the loopback debug ports 8001–8003
expose each service's dev endpoints (`/sleep`, `/wake_up`, …) to **any local
user** on this multi-user host, bypassing the router's 404s. Same exposure as
the current stack's port 8001; accepted, but "dev endpoints unreachable"
means "unreachable from the network," not "unreachable locally."

## 5. GPU memory budget (3× 80GB in use; calibration gates final numbers)

`gpu_memory_utilization` is a fraction of **total** GPU memory, and each
process accounts only for itself. Budget per GPU:

    Σ(slices on that GPU) + (CUDA ctx per process ~1–2GB) + boot/wake transients ≤ 0.88

Initial slices (recalibrated in Step 1):

| Service | Weights on disk | TP / GPU | util slice | Notes |
|---|---|---|---|---|
| vllm-text | 43GB (FP8) | 2 / GPU0+1 (exclusive) | 0.85 | full 262K context; KV ~25GB — no 131K fallback needed |
| vllm-ocr | 1.8GB | 1 / GPU2 | 0.10 | small vision model |
| vllm-embed | 12GB (8B) | 1 / GPU2 | 0.25 | `--max-model-len 8192`; raisable at calibration |

Per-GPU: GPU0 = GPU1 = 0.85 + 1 ctx ≈ 0.87; GPU2 = 0.10 + 0.25 = 0.35 + 2 ctx.
Disjoint pools — text never competes with the small services, and the small
services have an entire idle GPU of headroom.

Sleeping (level 2) residual is ~2.5GB per **process** (E2E measured ~5GB
total for one TP=2 instance = 1 proc/GPU). Under v3.3 placement: GPU0/GPU1
host 1 process each (text) → ~2.5GB residual; GPU2 hosts 2 (ocr + embed)
→ ~5GB. Measured in Step 1.

## 6. Router design

Python 3.12-slim container, **pinned** fastapi/httpx/uvicorn versions,
`restart: unless-stopped`. ~300 lines.

**Single process, single event loop, two listening sockets** (public :8000 +
admin :8010): run two `uvicorn.Server` instances as coroutines on **one**
asyncio loop (uvicorn can't bind two ports in one worker, and two separate
processes would break the in-process wake locks — the admin listener MUST
share the same locks and depth tracking as the request path).

### 6.1 Model registry (env or `models.json`)

```
text : { service: "vllm-text",  model: "Qwen3.6-35B-A3B-FP8", aliases: ["qwen3.6-35b-a3b-fp8", ...] }
ocr  : { service: "vllm-ocr",   model: "OvisOCR2" }
embed: { service: "vllm-embed", model: "Qwen3-Embedding-8B" }
```

(No 0.6B alias — the 8B is the confirmed model (2026-08-14); an alias would
mislead clients into thinking they got a different model than they did.)

Matching is case-insensitive. Unknown `model` → OpenAI-style 404
(`model_not_found`), **no wake triggered**.

### 6.2 Request path

1. Resolve target service:
   - Content-Type `application/json` → parse body, use `model` field;
     missing `model` defaults to text **only for** `/v1/chat/completions`
     and `/v1/completions`; other endpoints require an explicit model.
   - Non-JSON bodies (e.g. `multipart/form-data` image uploads for OCR —
     review C2) → **never fully parse**; extract only the small `model`
     text field from the multipart parts (without reading file parts) and
     forward the raw buffered body unchanged. A multipart request on
     `/v1/chat/completions` with no resolvable `model` **defaults to OCR**
     (that's the image-bearing path).
   - `/v1/embeddings` → embed service regardless of body.
2. Check service state (cached briefly — don't add a backend round trip to
   every request):
   - **awake** → proxy immediately (httpx streaming passthrough, no buffering).
   - **sleeping/offloading** → single-flight per service under an asyncio
     lock; concurrent callers await the same event. Re-check `is_sleeping`
     after acquiring the lock (another path may have just woken or slept it).
3. Wake sequence (proven in E2E on 2026-08-14):
   `POST /wake_up` → `POST /collective_rpc {"method":"reload_weights"}`
   → `POST /reset_prefix_cache` → poll `GET /health` until 200.
4. Proxy the held request.

**Proxy timeouts (review M4):** httpx connect timeout 5s; read timeout
**900s** (matching today's nginx — capped, never disabled, so a wedged
backend stream can't hang forever) for both normal and streaming paths;
connection pool sized for ≥16 concurrent requests. A 5s default read timeout
would kill nearly every generation.

**Other public endpoints:**
- `/v1/models` → aggregated list of all three registered models (clients
  commonly list before calling).
- `/health` → router liveness + per-service state summary (uptime checks).
- `/metrics` → pass through from all services, or omit; decide at
  implementation (was unauthenticated under nginx anyway).

### 6.2.1 Depth-aware error semantics

The router tracks each service's sleep depth from its own actions.
**Depth loss after router restart or direct `vllmctl` admin calls:** if
sleeping but depth unknown → treat as `offloaded` (conservative — worst case
the client waits slightly longer than needed, never gets a too-early retry).

Hold-first policy: requests queue during wake and only error if the
depth-dependent hold deadline is exceeded.

| Situation | Hold deadline | then Status | `Retry-After` | body `sleep_depth` |
|---|---|---|---|---|
| Wake from level 1 (sleep, ~1–6s) | 30s | 503 | `10` | `"sleeping"` |
| Wake from level 2 (offload, ~15–60s+) | 180s | 503 | `60` | `"offloaded"` |
| Container restarting (cold start = 2–10 min NFS load) | 300s | 503 | `600` | `"restarting"` |
| Unknown model | — | 404 | — | `model_not_found` |

503 body (OpenAI-style, parseable):

```json
{"error": {"type": "model_waking", "code": "model_waking",
           "message": "Model 'OvisOCR2' is waking from offload; retry shortly",
           "sleep_depth": "offloaded", "estimated_wake_seconds": 45}}
```

**Wake-sequence failure:** retry the sequence once, then the depth-aware 503.
Never proxy to a half-awake backend. All deadlines/estimates get re-tuned
from latencies logged during calibration (Step 1).

### 6.3 Tiered idle management (replaces `vllmctl idle-watch`)

| Tier | Trigger (idle) | Action | State after | Wake cost |
|---|---|---|---|---|
| Sleep | 15 min (`IDLE_SLEEP_MIN`) | `POST /sleep?level=1` | weights → host RAM | ~1–6s |
| Offload | 3 hr (`IDLE_OFFLOAD_MIN`) | `POST /sleep?level=2` | RAM freed; ~5GB/GPU ctx | ~15–60s (NFS) |

Race-safety (review M1): the idle manager takes the **same per-service
asyncio lock** as the wake path and re-checks under the lock that
(a) active-request count == 0 and (b) last-activity is still past the
threshold. Last-activity is refreshed at request **completion** as well as
arrival, and an active-request counter guards long streaming generations —
a 20-minute stream must not trigger sleep mid-flight.

**Verify at implementation:** `/sleep?level=2` on a service already sleeping
at level 1 must discard the offloaded weights (not no-op/error). Fallback:
wake → re-sleep at level 2.

**Verify at implementation:** host RAM — level 1 holds a full weights copy
(text 43GB + ocr + embed ≈ 57GB total if all three nap simultaneously).
Check `free -g` headroom.

**NFS risk (accepted, documented):** level-2 wake requires a healthy NFS
mount. If NFS is down, the model is unwakeable and the router 503s (with
`restarting`-grade Retry-After) until NFS returns. Mitigation option: cap the
text model at level 1 (weights stay in RAM). Decide after calibration.

### 6.4 Router restart mid-wake (review M3)

The wake lock is in-process and dies with the router. The new instance must
not fire a second concurrent wake at a backend already waking. Mitigation:
the wake sequence is effectively idempotent from the router's view (re-POST
`/wake_up` on an already-waking backend is safe; vLLM serializes it), and the
request path re-checks `is_sleeping` under the lock before acting.

### 6.5 Admin API — separate listener, `127.0.0.1:8010` only

- `GET  /admin/status` — per-service: running?, sleeping?, depth, last activity, wake in progress
- `POST /admin/wake/{key}` / `POST /admin/sleep/{key}[?level=1|2]` — manual control (updates the router's depth tracking)

### 6.6 Failure modes

| Failure | Router behavior |
|---|---|
| vLLM container down/crashing | Detect unhealthy; hold up to 300s (`restart: unless-stopped` recovers it — but note cold start is 2–10 min for text); then 503 `Retry-After: 600` |
| Wake sequence fails | Retry once, then depth-aware 503 (6.2.1); never proxy half-awake |
| Router crash | `restart: unless-stopped`; vLLM services unaffected; at most in-flight requests lost; depth resets to conservative `offloaded` |
| Host reboot | **All three vLLM containers cold-start AWAKE** (vLLM has no boot-asleep) — see §7; text on GPU0/1 and small services on GPU2 initialize in parallel without racing; recovery takes minutes, router serves `restarting` 503s meanwhile |
| NFS outage | Level-2 unwakeable → 503s until NFS returns (§6.3) |
| Request without `model` field | Defaults to text **only** on chat/completions; elsewhere 400 |

## 7. Compose changes

- Add `router` service (build `./router/`): public `8000:8000`, admin
  `127.0.0.1:8010:8010`, `depends_on` all vllm services
  (`condition: service_started`).
- **Boot ordering (simplified in v3.3):** no staggering needed — text
  initializes on GPU0/1 while OCR and embed initialize on GPU2, disjoint
  memory pools, no race. All services use plain `depends_on:
  service_started` (sleep makes healthchecks flappy, so never
  `service_healthy`). The earlier wait-for-healthy entrypoint choreography
  (review-2 issue #1) is obsolete under this placement.
- Split `vllm` into `vllm-text` / `vllm-ocr` / `vllm-embed` with per-model
  args baked in; `MODEL_NAME`/`EXTRA_ARGS`/`TP_SIZE`/`MAX_MODEL_LEN` in
  `.env` become obsolete for serving (kept only for rollback).
- Remove `nginx` service and `nginx.conf`.
- All services: `restart: unless-stopped`, `ipc: host`,
  `NCCL_P2P_DISABLE=1`, explicit `CUDA_VISIBLE_DEVICES`.

Per-model serving args (all include `--disable-custom-all-reduce
--enable-sleep-mode`, **explicit `--max-model-len` each** — review M2):

- **text:** `--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.85`
- **ocr:** `--max-model-len 32768 --gpu-memory-utilization 0.10` (+ vision args from OvisOCR2 model card, e.g. `--limit-mm-per-prompt`)
- **embed:** `--max-model-len 8192 --gpu-memory-utilization 0.25`

## 8. `vllmctl` changes

- `up [MODEL]` / `down [MODEL]` → HTTP to `127.0.0.1:8010` admin API (no
  docker needed for routine control).
- `status` → `GET 127.0.0.1:8010/admin/status` pretty-printed.
- `pull` unchanged (docker, rare, manual).
- `idle-watch` removed (router owns idle management).

## 9. Implementation order

0. **Backups + cutover plan (review M5):**
   `cp compose.yml compose.yml.v2.bak; cp vllmctl vllmctl.v2.bak`.
   **Status 2026-08-14: `renbaibing` is now in the docker group** (works
   directly in fresh login shells; use `sg docker -c "…"` in sessions
   started before the change — `.runas.py`/`.user.env` are obsolete).
   Old stack already brought down by user; GPUs idle. Rollback: restore
   `.v2.bak` files, `docker compose up -d`.
1. **Calibration (gates everything):** bring up the three services with
   initial slices (old stack stopped); measure awake footprints, sleeping
   residuals, level-1 and level-2 wake latencies, host RAM cost; adjust
   slices so each GPU ≤ 0.88 including contexts. Log latencies → tune the
   §6.2.1 deadlines from data.
2. `router/` app (registry, wake single-flight, raw-body-safe routing,
   streaming proxy with proper timeouts, tiered idle with lock, admin
   listener) + compose rewiring; nginx removed.
3. `vllmctl` rewrite of up/down/status.
4. E2E verification (below).
5. Docs: README, TODO, memory.

## 10. E2E test matrix

1. All asleep → `POST /v1/chat/completions` (text) → 200 after wake; latency logged.
2. While text awake → `POST /v1/embeddings` → embed wakes independently; both serve.
3. OCR request with **`multipart/form-data`** body (not just JSON base64) → routes correctly, no body parsing error.
4. Idle tiers (shortened timers): 1 min → level-1 sleep (host RAM grows by weights); 3 min → level-2 offload (RAM freed); auto-wake from both.
5. Streaming request during wake → first chunk after wake, no buffering.
6. Unknown model name → 404 `model_not_found`, no wake triggered.
7. Dev endpoints (`/sleep`, `/wake_up`, `/collective_rpc`, `/reset_prefix_cache`) **and** `/admin/*` on public :8000 → 404; admin reachable only via 127.0.0.1:8010.
8. Kill vllm-text container → request → `restarting` 503 (`Retry-After: 600`) during recovery → 200 after restart completes.
9. Concurrent 10× requests while sleeping → exactly one wake sequence (router logs).
10. Depth-aware errors (hold deadline forced to 1s): level-1 wake → 503 `Retry-After: 10`, `"sleeping"`; level-2 → `Retry-After: 60`, `"offloaded"`; body parses with `estimated_wake_seconds`.
11. Idle-race: fire a request at the moment the idle timer expires → request served, no sleep mid-request (lock + active-counter verified).
12. Router restart mid-wake → no double-wake crash; request eventually served.
13. Two services waking concurrently (text + embed requested simultaneously) → both complete, no lock cross-talk.
14. Path traversal: `/v1/../sleep`, `/v1%2f..%2fsleep` → 404.
15. Host reboot (if feasible to test): parallel boot (text on GPU0/1, small services on GPU2 — no race), router serves `restarting` 503s, all three become available.

## 11. Risks / open items

- **GPU slice calibration** gates final numbers (§5, Step 1). v3.3 placement
  (text exclusive on GPU0/1 at 0.85) removes the KV-starvation concern;
  calibration confirms.
- **GPU2 trust:** single-GPU tests pass (2026-08-17, twice), but it has a
  fault history (Aug 11 driver re-probe). Mitigation: E2E test 3+4 exercise
  it before rollout; if GPU2 degrades in production, move OCR/embed back to
  GPU0/1 with one env-var change each (slices re-tighten to v3.2 values).
- ~~Which embedding model~~ — **resolved 2026-08-14: Qwen3-Embedding-8B.**
- **Level 1→2 escalation** and **host RAM** (~57GB if all three nap) need verification (§6.3).
- **NFS outage strands level-2 models** — accepted risk or cap text at level 1 (§6.3).
- Wake latency must fit calling services' client timeouts — confirm their settings before rollout.