Initial commit: router front-door vLLM stack

Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2,
sleep mode) behind a FastAPI router that auto-wakes models on request.
Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with
Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010.
Routine control via vllmctl is pure HTTP — no docker on the request path.

Verified: 91 router unit tests + 15-test E2E on real hardware
(measurements in CALIBRATION.md; design record in
.claude/memory/router-front-door-plan.md).

Old nginx stack files removed before git init; design survives in
.claude/memory/sleep-mode-implementation-plan.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CALIBRATION.md

@@ -0,0 +1,266 @@
+# Step 1 calibration report — router front door plan v3.3
+
+**Date:** 2026-08-17
+**Scope:** plan §5 (GPU budget), §7 (compose), §9 step 1 (calibration).
+**State at end of calibration:** `vllm-text`, `vllm-ocr`, `vllm-embed` all
+**Up and awake**, serving on debug ports 8001/8002/8003. Router not built
+(another work stream); its service block is defined in compose but not started.
+
+## 0. Headline results
+
+| Metric | vllm-text (TP=2) | vllm-ocr (TP=1) | vllm-embed (TP=1) |
+|---|---|---|---|
+| GPUs | 0+1 | 2 | 2 |
+| util slice (final) | 0.85 | 0.10 | 0.25 |
+| Awake GPU footprint | 67.8–70.3 GB per GPU (0.83–0.86) | 7.9 GB (0.097) | 21.0 GB (0.256) |
+| Sleeping GPU residual (per process) | 5.5–5.7 GB per GPU | 3.4 GB | ~1.0 GB |
+| Sleep L1 latency (first/warm) | 6.7 s / ~1.5 s | 0.8 s / 0.1 s | 6.5 s / <0.1 s |
+| Sleep L2 latency (from awake) | 0.2 s | 0.1 s | 0.04 s |
+| Wake from L1 — `wake_up` only | **2.5–3.8 s** | ~0.3 s | ~1.1 s |
+| Wake from L1 — full sequence (with reload) | 23.4 s | 0.7 s | 4.0 s |
+| Wake from L2 — full sequence (mandatory) | 22.9 s | 0.6 s | 3.3 s |
+| Host RAM cost of L1 sleep (first cycle) | 39.7 GB | 2.5 GB | 20.7 GB |
+| Boot time (weights → serving) | ~5 min (warm cache) | ~2.5 min | ~1.5 min |
+
+**No slice changes were needed.** The initial v3.3 slices all fit (§2).
+
+## 1. Compose config actually used
+
+`/data/home/renbaibing/vllm/compose.yml` — four services, nginx removed.
+
+- **Common to all three vLLM services:** image `vllm/vllm-openai:v0.27.1`,
+  `${MODEL_ROOT}:/models:ro`, `ipc: host`, `gpus: all` + explicit
+  `CUDA_VISIBLE_DEVICES` (the plan's one-env-var rollback mechanism),
+  `VLLM_SERVER_DEV_MODE=1`, `NCCL_P2P_DISABLE=1`, `HF_TOKEN`, `restart:
+  unless-stopped`, loopback-only debug port `127.0.0.1:800X:8000`,
+  `--disable-custom-all-reduce --enable-sleep-mode`, explicit
+  `--max-model-len` each. No healthchecks anywhere (sleep makes `/health`
+  semantics misleading — see §5 — and `service_healthy` gating is forbidden
+  by plan §7).
+- **vllm-text** (GPU 0,1; debug 8001):
+  `/models/Qwen3.6-35B-A3B-FP8 --served-model-name Qwen3.6-35B-A3B-FP8
+  --tensor-parallel-size 2 --max-model-len 262144 --gpu-memory-utilization 0.85
+  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
+  --disable-custom-all-reduce --enable-sleep-mode`
+- **vllm-ocr** (GPU 2; debug 8002):
+  `/models/OvisOCR2 --served-model-name OvisOCR2 --tensor-parallel-size 1
+  --max-model-len 32768 --gpu-memory-utilization 0.10 --max-num-seqs 256
+  --disable-custom-all-reduce --enable-sleep-mode`
+  (`--max-num-seqs 256` is the one added flag — see §3.)
+- **vllm-embed** (GPU 2; debug 8003):
+  `/models/Qwen3-Embedding-8B --served-model-name Qwen3-Embedding-8B
+  --tensor-parallel-size 1 --max-model-len 8192 --gpu-memory-utilization 0.25
+  --disable-custom-all-reduce --enable-sleep-mode`
+  (no task flag needed — auto-detected, see §3.)
+- **router** (defined, NOT started): `build: ./router`, `8000:8000` public +
+  `127.0.0.1:8010:8010` admin, `depends_on` the three vLLM services with
+  `condition: service_started`.
+
+## 2. GPU budget verification (plan §5)
+
+Per-GPU sums, all awake (measured `nvidia-smi`, 81920 MiB per A800):
+
+| GPU | Contents | Slice sum | Measured used | Fraction |
+|---|---|---|---|---|
+| 0 | text (TP worker 0) | 0.85 + ctx | 67779–70249 MiB | 0.827–0.857 |
+| 1 | text (TP worker 1) | 0.85 + ctx | 67779–70249 MiB | 0.827–0.857 |
+| 2 | ocr 0.10 + embed 0.25 | 0.35 + 2 ctx | 28929 MiB | 0.353 |
+
+All ≤ 0.88 target. GPU0/1 peak at fresh boot 70249 MiB (0.857) — inside
+budget with ~11.7 GB headroom. Sleeping (all processes L1/L2): GPU0/1 →
+5.7 GB each, GPU2 → 4.5 GB total.
+
+Text KV cache (from logs): **Available KV cache 41.72 GiB per GPU, GPU KV
+cache 4,309,772 tokens, "Maximum concurrency for 262,144 tokens per request:
+16.44x"**, CUDA graphs 1.62 GiB, weights 17.48 GiB/GPU (FP8). Far above the
+~20 GB concern threshold — the plan's KV-starvation worry is fully retired:
+full 262K context served with 16x concurrency. (vLLM notes util 0.85 with
+CUDA-graph profiling ≈ 0.8316 effective; no action needed.)
+
+OCR KV: 2.11 GiB → 173,056 tokens → 5.28x at 32K. Embed: pooling model
+(KV small); max_model_len 8192 as planned.
+
+No OOM at any point; no slice adjustments made.
+
+## 3. Model-specific flags discovered
+
+**OvisOCR2 (vllm-ocr):**
+- Architecture `Qwen3_5ForConditionalGeneration` is natively registered in
+  v0.27.1 → **no `--trust-remote-code` needed** (left out).
+- dtype: `auto` resolves to bfloat16 from config; no flag needed.
+- Vision limits: defaults fine (tested with a 640x160 PNG via
+  `image_url` base64 — correct OCR text back). No `--limit-mm-per-prompt`.
+- The README's `gdn_prefill_backend="triton"` (written for vllm 0.22.1) was
+  NOT needed; v0.27.1 resolves the GDN backend automatically (`auto`).
+- **Required addition: `--max-num-seqs 256`.** Without it boot fails:
+  `max_num_seqs (1024) exceeds available Mamba cache blocks (316)` — the
+  hybrid GDN model needs one Mamba cache block per decode sequence and the
+  0.10 slice only fits 316. 256 < 316; OCR concurrency of 256 is far beyond
+  any realistic page-parsing load. (Alternative would have been raising util;
+  unnecessary.)
+
+**Qwen3-Embedding-8B (vllm-embed):**
+- **No flag needed.** v0.27.1 auto-detects: "Found pooling configuration"
+  (reads sentence-transformers `modules.json`), logs
+  `--runner auto → pooling`, `--convert auto → embed`, and builds
+  `PoolerConfig(seq_pooling_type='LAST', use_activation=True)` — i.e. exactly
+  Qwen3-Embedding semantics (last-token pooling + normalized output).
+  Verified: `/v1/embeddings` returns dim=4096, norm=1.0000.
+- Note for the router team: v0.27.1 has **no `--task` flag** (it was replaced
+  by `--runner`/`--convert`). If explicitness is ever wanted, the equivalent
+  is `--runner pooling --convert embed`; do NOT pass `--task embed` (unknown
+  arg → boot failure).
+
+**Qwen3.6-35B-A3B-FP8 (vllm-text):** booted with plan args verbatim;
+fp8 block quantization auto-detected (`quantization=fp8`), runs on A800
+(Ampere) via the fp8/marlin path, no flags required.
+
+## 4. Sleep/wake measurements (per service)
+
+All numbers from the loopback debug ports; wake "full sequence" =
+`POST /wake_up` → `POST /collective_rpc {"method":"reload_weights"}` →
+`POST /reset_prefix_cache` → `GET /health` 200.
+
+### vllm-text
+| Step | Latency |
+|---|---|
+| sleep L1 (first, allocates pinned pool) | 6.69 s |
+| sleep L1 (warm, pool reused) | ~1.5 s |
+| GPU after L1 | 5477 MiB per GPU (from 70249) |
+| RAM delta after first L1 | −39.7 GB (weights 35 GB + buffers) |
+| wake L1, `wake_up` only | 2.47 s (earlier run 3.7 s) |
+| wake L1, full sequence | 23.42 s (reload_weights alone 20.94 s) |
+| sleep L2 from awake | 0.22 s; GPU 5717 MiB; RAM −0.1 GB |
+| wake L2, full sequence | 22.94 s (wake_up 1.75 + reload 21.17) |
+| L1→L2 escalation | HTTP 200 in 0.01 s, **no RAM freed**, re-wake 23.7 s |
+
+### vllm-ocr
+| Step | Latency |
+|---|---|
+| sleep L1 | 0.82 s; GPU 8305→3459 MiB; RAM −2.5 GB |
+| wake L1 full sequence | 0.68 s |
+| sleep L2 | 0.11 s; GPU 3459 MiB |
+| wake L2 full sequence | 0.63 s |
+| L1→L2 escalation | 200 in 0.01 s, RAM retained, re-wake 0.67 s |
+
+### vllm-embed
+| Step | Latency |
+|---|---|
+| sleep L1 | 6.51 s (D2H copy); GPU 20996→~1050 MiB (GPU2 total 8981 with OCR awake); RAM −20.7 GB |
+| wake L1 full sequence | 4.00 s (wake_up 1.06 + reload 2.93) |
+| sleep L2 | 0.04 s |
+| wake L2 full sequence | 3.31 s (reload 3.08 s from page-cached NFS) |
+| L1→L2 escalation | 200 in 0.01 s, RAM retained, re-wake 4.00 s |
+
+### Host RAM (plan §6.3 verification)
+Host total **1082 GB** (`free -g`: 1007 GiB total column, 903 GiB available
+with all three awake and page cache warm). Worst case all three nap at L1
+simultaneously (first cycle after boot): 39.7 + 20.7 + 2.5 ≈ **63 GB** of
+pinned host memory (plan estimated ~57 GB). With ~900 GB still available,
+simultaneous L1 naps are entirely safe. Note: the pinned pools are reused
+across sleep cycles — after the first L1→wake cycle, subsequent L1 sleeps
+barely move `free` (allocator reuse), so don't alarm if the second nap shows
+a ~0 delta.
+
+## 5. Router-relevant behavior findings (important)
+
+1. **`/health` returns 200 while a service is SLEEPING** (and `/v1/models`
+   too). Health is a liveness check of the API server, NOT an awake check.
+   The router must gate on **`/is_sleeping`**.
+2. **A request sent to a sleeping service does not error — it HANGS**
+   (queued behind the paused scheduler; verified >2 min). Never proxy to a
+   backend without first checking `is_sleeping`.
+3. **Requests admitted between `wake_up` and `reload_weights` get HTTP 200
+   with GARBAGE content** (verified twice: OCR and text, reasoning field full
+   of `!!!!...`). The full wake sequence must COMPLETE before the held
+   request is proxied. "Never proxy to a half-awake backend" is not
+   theoretical — it produces silently-wrong 200s.
+4. **L1 wake does NOT need `reload_weights`.** After `wake_up` alone,
+   text produced bit-identical output to the reloaded state (same prompt,
+   temperature 0, identical reasoning text; correct answers 17*23=391,
+   6*7=42). `reload_weights` after L1 costs ~21 s for text and adds nothing.
+   → Router rule: **L1 wake = `wake_up` (+ optional reset_prefix_cache),
+   ~2.5–4 s; L2 wake = full sequence with `reload_weights`, ~23 s (text).**
+   If depth is unknown (router restart), use the full sequence.
+5. **`POST /sleep?level=2` on a service already at level 1 is a well-behaved
+   NO-OP** (plan §6.3 "verify"): HTTP 200 in ~0.01 s, `is_sleeping` stays
+   true, **the L1 host-RAM copy is retained** (allocator code: level-2
+   `sleep(offload_tags=())` never touches existing `cpu_backup_tensor`s),
+   and the next wake is still RAM-fast. It does NOT free host RAM.
+   → Tier escalation must be implemented as **wake (cheap, from RAM) then
+   `sleep?level=2` from awake**, not as a direct L1→L2 call.
+6. **Concurrent `wake_up`s are safe**: 5 parallel `POST /wake_up` at a
+   sleeping text service all returned 200 in ~4.03 s each (serialized by the
+   engine). Combined with finding 3, this confirms plan §6.4's
+   idempotent-re-wake assumption.
+7. `reset_prefix_cache` is instant (≤0.01 s) and harmless (prefix caching is
+   disabled for the hybrid text model anyway; enabled for embed).
+8. `/sleep` also accepts `?mode=` (default `abort` discards in-flight
+   requests at sleep time) and level 0 (scheduler pause only, no memory
+   change); `wake_up` accepts `?tags=` filtering. Not needed by the router
+   today, but available.
+
+## 6. Recommended final slices (unchanged from v3.3)
+
+| Service | GPU(s) | util | Notes |
+|---|---|---|---|
+| vllm-text | 0,1 (TP=2) | **0.85** | KV 41.7 GiB/GPU, 16.4x @ 262K — ample |
+| vllm-ocr | 2 | **0.10** | with `--max-num-seqs 256` |
+| vllm-embed | 2 | **0.25** | 21 GB measured; ~15–16 GB is weights |
+
+GPU2 has ~50 GB of headroom if embed ever needs `--max-model-len` above 8192.
+
+## 7. Recommended router deadlines (plan §6.2.1)
+
+Measured: L1 wake (wake_up only) worst case ~4 s; L2 wake (full sequence)
+text ~23 s warm — all files page-cached; a true cold-NFS L2 wake has not been
+measured (NFS streams at ~1–6 GB/s here depending on cache; 37.5 GB cold
+could add ~30–60 s). Boot from scratch: text ~5 min warm, 2–10 min cold
+(NOTES-2026-08-13).
+
+| Situation | Plan value | Recommendation |
+|---|---|---|
+| Hold, wake from L1 | 30 s | **keep 30 s** (comfortable; measured ≤4 s). Could drop to 15 s. `Retry-After: 10` fine. `estimated_wake_seconds`: **5** |
+| Hold, wake from L2 | 180 s | **keep 180 s** (measured 23 s warm; covers ~2 min cold-NFS margin). `Retry-After: 60` fine. `estimated_wake_seconds`: **30** (60 if NFS suspected cold) |
+| Hold, container restarting | 300 s | **raise to 600 s** — cold text boot can reach 10 min; with 300 s the client gets a 503 exactly while recovery is still in progress, then must re-poll anyway. Keep `Retry-After: 600` |
+
+Also recommend the router implement the depth-aware fast path from §5.4
+(skip `reload_weights` for known-L1 wakes) — it turns the common case
+(15-min idle tier) into a ~3 s wait for the text model instead of ~23 s.
+
+## 8. Incidents and resolutions during calibration
+
+- **Qwen3.6-35B-A3B-FP8 on disk was corrupt/incomplete** (16 layer files
+  wrong-sized, 18 files missing/zero-byte, tokenizer/index empty) — caused
+  by six stale, hung `hf download` processes from 2026-08-14 that had raced
+  each other on the same `--local-dir`. Resolution: sha256-verified every
+  file against the HF manifest, salvaged 14 intact files, re-downloaded the
+  remaining 24.8 GB (HuggingFace via proxy was ~1–3 MB/s; ModelScope's
+  mirror of the same public repo, direct + anonymous, averaged 3–6 MB/s;
+  every byte sha256-verified against the official HF manifest — ModelScope
+  was only a transport). New verified copy now lives at
+  `/data/home/renbaibing/huggingface/Qwen3.6-35B-A3B-FP8`; the corrupt
+  original is preserved at `.../Qwen3.6-35B-A3B-FP8.corrupt-20260817`
+  (36 GB — safe to delete once trusted).
+  Two stale-download casualties were also found and fixed:
+  `Qwen3-Embedding-8B/model-00003-of-00004.safetensors` was truncated
+  (re-fetched, verified) and OvisOCR2 verified fully intact.
+  The six stale `hf download` processes are STILL hung (killing them was
+  outside this session's permissions) — an operator should kill them; their
+  target dir no longer exists, so they are harmless but should not be
+  restarted as-is.
+- **vllm-embed boot crash** (`SafetensorError: incomplete metadata`) — the
+  truncated shard above; resolved by re-download.
+- **vllm-ocr boot crash** (`max_num_seqs (1024) exceeds available Mamba
+  cache blocks (316)`) — resolved with `--max-num-seqs 256` (kept util 0.10).
+
+## 9. Final functional verification (all three awake)
+
+- text: `POST /v1/chat/completions` → correct answers ("Paris", 391, 42),
+  reasoning parser active, ~146–157 tok/s decode. Tool-call/reasoning args
+  accepted at boot.
+- ocr: chat completion with base64 PNG image → correct text extraction
+  ("vLLM calibration 2026-08-17 / GPU2 sleep-mode test image"), 0.4 s.
+- embed: `POST /v1/embeddings` → 4096-dim unit-norm vectors.
+- All three: `GET /health` 200, `GET /v1/models` lists the right model,
+  `GET /is_sleeping` false.