vllm-frontdoor/CALIBRATION.md

# Step 1 calibration report — router front door plan v3.3

**Date:** 2026-08-17
**Scope:** plan §5 (GPU budget), §7 (compose), §9 step 1 (calibration).
**State at end of calibration:** `vllm-text`, `vllm-ocr`, `vllm-embed` all
**Up and awake**, serving on debug ports 8001/8002/8003. Router not built
(another work stream); its service block is defined in compose but not started.

## 0. Headline results

| Metric | vllm-text (TP=2) | vllm-ocr (TP=1) | vllm-embed (TP=1) |
|---|---|---|---|
| GPUs | 0+1 | 2 | 2 |
| util slice (final) | 0.85 | 0.10 | 0.25 |
| Awake GPU footprint | 67.8–70.3 GB per GPU (0.83–0.86) | 7.9 GB (0.097) | 21.0 GB (0.256) |
| Sleeping GPU residual (per process) | 5.5–5.7 GB per GPU | 3.4 GB | ~1.0 GB |
| Sleep L1 latency (first/warm) | 6.7 s / ~1.5 s | 0.8 s / 0.1 s | 6.5 s / <0.1 s |
| Sleep L2 latency (from awake) | 0.2 s | 0.1 s | 0.04 s |
| Wake from L1 — `wake_up` only | **2.5–3.8 s** | ~0.3 s | ~1.1 s |
| Wake from L1 — full sequence (with reload) | 23.4 s | 0.7 s | 4.0 s |
| Wake from L2 — full sequence (mandatory) | 22.9 s | 0.6 s | 3.3 s |
| Host RAM cost of L1 sleep (first cycle) | 39.7 GB | 2.5 GB | 20.7 GB |
| Boot time (weights → serving) | ~5 min (warm cache) | ~2.5 min | ~1.5 min |

**No slice changes were needed.** The initial v3.3 slices all fit (§2).

## 1. Compose config actually used

`/data/home/renbaibing/vllm/compose.yml` — four services, nginx removed.

- **Common to all three vLLM services:** image `vllm/vllm-openai:v0.27.1`,
  `${MODEL_ROOT}:/models:ro`, `ipc: host`, `gpus: all` + explicit
  `CUDA_VISIBLE_DEVICES` (the plan's one-env-var rollback mechanism),
  `VLLM_SERVER_DEV_MODE=1`, `NCCL_P2P_DISABLE=1`, `HF_TOKEN`, `restart:
  unless-stopped`, loopback-only debug port `127.0.0.1:800X:8000`,
  `--disable-custom-all-reduce --enable-sleep-mode`, explicit
  `--max-model-len` each. No healthchecks anywhere (sleep makes `/health`
  semantics misleading — see §5 — and `service_healthy` gating is forbidden
  by plan §7).
- **vllm-text** (GPU 0,1; debug 8001):
  `/models/Qwen3.6-35B-A3B-FP8 --served-model-name Qwen3.6-35B-A3B-FP8
  --tensor-parallel-size 2 --max-model-len 262144 --gpu-memory-utilization 0.85
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
  --disable-custom-all-reduce --enable-sleep-mode`
- **vllm-ocr** (GPU 2; debug 8002):
  `/models/OvisOCR2 --served-model-name OvisOCR2 --tensor-parallel-size 1
  --max-model-len 32768 --gpu-memory-utilization 0.10 --max-num-seqs 256
  --disable-custom-all-reduce --enable-sleep-mode`
  (`--max-num-seqs 256` is the one added flag — see §3.)
- **vllm-embed** (GPU 2; debug 8003):
  `/models/Qwen3-Embedding-8B --served-model-name Qwen3-Embedding-8B
  --tensor-parallel-size 1 --max-model-len 8192 --gpu-memory-utilization 0.25
  --disable-custom-all-reduce --enable-sleep-mode`
  (no task flag needed — auto-detected, see §3.)
- **router** (defined, NOT started): `build: ./router`, `8000:8000` public +
  `127.0.0.1:8010:8010` admin, `depends_on` the three vLLM services with
  `condition: service_started`.

## 2. GPU budget verification (plan §5)

Per-GPU sums, all awake (measured `nvidia-smi`, 81920 MiB per A800):

| GPU | Contents | Slice sum | Measured used | Fraction |
|---|---|---|---|---|
| 0 | text (TP worker 0) | 0.85 + ctx | 67779–70249 MiB | 0.827–0.857 |
| 1 | text (TP worker 1) | 0.85 + ctx | 67779–70249 MiB | 0.827–0.857 |
| 2 | ocr 0.10 + embed 0.25 | 0.35 + 2 ctx | 28929 MiB | 0.353 |

All ≤ 0.88 target. GPU0/1 peak at fresh boot 70249 MiB (0.857) — inside
budget with ~11.7 GB headroom. Sleeping (all processes L1/L2): GPU0/1 →
5.7 GB each, GPU2 → 4.5 GB total.

Text KV cache (from logs): **Available KV cache 41.72 GiB per GPU, GPU KV
cache 4,309,772 tokens, "Maximum concurrency for 262,144 tokens per request:
16.44x"**, CUDA graphs 1.62 GiB, weights 17.48 GiB/GPU (FP8). Far above the
~20 GB concern threshold — the plan's KV-starvation worry is fully retired:
full 262K context served with 16x concurrency. (vLLM notes util 0.85 with
CUDA-graph profiling ≈ 0.8316 effective; no action needed.)

OCR KV: 2.11 GiB → 173,056 tokens → 5.28x at 32K. Embed: pooling model
(KV small); max_model_len 8192 as planned.

No OOM at any point; no slice adjustments made.

## 3. Model-specific flags discovered

**OvisOCR2 (vllm-ocr):**
- Architecture `Qwen3_5ForConditionalGeneration` is natively registered in
  v0.27.1 → **no `--trust-remote-code` needed** (left out).
- dtype: `auto` resolves to bfloat16 from config; no flag needed.
- Vision limits: defaults fine (tested with a 640x160 PNG via
  `image_url` base64 — correct OCR text back). No `--limit-mm-per-prompt`.
- The README's `gdn_prefill_backend="triton"` (written for vllm 0.22.1) was
  NOT needed; v0.27.1 resolves the GDN backend automatically (`auto`).
- **Required addition: `--max-num-seqs 256`.** Without it boot fails:
  `max_num_seqs (1024) exceeds available Mamba cache blocks (316)` — the
  hybrid GDN model needs one Mamba cache block per decode sequence and the
  0.10 slice only fits 316. 256 < 316; OCR concurrency of 256 is far beyond
  any realistic page-parsing load. (Alternative would have been raising util;
  unnecessary.)

**Qwen3-Embedding-8B (vllm-embed):**
- **No flag needed.** v0.27.1 auto-detects: "Found pooling configuration"
  (reads sentence-transformers `modules.json`), logs
  `--runner auto → pooling`, `--convert auto → embed`, and builds
  `PoolerConfig(seq_pooling_type='LAST', use_activation=True)` — i.e. exactly
  Qwen3-Embedding semantics (last-token pooling + normalized output).
  Verified: `/v1/embeddings` returns dim=4096, norm=1.0000.
- Note for the router team: v0.27.1 has **no `--task` flag** (it was replaced
  by `--runner`/`--convert`). If explicitness is ever wanted, the equivalent
  is `--runner pooling --convert embed`; do NOT pass `--task embed` (unknown
  arg → boot failure).

**Qwen3.6-35B-A3B-FP8 (vllm-text):** booted with plan args verbatim;
fp8 block quantization auto-detected (`quantization=fp8`), runs on A800
(Ampere) via the fp8/marlin path, no flags required.

## 4. Sleep/wake measurements (per service)

All numbers from the loopback debug ports; wake "full sequence" =
`POST /wake_up` → `POST /collective_rpc {"method":"reload_weights"}` →
`POST /reset_prefix_cache` → `GET /health` 200.

### vllm-text
| Step | Latency |
|---|---|
| sleep L1 (first, allocates pinned pool) | 6.69 s |
| sleep L1 (warm, pool reused) | ~1.5 s |
| GPU after L1 | 5477 MiB per GPU (from 70249) |
| RAM delta after first L1 | −39.7 GB (weights 35 GB + buffers) |
| wake L1, `wake_up` only | 2.47 s (earlier run 3.7 s) |
| wake L1, full sequence | 23.42 s (reload_weights alone 20.94 s) |
| sleep L2 from awake | 0.22 s; GPU 5717 MiB; RAM −0.1 GB |
| wake L2, full sequence | 22.94 s (wake_up 1.75 + reload 21.17) |
| L1→L2 escalation | HTTP 200 in 0.01 s, **no RAM freed**, re-wake 23.7 s |

### vllm-ocr
| Step | Latency |
|---|---|
| sleep L1 | 0.82 s; GPU 8305→3459 MiB; RAM −2.5 GB |
| wake L1 full sequence | 0.68 s |
| sleep L2 | 0.11 s; GPU 3459 MiB |
| wake L2 full sequence | 0.63 s |
| L1→L2 escalation | 200 in 0.01 s, RAM retained, re-wake 0.67 s |

### vllm-embed
| Step | Latency |
|---|---|
| sleep L1 | 6.51 s (D2H copy); GPU 20996→~1050 MiB (GPU2 total 8981 with OCR awake); RAM −20.7 GB |
| wake L1 full sequence | 4.00 s (wake_up 1.06 + reload 2.93) |
| sleep L2 | 0.04 s |
| wake L2 full sequence | 3.31 s (reload 3.08 s from page-cached NFS) |
| L1→L2 escalation | 200 in 0.01 s, RAM retained, re-wake 4.00 s |

### Host RAM (plan §6.3 verification)
Host total **1082 GB** (`free -g`: 1007 GiB total column, 903 GiB available
with all three awake and page cache warm). Worst case all three nap at L1
simultaneously (first cycle after boot): 39.7 + 20.7 + 2.5 ≈ **63 GB** of
pinned host memory (plan estimated ~57 GB). With ~900 GB still available,
simultaneous L1 naps are entirely safe. Note: the pinned pools are reused
across sleep cycles — after the first L1→wake cycle, subsequent L1 sleeps
barely move `free` (allocator reuse), so don't alarm if the second nap shows
a ~0 delta.

## 5. Router-relevant behavior findings (important)

1. **`/health` returns 200 while a service is SLEEPING** (and `/v1/models`
   too). Health is a liveness check of the API server, NOT an awake check.
   The router must gate on **`/is_sleeping`**.
2. **A request sent to a sleeping service does not error — it HANGS**
   (queued behind the paused scheduler; verified >2 min). Never proxy to a
   backend without first checking `is_sleeping`.
3. **Requests admitted between `wake_up` and `reload_weights` get HTTP 200
   with GARBAGE content** (verified twice: OCR and text, reasoning field full
   of `!!!!...`). The full wake sequence must COMPLETE before the held
   request is proxied. "Never proxy to a half-awake backend" is not
   theoretical — it produces silently-wrong 200s.
4. **L1 wake does NOT need `reload_weights`.** After `wake_up` alone,
   text produced bit-identical output to the reloaded state (same prompt,
   temperature 0, identical reasoning text; correct answers 17*23=391,
   6*7=42). `reload_weights` after L1 costs ~21 s for text and adds nothing.
   → Router rule: **L1 wake = `wake_up` (+ optional reset_prefix_cache),
   ~2.5–4 s; L2 wake = full sequence with `reload_weights`, ~23 s (text).**
   If depth is unknown (router restart), use the full sequence.
5. **`POST /sleep?level=2` on a service already at level 1 is a well-behaved
   NO-OP** (plan §6.3 "verify"): HTTP 200 in ~0.01 s, `is_sleeping` stays
   true, **the L1 host-RAM copy is retained** (allocator code: level-2
   `sleep(offload_tags=())` never touches existing `cpu_backup_tensor`s),
   and the next wake is still RAM-fast. It does NOT free host RAM.
   → Tier escalation must be implemented as **wake (cheap, from RAM) then
   `sleep?level=2` from awake**, not as a direct L1→L2 call.
6. **Concurrent `wake_up`s are safe**: 5 parallel `POST /wake_up` at a
   sleeping text service all returned 200 in ~4.03 s each (serialized by the
   engine). Combined with finding 3, this confirms plan §6.4's
   idempotent-re-wake assumption.
7. `reset_prefix_cache` is instant (≤0.01 s) and harmless (prefix caching is
   disabled for the hybrid text model anyway; enabled for embed).
8. `/sleep` also accepts `?mode=` (default `abort` discards in-flight
   requests at sleep time) and level 0 (scheduler pause only, no memory
   change); `wake_up` accepts `?tags=` filtering. Not needed by the router
   today, but available.

## 6. Recommended final slices (unchanged from v3.3)

| Service | GPU(s) | util | Notes |
|---|---|---|---|
| vllm-text | 0,1 (TP=2) | **0.85** | KV 41.7 GiB/GPU, 16.4x @ 262K — ample |
| vllm-ocr | 2 | **0.10** | with `--max-num-seqs 256` |
| vllm-embed | 2 | **0.25** | 21 GB measured; ~15–16 GB is weights |

GPU2 has ~50 GB of headroom if embed ever needs `--max-model-len` above 8192.

## 7. Recommended router deadlines (plan §6.2.1)

Measured: L1 wake (wake_up only) worst case ~4 s; L2 wake (full sequence)
text ~23 s warm — all files page-cached; a true cold-NFS L2 wake has not been
measured (NFS streams at ~1–6 GB/s here depending on cache; 37.5 GB cold
could add ~30–60 s). Boot from scratch: text ~5 min warm, 2–10 min cold
(NOTES-2026-08-13).

| Situation | Plan value | Recommendation |
|---|---|---|
| Hold, wake from L1 | 30 s | **keep 30 s** (comfortable; measured ≤4 s). Could drop to 15 s. `Retry-After: 10` fine. `estimated_wake_seconds`: **5** |
| Hold, wake from L2 | 180 s | **keep 180 s** (measured 23 s warm; covers ~2 min cold-NFS margin). `Retry-After: 60` fine. `estimated_wake_seconds`: **30** (60 if NFS suspected cold) |
| Hold, container restarting | 300 s | **raise to 600 s** — cold text boot can reach 10 min; with 300 s the client gets a 503 exactly while recovery is still in progress, then must re-poll anyway. Keep `Retry-After: 600` |

Also recommend the router implement the depth-aware fast path from §5.4
(skip `reload_weights` for known-L1 wakes) — it turns the common case
(15-min idle tier) into a ~3 s wait for the text model instead of ~23 s.

## 8. Incidents and resolutions during calibration

- **Qwen3.6-35B-A3B-FP8 on disk was corrupt/incomplete** (16 layer files
  wrong-sized, 18 files missing/zero-byte, tokenizer/index empty) — caused
  by six stale, hung `hf download` processes from 2026-08-14 that had raced
  each other on the same `--local-dir`. Resolution: sha256-verified every
  file against the HF manifest, salvaged 14 intact files, re-downloaded the
  remaining 24.8 GB (HuggingFace via proxy was ~1–3 MB/s; ModelScope's
  mirror of the same public repo, direct + anonymous, averaged 3–6 MB/s;
  every byte sha256-verified against the official HF manifest — ModelScope
  was only a transport). New verified copy now lives at
  `/data/home/renbaibing/huggingface/Qwen3.6-35B-A3B-FP8`; the corrupt
  original is preserved at `.../Qwen3.6-35B-A3B-FP8.corrupt-20260817`
  (36 GB — safe to delete once trusted).
  Two stale-download casualties were also found and fixed:
  `Qwen3-Embedding-8B/model-00003-of-00004.safetensors` was truncated
  (re-fetched, verified) and OvisOCR2 verified fully intact.
  The six stale `hf download` processes are STILL hung (killing them was
  outside this session's permissions) — an operator should kill them; their
  target dir no longer exists, so they are harmless but should not be
  restarted as-is.
- **vllm-embed boot crash** (`SafetensorError: incomplete metadata`) — the
  truncated shard above; resolved by re-download.
- **vllm-ocr boot crash** (`max_num_seqs (1024) exceeds available Mamba
  cache blocks (316)`) — resolved with `--max-num-seqs 256` (kept util 0.10).

## 9. Final functional verification (all three awake)

- text: `POST /v1/chat/completions` → correct answers ("Paris", 391, 42),
  reasoning parser active, ~146–157 tok/s decode. Tool-call/reasoning args
  accepted at boot.
- ocr: chat completion with base64 PNG image → correct text extraction
  ("vLLM calibration 2026-08-17 / GPU2 sleep-mode test image"), 0.4 s.
- embed: `POST /v1/embeddings` → 4096-dim unit-norm vectors.
- All three: `GET /health` 200, `GET /v1/models` lists the right model,
  `GET /is_sleeping` false.