vllm-frontdoor/CALIBRATION.md

Step 1 calibration report — router front door plan v3.3

Date: 2026-08-17 Scope: plan §5 (GPU budget), §7 (compose), §9 step 1 (calibration). State at end of calibration: vllm-text, vllm-ocr, vllm-embed all Up and awake, serving on debug ports 8001/8002/8003. Router not built (another work stream); its service block is defined in compose but not started.

0. Headline results

Metric	vllm-text (TP=2)	vllm-ocr (TP=1)	vllm-embed (TP=1)
GPUs	0+1	2	2
util slice (final)	0.85	0.10	0.25
Awake GPU footprint	67.8–70.3 GB per GPU (0.83–0.86)	7.9 GB (0.097)	21.0 GB (0.256)
Sleeping GPU residual (per process)	5.5–5.7 GB per GPU	3.4 GB	~1.0 GB
Sleep L1 latency (first/warm)	6.7 s / ~1.5 s	0.8 s / 0.1 s	6.5 s / <0.1 s
Sleep L2 latency (from awake)	0.2 s	0.1 s	0.04 s
Wake from L1 — wake_up only	2.5–3.8 s	~0.3 s	~1.1 s
Wake from L1 — full sequence (with reload)	23.4 s	0.7 s	4.0 s
Wake from L2 — full sequence (mandatory)	22.9 s	0.6 s	3.3 s
Host RAM cost of L1 sleep (first cycle)	39.7 GB	2.5 GB	20.7 GB
Boot time (weights → serving)	~5 min (warm cache)	~2.5 min	~1.5 min

No slice changes were needed. The initial v3.3 slices all fit (§2).

1. Compose config actually used

/data/home/renbaibing/vllm/compose.yml — four services, nginx removed.

• Common to all three vLLM services: image vllm/vllm-openai:v0.27.1, ${MODEL_ROOT}:/models:ro, ipc: host, gpus: all + explicit CUDA_VISIBLE_DEVICES (the plan's one-env-var rollback mechanism), VLLM_SERVER_DEV_MODE=1, NCCL_P2P_DISABLE=1, HF_TOKEN, restart: unless-stopped, loopback-only debug port 127.0.0.1:800X:8000, --disable-custom-all-reduce --enable-sleep-mode, explicit --max-model-len each. No healthchecks anywhere (sleep makes /health semantics misleading — see §5 — and service_healthy gating is forbidden by plan §7).
• vllm-text (GPU 0,1; debug 8001): /models/Qwen3.6-35B-A3B-FP8 --served-model-name Qwen3.6-35B-A3B-FP8 --tensor-parallel-size 2 --max-model-len 262144 --gpu-memory-utilization 0.85 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --disable-custom-all-reduce --enable-sleep-mode
• vllm-ocr (GPU 2; debug 8002): /models/OvisOCR2 --served-model-name OvisOCR2 --tensor-parallel-size 1 --max-model-len 32768 --gpu-memory-utilization 0.10 --max-num-seqs 256 --disable-custom-all-reduce --enable-sleep-mode (--max-num-seqs 256 is the one added flag — see §3.)
• vllm-embed (GPU 2; debug 8003): /models/Qwen3-Embedding-8B --served-model-name Qwen3-Embedding-8B --tensor-parallel-size 1 --max-model-len 8192 --gpu-memory-utilization 0.25 --disable-custom-all-reduce --enable-sleep-mode (no task flag needed — auto-detected, see §3.)
• router (defined, NOT started): build: ./router, 8000:8000 public + 127.0.0.1:8010:8010 admin, depends_on the three vLLM services with condition: service_started.

2. GPU budget verification (plan §5)

Per-GPU sums, all awake (measured nvidia-smi, 81920 MiB per A800):

GPU	Contents	Slice sum	Measured used	Fraction
0	text (TP worker 0)	0.85 + ctx	67779–70249 MiB	0.827–0.857
1	text (TP worker 1)	0.85 + ctx	67779–70249 MiB	0.827–0.857
2	ocr 0.10 + embed 0.25	0.35 + 2 ctx	28929 MiB	0.353

All ≤ 0.88 target. GPU0/1 peak at fresh boot 70249 MiB (0.857) — inside budget with ~11.7 GB headroom. Sleeping (all processes L1/L2): GPU0/1 → 5.7 GB each, GPU2 → 4.5 GB total.

Text KV cache (from logs): Available KV cache 41.72 GiB per GPU, GPU KV cache 4,309,772 tokens, "Maximum concurrency for 262,144 tokens per request: 16.44x", CUDA graphs 1.62 GiB, weights 17.48 GiB/GPU (FP8). Far above the ~20 GB concern threshold — the plan's KV-starvation worry is fully retired: full 262K context served with 16x concurrency. (vLLM notes util 0.85 with CUDA-graph profiling ≈ 0.8316 effective; no action needed.)

OCR KV: 2.11 GiB → 173,056 tokens → 5.28x at 32K. Embed: pooling model (KV small); max_model_len 8192 as planned.

No OOM at any point; no slice adjustments made.

3. Model-specific flags discovered

OvisOCR2 (vllm-ocr):

• Architecture Qwen3_5ForConditionalGeneration is natively registered in v0.27.1 → no --trust-remote-code needed (left out).
• dtype: auto resolves to bfloat16 from config; no flag needed.
• Vision limits: defaults fine (tested with a 640x160 PNG via image_url base64 — correct OCR text back). No --limit-mm-per-prompt.
• The README's gdn_prefill_backend="triton" (written for vllm 0.22.1) was NOT needed; v0.27.1 resolves the GDN backend automatically (auto).
• Required addition: --max-num-seqs 256. Without it boot fails: max_num_seqs (1024) exceeds available Mamba cache blocks (316) — the hybrid GDN model needs one Mamba cache block per decode sequence and the 0.10 slice only fits 316. 256 < 316; OCR concurrency of 256 is far beyond any realistic page-parsing load. (Alternative would have been raising util; unnecessary.)

Qwen3-Embedding-8B (vllm-embed):

• No flag needed. v0.27.1 auto-detects: "Found pooling configuration" (reads sentence-transformers modules.json), logs --runner auto → pooling, --convert auto → embed, and builds PoolerConfig(seq_pooling_type='LAST', use_activation=True) — i.e. exactly Qwen3-Embedding semantics (last-token pooling + normalized output). Verified: /v1/embeddings returns dim=4096, norm=1.0000.
• Note for the router team: v0.27.1 has no --task flag (it was replaced by --runner/--convert). If explicitness is ever wanted, the equivalent is --runner pooling --convert embed; do NOT pass --task embed (unknown arg → boot failure).

Qwen3.6-35B-A3B-FP8 (vllm-text): booted with plan args verbatim; fp8 block quantization auto-detected (quantization=fp8), runs on A800 (Ampere) via the fp8/marlin path, no flags required.

4. Sleep/wake measurements (per service)

All numbers from the loopback debug ports; wake "full sequence" = POST /wake_up → POST /collective_rpc {"method":"reload_weights"} → POST /reset_prefix_cache → GET /health 200.

vllm-text

Step	Latency
sleep L1 (first, allocates pinned pool)	6.69 s
sleep L1 (warm, pool reused)	~1.5 s
GPU after L1	5477 MiB per GPU (from 70249)
RAM delta after first L1	−39.7 GB (weights 35 GB + buffers)
wake L1, wake_up only	2.47 s (earlier run 3.7 s)
wake L1, full sequence	23.42 s (reload_weights alone 20.94 s)
sleep L2 from awake	0.22 s; GPU 5717 MiB; RAM −0.1 GB
wake L2, full sequence	22.94 s (wake_up 1.75 + reload 21.17)
L1→L2 escalation	HTTP 200 in 0.01 s, no RAM freed, re-wake 23.7 s

vllm-ocr

Step	Latency
sleep L1	0.82 s; GPU 8305→3459 MiB; RAM −2.5 GB
wake L1 full sequence	0.68 s
sleep L2	0.11 s; GPU 3459 MiB
wake L2 full sequence	0.63 s
L1→L2 escalation	200 in 0.01 s, RAM retained, re-wake 0.67 s

vllm-embed

Step	Latency
sleep L1	6.51 s (D2H copy); GPU 20996→~1050 MiB (GPU2 total 8981 with OCR awake); RAM −20.7 GB
wake L1 full sequence	4.00 s (wake_up 1.06 + reload 2.93)
sleep L2	0.04 s
wake L2 full sequence	3.31 s (reload 3.08 s from page-cached NFS)
L1→L2 escalation	200 in 0.01 s, RAM retained, re-wake 4.00 s

Host RAM (plan §6.3 verification)

Host total 1082 GB (free -g: 1007 GiB total column, 903 GiB available with all three awake and page cache warm). Worst case all three nap at L1 simultaneously (first cycle after boot): 39.7 + 20.7 + 2.5 ≈ 63 GB of pinned host memory (plan estimated ~57 GB). With ~900 GB still available, simultaneous L1 naps are entirely safe. Note: the pinned pools are reused across sleep cycles — after the first L1→wake cycle, subsequent L1 sleeps barely move free (allocator reuse), so don't alarm if the second nap shows a ~0 delta.

5. Router-relevant behavior findings (important)

1. /health returns 200 while a service is SLEEPING (and /v1/models too). Health is a liveness check of the API server, NOT an awake check. The router must gate on /is_sleeping.
2. A request sent to a sleeping service does not error — it HANGS (queued behind the paused scheduler; verified >2 min). Never proxy to a backend without first checking is_sleeping.
3. Requests admitted between wake_up and reload_weights get HTTP 200 with GARBAGE content (verified twice: OCR and text, reasoning field full of !!!!...). The full wake sequence must COMPLETE before the held request is proxied. "Never proxy to a half-awake backend" is not theoretical — it produces silently-wrong 200s.
4. L1 wake does NOT need reload_weights. After wake_up alone, text produced bit-identical output to the reloaded state (same prompt, temperature 0, identical reasoning text; correct answers 1723=391, 67=42). reload_weights after L1 costs ~21 s for text and adds nothing. → Router rule: L1 wake = wake_up (+ optional reset_prefix_cache), ~2.5–4 s; L2 wake = full sequence with reload_weights, ~23 s (text). If depth is unknown (router restart), use the full sequence.
5. POST /sleep?level=2 on a service already at level 1 is a well-behaved NO-OP (plan §6.3 "verify"): HTTP 200 in ~0.01 s, is_sleeping stays true, the L1 host-RAM copy is retained (allocator code: level-2 sleep(offload_tags=()) never touches existing cpu_backup_tensors), and the next wake is still RAM-fast. It does NOT free host RAM. → Tier escalation must be implemented as wake (cheap, from RAM) then sleep?level=2 from awake, not as a direct L1→L2 call.
6. Concurrent wake_ups are safe: 5 parallel POST /wake_up at a sleeping text service all returned 200 in ~4.03 s each (serialized by the engine). Combined with finding 3, this confirms plan §6.4's idempotent-re-wake assumption.
7. reset_prefix_cache is instant (≤0.01 s) and harmless (prefix caching is disabled for the hybrid text model anyway; enabled for embed).
8. /sleep also accepts ?mode= (default abort discards in-flight requests at sleep time) and level 0 (scheduler pause only, no memory change); wake_up accepts ?tags= filtering. Not needed by the router today, but available.

6. Recommended final slices (unchanged from v3.3)

Service	GPU(s)	util	Notes
vllm-text	0,1 (TP=2)	0.85	KV 41.7 GiB/GPU, 16.4x @ 262K — ample
vllm-ocr	2	0.10	with --max-num-seqs 256
vllm-embed	2	0.25	21 GB measured; ~15–16 GB is weights

GPU2 has ~50 GB of headroom if embed ever needs --max-model-len above 8192.

7. Recommended router deadlines (plan §6.2.1)

Measured: L1 wake (wake_up only) worst case ~4 s; L2 wake (full sequence) text ~23 s warm — all files page-cached; a true cold-NFS L2 wake has not been measured (NFS streams at ~1–6 GB/s here depending on cache; 37.5 GB cold could add ~30–60 s). Boot from scratch: text ~5 min warm, 2–10 min cold (NOTES-2026-08-13).

Situation	Plan value	Recommendation
Hold, wake from L1	30 s	keep 30 s (comfortable; measured ≤4 s). Could drop to 15 s. Retry-After: 10 fine. estimated_wake_seconds: 5
Hold, wake from L2	180 s	keep 180 s (measured 23 s warm; covers ~2 min cold-NFS margin). Retry-After: 60 fine. estimated_wake_seconds: 30 (60 if NFS suspected cold)
Hold, container restarting	300 s	raise to 600 s — cold text boot can reach 10 min; with 300 s the client gets a 503 exactly while recovery is still in progress, then must re-poll anyway. Keep Retry-After: 600

Also recommend the router implement the depth-aware fast path from §5.4 (skip reload_weights for known-L1 wakes) — it turns the common case (15-min idle tier) into a ~3 s wait for the text model instead of ~23 s.

8. Incidents and resolutions during calibration

• Qwen3.6-35B-A3B-FP8 on disk was corrupt/incomplete (16 layer files wrong-sized, 18 files missing/zero-byte, tokenizer/index empty) — caused by six stale, hung hf download processes from 2026-08-14 that had raced each other on the same --local-dir. Resolution: sha256-verified every file against the HF manifest, salvaged 14 intact files, re-downloaded the remaining 24.8 GB (HuggingFace via proxy was ~1–3 MB/s; ModelScope's mirror of the same public repo, direct + anonymous, averaged 3–6 MB/s; every byte sha256-verified against the official HF manifest — ModelScope was only a transport). New verified copy now lives at /data/home/renbaibing/huggingface/Qwen3.6-35B-A3B-FP8; the corrupt original is preserved at .../Qwen3.6-35B-A3B-FP8.corrupt-20260817 (36 GB — safe to delete once trusted). Two stale-download casualties were also found and fixed: Qwen3-Embedding-8B/model-00003-of-00004.safetensors was truncated (re-fetched, verified) and OvisOCR2 verified fully intact. The six stale hf download processes are STILL hung (killing them was outside this session's permissions) — an operator should kill them; their target dir no longer exists, so they are harmless but should not be restarted as-is.
• vllm-embed boot crash (SafetensorError: incomplete metadata) — the truncated shard above; resolved by re-download.
• vllm-ocr boot crash (max_num_seqs (1024) exceeds available Mamba cache blocks (316)) — resolved with --max-num-seqs 256 (kept util 0.10).

9. Final functional verification (all three awake)

• text: POST /v1/chat/completions → correct answers ("Paris", 391, 42), reasoning parser active, ~146–157 tok/s decode. Tool-call/reasoning args accepted at boot.
• ocr: chat completion with base64 PNG image → correct text extraction ("vLLM calibration 2026-08-17 / GPU2 sleep-mode test image"), 0.4 s.
• embed: POST /v1/embeddings → 4096-dim unit-norm vectors.
• All three: GET /health 200, GET /v1/models lists the right model, GET /is_sleeping false.