vLLM model server — router front door, auto wake-on-request

History: 2026-08-13 rework + GPU-fault diagnosis (NOTES-2026-08-13.md), GPU2 re-check + root cause (NOTES-2026-08-17-gpu2-recheck.md), measured GPU/wake numbers (CALIBRATION.md). Hardware sanity scripts live in diag/ (p2p_check.py is the key one).

Three models are always available at one OpenAI-compatible endpoint. Calling services never need to know whether a model is loaded — the router front door wakes it on request:

http://<host>:8000/v1/...

Everything runs inside the Docker stack. No host-side cron, watchers, or docker commands on the request path.

Models

Type	Model name (use in model field)	Service / GPUs
Text / chat	Qwen3.6-35B-A3B-FP8 (aliases: qwen, text, …)	vllm-text, TP=2, GPU0+1
OCR (vision)	OvisOCR2 (aliases: ocr, ovis)	vllm-ocr, GPU2
Embeddings	Qwen3-Embedding-8B (aliases: embed, …)	vllm-embed, GPU2

GET /v1/models lists all three. Model matching is case-insensitive; an unknown name returns a 404 model_not_found (no wake is triggered).

What a request sees

• Awake model → proxied immediately (streaming passes through unbuffered).
• Sleeping model → the request is held while the router wakes it (single-flight: 10 concurrent requests → one wake), then proxied.
• Wake exceeds the hold deadline → 503 with a depth-calibrated Retry-After header AND an OpenAI-style JSON body:

Waking from	Typical wake	Hold deadline	Retry-After	body sleep_depth
level-1 sleep (weights in RAM)	2.5–4 s (text), <1 s (small)	30 s	10	"sleeping"
level-2 offload (weights on NFS)	~23 s (text), 1–4 s (small)	180 s	60	"offloaded"
container restart / cold boot	2–10 min (text, NFS)	600 s	600	"restarting"

{"error": {"type": "model_waking", "code": "model_waking",
           "message": "Model '…' is waking from offload; retry shortly",
           "sleep_depth": "offloaded", "estimated_wake_seconds": 60}}

Idle management (automatic)

After idle	Action	Effect
15 min	sleep (level 1)	GPU freed; weights parked in host RAM
3 h	offload (level 2)	host RAM freed too; ~5 GB/GPU CUDA context remains

First request after either tier wakes the model transparently. Timers: IDLE_SLEEP_MIN / IDLE_OFFLOAD_MIN env on the router service.

Quick start

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.6-35B-A3B-FP8","messages":[{"role":"user","content":"hi"}]}'

curl http://127.0.0.1:8000/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3-Embedding-8B","input":"hello"}'

vllmctl

./vllmctl status            # per-model state table (sleep depth, activity)
./vllmctl up [MODEL]        # force-wake (HTTP to the admin API — no docker)
./vllmctl sleep [MODEL]     # level-1 sleep now
./vllmctl down [MODEL|all]  # level-2 offload now
./vllmctl list              # models on disk + served
./vllmctl logs [-f] [N] [SVC]
./vllmctl restart SVC
./vllmctl pull <hf-repo>    # download a new model (needs docker)

up/down/sleep/status are pure HTTP against the admin listener; routine control never needs docker.

Architecture

callers ──► :8000 ──► router (FastAPI, ./router)      [only public ingress]
                       ├─► vllm-text   GPU0+1 (TP=2, util 0.85, 262K ctx)
                       ├─► vllm-ocr    GPU2   (TP=1, util 0.10)
                       └─► vllm-embed  GPU2   (TP=1, util 0.25)
127.0.0.1:8010 ──► router admin API (vllmctl; not reachable off-host)
127.0.0.1:8001-8003 ──► direct debug access to each vLLM (local users only)

• Each vLLM service is always running with sleep mode (--enable-sleep-mode, VLLM_SERVER_DEV_MODE=1). The router owns wake/sleep/idle, model routing, streaming proxy, and endpoint hiding (dev endpoints and /admin/* 404 on the public port; paths are normalized against traversal).
• The router is a single process with two listeners (public + admin) on one event loop, so admin calls share the request path's locks and depth state.
• Wake-intent is persisted (named volume router-state), so a router restart mid-wake conservatively completes reload_weights before serving — an interrupted level-2 wake can never serve garbage.
• All services: restart: unless-stopped (host reboots self-heal; requests get "restarting" 503s during the gap).

Router tunables

All env-driven with sane defaults — see the table at the top of router/config.py (registry/aliases, listeners, proxy timeouts, wake hold deadlines, idle tiers, state file). Unit tests: router/tests/ (91 tests, no GPU needed).

Hardware constraints (do not remove)

• GPU P2P DMA corrupts data on this host (4/6 pairs; see NOTES). NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce are mandatory on every vLLM service. Host-staged copies are verified clean.
• Text spans GPU0+GPU1 exclusively; OCR and embed live on GPU2 (single-GPU services never issue P2P). Moving a service = edit its CUDA_VISIBLE_DEVICES in compose.yml, then re-tighten memory slices (per-GPU sums ≤ 0.88 including CUDA contexts).
• Measured footprints in CALIBRATION.md.

How docker access works

renbaibing is in the docker group (since 2026-08-14) — docker and docker compose run directly, no sudo. Shells opened before the group change need sg docker -c "…" or a fresh login. The old .runas.py / .user.env mechanism is obsolete.

Operations

docker compose up -d            # whole stack
docker compose restart router   # front door only (state survives)
docker logs vllm-router         # structured wake/sleep/idle events
curl 127.0.0.1:8010/admin/status

The pre-router (nginx) stack was removed 2026-08-17 (before git init) — it survives only as a design doc (.claude/memory/sleep-mode-implementation-plan.md). Rollback points are git commits from 2026-08-17 onward.

Notes

• .env provides VLLM_VERSION, MODEL_ROOT, HF_TOKEN (used by compose) plus client-side vars (OPENAI_BASE_URL etc.). All per-model serving args live in compose.yml.
• A .corrupt-20260817 copy of the FP8 model sits next to the repaired original — delete it once the repaired copy is trusted (43 GB).
• Design/decision record: .claude/memory/router-front-door-plan.md (v3.3).