bing
80eef4ce6a
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
157 lines
6.5 KiB
Markdown
157 lines
6.5 KiB
Markdown
# vLLM model server — router front door, auto wake-on-request
|
||
|
||
> History: 2026-08-13 rework + GPU-fault diagnosis (`NOTES-2026-08-13.md`),
|
||
> GPU2 re-check + root cause (`NOTES-2026-08-17-gpu2-recheck.md`),
|
||
> measured GPU/wake numbers (`CALIBRATION.md`).
|
||
> Hardware sanity scripts live in **`diag/`** (`p2p_check.py` is the key one).
|
||
|
||
Three models are **always available** at one OpenAI-compatible endpoint.
|
||
Calling services never need to know whether a model is loaded — the router
|
||
front door wakes it on request:
|
||
|
||
```
|
||
http://<host>:8000/v1/...
|
||
```
|
||
|
||
Everything runs inside the Docker stack. No host-side cron, watchers, or
|
||
docker commands on the request path.
|
||
|
||
## Models
|
||
|
||
| Type | Model name (use in `model` field) | Service / GPUs |
|
||
|---|---|---|
|
||
| Text / chat | `Qwen3.6-35B-A3B-FP8` (aliases: `qwen`, `text`, …) | `vllm-text`, TP=2, GPU0+1 |
|
||
| OCR (vision) | `OvisOCR2` (aliases: `ocr`, `ovis`) | `vllm-ocr`, GPU2 |
|
||
| Embeddings | `Qwen3-Embedding-8B` (aliases: `embed`, …) | `vllm-embed`, GPU2 |
|
||
|
||
`GET /v1/models` lists all three. Model matching is case-insensitive; an
|
||
unknown name returns a 404 `model_not_found` (no wake is triggered).
|
||
|
||
## What a request sees
|
||
|
||
- **Awake model** → proxied immediately (streaming passes through unbuffered).
|
||
- **Sleeping model** → the request is *held* while the router wakes it
|
||
(single-flight: 10 concurrent requests → one wake), then proxied.
|
||
- **Wake exceeds the hold deadline** → `503` with a depth-calibrated
|
||
`Retry-After` header AND an OpenAI-style JSON body:
|
||
|
||
| Waking from | Typical wake | Hold deadline | `Retry-After` | body `sleep_depth` |
|
||
|---|---|---|---|---|
|
||
| level-1 sleep (weights in RAM) | 2.5–4 s (text), <1 s (small) | 30 s | `10` | `"sleeping"` |
|
||
| level-2 offload (weights on NFS) | ~23 s (text), 1–4 s (small) | 180 s | `60` | `"offloaded"` |
|
||
| container restart / cold boot | 2–10 min (text, NFS) | 600 s | `600` | `"restarting"` |
|
||
|
||
```json
|
||
{"error": {"type": "model_waking", "code": "model_waking",
|
||
"message": "Model '…' is waking from offload; retry shortly",
|
||
"sleep_depth": "offloaded", "estimated_wake_seconds": 60}}
|
||
```
|
||
|
||
## Idle management (automatic)
|
||
|
||
| After idle | Action | Effect |
|
||
|---|---|---|
|
||
| 15 min | sleep (level 1) | GPU freed; weights parked in host RAM |
|
||
| 3 h | offload (level 2) | host RAM freed too; ~5 GB/GPU CUDA context remains |
|
||
|
||
First request after either tier wakes the model transparently. Timers:
|
||
`IDLE_SLEEP_MIN` / `IDLE_OFFLOAD_MIN` env on the router service.
|
||
|
||
## Quick start
|
||
|
||
```bash
|
||
curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
|
||
-d '{"model":"Qwen3.6-35B-A3B-FP8","messages":[{"role":"user","content":"hi"}]}'
|
||
|
||
curl http://127.0.0.1:8000/v1/embeddings -H 'Content-Type: application/json' \
|
||
-d '{"model":"Qwen3-Embedding-8B","input":"hello"}'
|
||
```
|
||
|
||
### vllmctl
|
||
|
||
```bash
|
||
./vllmctl status # per-model state table (sleep depth, activity)
|
||
./vllmctl up [MODEL] # force-wake (HTTP to the admin API — no docker)
|
||
./vllmctl sleep [MODEL] # level-1 sleep now
|
||
./vllmctl down [MODEL|all] # level-2 offload now
|
||
./vllmctl list # models on disk + served
|
||
./vllmctl logs [-f] [N] [SVC]
|
||
./vllmctl restart SVC
|
||
./vllmctl pull <hf-repo> # download a new model (needs docker)
|
||
```
|
||
|
||
`up`/`down`/`sleep`/`status` are pure HTTP against the admin listener;
|
||
routine control never needs docker.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
callers ──► :8000 ──► router (FastAPI, ./router) [only public ingress]
|
||
├─► vllm-text GPU0+1 (TP=2, util 0.85, 262K ctx)
|
||
├─► vllm-ocr GPU2 (TP=1, util 0.10)
|
||
└─► vllm-embed GPU2 (TP=1, util 0.25)
|
||
127.0.0.1:8010 ──► router admin API (vllmctl; not reachable off-host)
|
||
127.0.0.1:8001-8003 ──► direct debug access to each vLLM (local users only)
|
||
```
|
||
|
||
- Each vLLM service is always running with sleep mode
|
||
(`--enable-sleep-mode`, `VLLM_SERVER_DEV_MODE=1`). The router owns
|
||
wake/sleep/idle, model routing, streaming proxy, and endpoint hiding
|
||
(dev endpoints and `/admin/*` 404 on the public port; paths are
|
||
normalized against traversal).
|
||
- The router is a single process with two listeners (public + admin) on one
|
||
event loop, so admin calls share the request path's locks and depth state.
|
||
- Wake-intent is persisted (named volume `router-state`), so a router
|
||
restart mid-wake conservatively completes `reload_weights` before serving
|
||
— an interrupted level-2 wake can never serve garbage.
|
||
- All services: `restart: unless-stopped` (host reboots self-heal; requests
|
||
get `"restarting"` 503s during the gap).
|
||
|
||
### Router tunables
|
||
|
||
All env-driven with sane defaults — see the table at the top of
|
||
`router/config.py` (registry/aliases, listeners, proxy timeouts, wake hold
|
||
deadlines, idle tiers, state file). Unit tests: `router/tests/`
|
||
(91 tests, no GPU needed).
|
||
|
||
## Hardware constraints (do not remove)
|
||
|
||
- GPU P2P DMA **corrupts data** on this host (4/6 pairs; see NOTES).
|
||
`NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce` are mandatory on
|
||
every vLLM service. Host-staged copies are verified clean.
|
||
- Text spans GPU0+GPU1 exclusively; OCR and embed live on GPU2 (single-GPU
|
||
services never issue P2P). Moving a service = edit its
|
||
`CUDA_VISIBLE_DEVICES` in `compose.yml`, then re-tighten memory slices
|
||
(per-GPU sums ≤ 0.88 including CUDA contexts).
|
||
- Measured footprints in `CALIBRATION.md`.
|
||
|
||
## How docker access works
|
||
|
||
`renbaibing` is in the `docker` group (since 2026-08-14) — `docker` and
|
||
`docker compose` run directly, no sudo. Shells opened **before** the group
|
||
change need `sg docker -c "…"` or a fresh login. The old `.runas.py` /
|
||
`.user.env` mechanism is obsolete.
|
||
|
||
## Operations
|
||
|
||
```bash
|
||
docker compose up -d # whole stack
|
||
docker compose restart router # front door only (state survives)
|
||
docker logs vllm-router # structured wake/sleep/idle events
|
||
curl 127.0.0.1:8010/admin/status
|
||
```
|
||
|
||
The pre-router (nginx) stack was removed 2026-08-17 (before `git init`) —
|
||
it survives only as a design doc
|
||
(`.claude/memory/sleep-mode-implementation-plan.md`). Rollback points are
|
||
git commits from 2026-08-17 onward.
|
||
|
||
## Notes
|
||
|
||
- `.env` provides `VLLM_VERSION`, `MODEL_ROOT`, `HF_TOKEN` (used by
|
||
compose) plus client-side vars (`OPENAI_BASE_URL` etc.). All per-model
|
||
serving args live in `compose.yml`.
|
||
- A `.corrupt-20260817` copy of the FP8 model sits next to the repaired
|
||
original — delete it once the repaired copy is trusted (43 GB).
|
||
- Design/decision record: `.claude/memory/router-front-door-plan.md` (v3.3).
|