# vLLM model server — router front door, auto wake-on-request

> History: 2026-08-13 rework + GPU-fault diagnosis (`NOTES-2026-08-13.md`),
> GPU2 re-check + root cause (`NOTES-2026-08-17-gpu2-recheck.md`),
> measured GPU/wake numbers (`CALIBRATION.md`).
> Hardware sanity scripts live in **`diag/`** (`p2p_check.py` is the key one).

Three models are **always available** at one OpenAI-compatible endpoint.
Calling services never need to know whether a model is loaded — the router
front door wakes it on request:

```
http://<host>:8000/v1/...
```

Everything runs inside the Docker stack. No host-side cron, watchers, or
docker commands on the request path.

## Models

| Type | Model name (use in `model` field) | Service / GPUs |
|---|---|---|
| Text / chat | `Qwen3.6-35B-A3B-FP8` (aliases: `qwen`, `text`, …) | `vllm-text`, TP=2, GPU0+1 |
| OCR (vision) | `OvisOCR2` (aliases: `ocr`, `ovis`) | `vllm-ocr`, GPU2 |
| Embeddings | `Qwen3-Embedding-8B` (aliases: `embed`, …) | `vllm-embed`, GPU2 |

`GET /v1/models` lists all three. Model matching is case-insensitive; an
unknown name returns a 404 `model_not_found` (no wake is triggered).

## What a request sees

- **Awake model** → proxied immediately (streaming passes through unbuffered).
- **Sleeping model** → the request is *held* while the router wakes it
  (single-flight: 10 concurrent requests → one wake), then proxied.
- **Wake exceeds the hold deadline** → `503` with a depth-calibrated
  `Retry-After` header AND an OpenAI-style JSON body:

| Waking from | Typical wake | Hold deadline | `Retry-After` | body `sleep_depth` |
|---|---|---|---|---|
| level-1 sleep (weights in RAM) | 2.5–4 s (text), <1 s (small) | 30 s | `10` | `"sleeping"` |
| level-2 offload (weights on NFS) | ~23 s (text), 1–4 s (small) | 180 s | `60` | `"offloaded"` |
| container restart / cold boot | 2–10 min (text, NFS) | 600 s | `600` | `"restarting"` |

```json
{"error": {"type": "model_waking", "code": "model_waking",
           "message": "Model '…' is waking from offload; retry shortly",
           "sleep_depth": "offloaded", "estimated_wake_seconds": 60}}
```

## Idle management (automatic)

| After idle | Action | Effect |
|---|---|---|
| 15 min | sleep (level 1) | GPU freed; weights parked in host RAM |
| 3 h | offload (level 2) | host RAM freed too; ~5 GB/GPU CUDA context remains |

First request after either tier wakes the model transparently. Timers:
`IDLE_SLEEP_MIN` / `IDLE_OFFLOAD_MIN` env on the router service.

## Quick start

```bash
curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.6-35B-A3B-FP8","messages":[{"role":"user","content":"hi"}]}'

curl http://127.0.0.1:8000/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3-Embedding-8B","input":"hello"}'
```

### vllmctl

```bash
./vllmctl status            # per-model state table (sleep depth, activity)
./vllmctl up [MODEL]        # force-wake (HTTP to the admin API — no docker)
./vllmctl sleep [MODEL]     # level-1 sleep now
./vllmctl down [MODEL|all]  # level-2 offload now
./vllmctl list              # models on disk + served
./vllmctl logs [-f] [N] [SVC]
./vllmctl restart SVC
./vllmctl pull <hf-repo>    # download a new model (needs docker)
```

`up`/`down`/`sleep`/`status` are pure HTTP against the admin listener;
routine control never needs docker.

## Architecture

```
callers ──► :8000 ──► router (FastAPI, ./router)      [only public ingress]
                       ├─► vllm-text   GPU0+1 (TP=2, util 0.85, 262K ctx)
                       ├─► vllm-ocr    GPU2   (TP=1, util 0.10)
                       └─► vllm-embed  GPU2   (TP=1, util 0.25)
127.0.0.1:8010 ──► router admin API (vllmctl; not reachable off-host)
127.0.0.1:8001-8003 ──► direct debug access to each vLLM (local users only)
```

- Each vLLM service is always running with sleep mode
  (`--enable-sleep-mode`, `VLLM_SERVER_DEV_MODE=1`). The router owns
  wake/sleep/idle, model routing, streaming proxy, and endpoint hiding
  (dev endpoints and `/admin/*` 404 on the public port; paths are
  normalized against traversal).
- The router is a single process with two listeners (public + admin) on one
  event loop, so admin calls share the request path's locks and depth state.
- Wake-intent is persisted (named volume `router-state`), so a router
  restart mid-wake conservatively completes `reload_weights` before serving
  — an interrupted level-2 wake can never serve garbage.
- All services: `restart: unless-stopped` (host reboots self-heal; requests
  get `"restarting"` 503s during the gap).

### Router tunables

All env-driven with sane defaults — see the table at the top of
`router/config.py` (registry/aliases, listeners, proxy timeouts, wake hold
deadlines, idle tiers, state file). Unit tests: `router/tests/`
(91 tests, no GPU needed).

## Hardware constraints (do not remove)

- GPU P2P DMA **corrupts data** on this host (4/6 pairs; see NOTES).
  `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce` are mandatory on
  every vLLM service. Host-staged copies are verified clean.
- Text spans GPU0+GPU1 exclusively; OCR and embed live on GPU2 (single-GPU
  services never issue P2P). Moving a service = edit its
  `CUDA_VISIBLE_DEVICES` in `compose.yml`, then re-tighten memory slices
  (per-GPU sums ≤ 0.88 including CUDA contexts).
- Measured footprints in `CALIBRATION.md`.

## How docker access works

`renbaibing` is in the `docker` group (since 2026-08-14) — `docker` and
`docker compose` run directly, no sudo. Shells opened **before** the group
change need `sg docker -c "…"` or a fresh login. The old `.runas.py` /
`.user.env` mechanism is obsolete.

## Operations

```bash
docker compose up -d            # whole stack
docker compose restart router   # front door only (state survives)
docker logs vllm-router         # structured wake/sleep/idle events
curl 127.0.0.1:8010/admin/status
```

The pre-router (nginx) stack was removed 2026-08-17 (before `git init`) —
it survives only as a design doc
(`.claude/memory/sleep-mode-implementation-plan.md`). Rollback points are
git commits from 2026-08-17 onward.

## Notes

- `.env` provides `VLLM_VERSION`, `MODEL_ROOT`, `HF_TOKEN` (used by
  compose) plus client-side vars (`OPENAI_BASE_URL` etc.). All per-model
  serving args live in `compose.yml`.
- A `.corrupt-20260817` copy of the FP8 model sits next to the repaired
  original — delete it once the repaired copy is trusted (43 GB).
- Design/decision record: `.claude/memory/router-front-door-plan.md` (v3.3).