bing
80eef4ce6a
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
211 lines
10 KiB
Markdown
211 lines
10 KiB
Markdown
# vLLM rework — 2026-08-13
|
||
|
||
Complete record of what was changed on this box today, why, the diagnosis of
|
||
the broken inference, and what the system admin needs to know about the GPUs.
|
||
|
||
---
|
||
|
||
## 1. Starting point
|
||
|
||
- One compose-managed container `vllm` (image `vllm/vllm-openai:latest`,
|
||
actually **v0.22.0**, built 2026-05-29), serving
|
||
`Qwen3.6-35B-A3B` (67 GB, hybrid MoE + linear-attention "GDN" model) from
|
||
NFS (`/data/home/renbaibing/huggingface`) with `--tensor-parallel-size 2`
|
||
on GPUs 0+1, port 8000, 262144 context.
|
||
- The compose file hardcoded that one model (volume + full command), so
|
||
switching models meant hand-editing YAML, and the container sat on both
|
||
GPUs permanently (`restart: unless-stopped`) even though requests are rare.
|
||
- User `renbaibing` has no docker socket access; docker rights live with
|
||
user `x640` (sudo group). Credentials for that were provided in `.user.env`.
|
||
|
||
## 2. What was built
|
||
|
||
### Files
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `compose.yml` | Rewritten: model-agnostic. Mounts the whole model root `/data/home/renbaibing/huggingface` read-only at `/models`; serves `/models/${MODEL_NAME}`. Image tag, TP size, context length, GPU mem fraction and extra flags all env-driven. Adds a docker healthcheck on `/health`. Includes the P2P workaround env (`NCCL_P2P_DISABLE=1`, see §4). |
|
||
| `.env` | All serving knobs: `MODEL_NAME`, `MODEL_ROOT`, `VLLM_PORT`, `TP_SIZE`, `MAX_MODEL_LEN`, `GPU_MEM_UTIL`, `EXTRA_ARGS`, `VLLM_VERSION` (image pin), `NCCL_P2P_DISABLE`. |
|
||
| `vllmctl` | Management CLI (bash). Subcommands below. Runs docker as `x640` via the pty helper. |
|
||
| `.runas.py` | Helper that runs a command as `x640` (`su` + `sudo -S`) over a pty, answering the password prompts from `.user.env`, keeping prompt text out of command output. |
|
||
| `.user.env` | `DOCKER_USER` / `DOCKER_USER_PASSWORD` for x640. **chmod 600.** |
|
||
| `README.md` | Usage guide. |
|
||
| `diag/p2p_check.py` | Pairwise GPU P2P corruption test (the decisive hardware test — §4). |
|
||
| `diag/gpu_integrity.py` | Per-GPU H2D/D2H + matmul sanity. |
|
||
| `diag/weight_check.py` | Full model tensor finiteness scan. |
|
||
| `idle-watch.log`, `.idle.pid`, `.idle.minutes` | Runtime state of the idle watcher. |
|
||
|
||
### `vllmctl` subcommands
|
||
|
||
```
|
||
list models on disk + what is loaded + sizes
|
||
up [MODEL] load/serve MODEL (recreates container when switching)
|
||
down stop container → GPU memory freed in ~1 s
|
||
restart [MODEL] down + up
|
||
status container/health, served model, GPU memory, watcher
|
||
logs [-f] [N] container logs
|
||
pull HF_REPO [NAME] download a model into MODEL_ROOT (uses HF_TOKEN from .env)
|
||
idle-watch on [MIN] daemon: auto-`down` after MIN min without requests (default 15)
|
||
idle-watch off|status
|
||
```
|
||
|
||
The idle watcher polls `/metrics` every 30 s (`vllm:num_requests_running` /
|
||
`num_requests_waiting`, falling back to its own activity tracking on older
|
||
vLLM builds that lack `time_since_last_request_seconds`). It keeps watching
|
||
after an unload, so later manual `up`s are guarded too.
|
||
|
||
### Docker access
|
||
|
||
Every docker command runs as `x640` via `sudo -S` driven through the pty
|
||
helper (plain `sudo` refuses without a terminal; `sudo -S` works). If an
|
||
admin ever runs `sudo usermod -aG docker renbaibing`, the helper can be
|
||
retired by changing `dexec()` in `vllmctl` to a plain call.
|
||
|
||
## 3. Upgrade v0.22.0 → v0.27.1 (pinned)
|
||
|
||
- `compose.yml` now uses `vllm/vllm-openai:${VLLM_VERSION:-v0.27.1}`;
|
||
`.env` pins `VLLM_VERSION=v0.27.1` (image pulled 2026-08-13, 30.8 GB).
|
||
- Reason: v0.22's vendored FLA/GDN Triton kernels for this hybrid model have
|
||
known bug classes upstream, and v0.27.1's hybrid-cache handling is much
|
||
further along. (The NaN we saw turned out to be hardware — §4 — but the
|
||
newer image is the right baseline anyway.)
|
||
- Rollback: set `VLLM_VERSION=latest` (the old v0.22 image is still on disk)
|
||
and `./vllmctl restart`.
|
||
- Flag note: `--enable-auto-tool-choice`, `--tool-call-parser qwen3_coder`,
|
||
`--reasoning-parser qwen3` all still accepted by v0.27.1.
|
||
`VLLM_DISABLE_CUSTOM_ALL_REDUCE` is **not** a valid env var in v0.27 —
|
||
use the `--disable-custom-all-reduce` flag (in `EXTRA_ARGS`) instead.
|
||
|
||
## 4. The broken-inference diagnosis (root cause: GPU P2P DMA corruption)
|
||
|
||
### Symptoms after recreating the container
|
||
|
||
- Every request returned `!!!!!…` garbage; logits contained NaN
|
||
(`Out of range float values are not JSON compliant: nan`).
|
||
- Generation crawled at ~0.25 tok/s with GPUs at "100 % util" but only
|
||
~50 W draw (spin-wait, not compute).
|
||
- On v0.27.1, startup warmup hung indefinitely; worker processes eventually
|
||
died silently (zombies) while their GPU kernels kept spinning.
|
||
|
||
### Hypotheses ruled out
|
||
|
||
| Hypothesis | Evidence against |
|
||
|---|---|
|
||
| My compose changes broke it | Same image sha (`0fec7ec5…`), identical final argv in logs, same model path |
|
||
| Corrupt weights on NFS | `diag/weight_check.py`: all 26 shards, 1045 tensors, zero NaN/Inf |
|
||
| GPU compute/H2D-D2H faults | `diag/gpu_integrity.py`: all 3 GPUs pass copies + matmul; volatile ECC counters 0; no retired pages |
|
||
| vLLM compiled-kernel bug | `--enforce-eager` produced identical garbage |
|
||
| Host load / NFS | Host idle (load ~1 on 128 cores), NFS read 6 GB/s cached |
|
||
|
||
### The decisive test — `diag/p2p_check.py`
|
||
|
||
Host→GPU→GPU→host round-trip per ordered pair (16 M floats each):
|
||
|
||
| Transfer | Result |
|
||
|---|---|
|
||
| GPU0→GPU1 | **corrupt — 16,777,215 / 16,777,216 elements wrong** |
|
||
| GPU1→GPU0 | clean |
|
||
| GPU1→GPU2 | clean |
|
||
| GPU2→GPU1 | **corrupt (all elements)** |
|
||
| GPU0→GPU2 | **corrupt (all elements)** |
|
||
| GPU2→GPU0 | **corrupt (all elements)** |
|
||
|
||
Tensor parallel does an all-reduce across GPU0↔GPU1 on every layer — with
|
||
that path corrupt, activations turn to NaN and collectives hang. That single
|
||
finding explains the garbage, the 0.25 tok/s spin, and the wedged warmups.
|
||
|
||
### Corroborating kernel-log evidence (`sudo dmesg -T`)
|
||
|
||
- **2026-07-06**: cluster of `NVRM: Xid (PCI:0000:3d:00): 31 … MMU Fault:
|
||
ENGINE CE2 … FAULT_PDE ACCESS_TYPE_VIRT_WRITE` — copy-engine DMA faults on
|
||
**GPU0**. This coincides with when the model stopped giving good replies
|
||
(last good use: July).
|
||
- **2026-08-13 09:58:30** (during vLLM warmup): `DMAR: [DMA Write NO_PASID]
|
||
Request device [3d:00.0] fault addr 0xccfff000 [fault reason 0x71] SM:
|
||
Present bit in first-level paging entry is clear` (+ repeats, "122
|
||
callbacks suppressed") — IOMMU rejecting GPU0 DMA writes.
|
||
- **2026-07-28**: `nvidia-persistenced` (re)started — someone already
|
||
serviced the NVIDIA stack after the July faults.
|
||
- **2026-08-11**: `nvidia 0000:ab:00.0: Using 47-bit DMA addresses` — GPU2
|
||
was re-probed.
|
||
- `nvidia-smi nvlink -e`: *all NVLink links inActive* on all three A800s
|
||
(SXM4 boards — NVLink should normally be up).
|
||
- `nvidia-smi topo -m`: GPU0↔GPU1 connected via `NODE` (PCIe through host
|
||
bridges), no `NV#` links.
|
||
|
||
### Workaround in place
|
||
|
||
In `compose.yml` / `.env`:
|
||
|
||
```
|
||
NCCL_P2P_DISABLE=1
|
||
EXTRA_ARGS=... --disable-custom-all-reduce
|
||
```
|
||
|
||
i.e. NCCL is forbidden from using direct GPU↔GPU DMA (it stages reductions
|
||
through host shared memory) and vLLM's own P2P-based custom all-reduce is
|
||
off. Performance impact is negligible here: measured **~146 tok/s** decode
|
||
with the workaround vs 0.25 tok/s with corruption.
|
||
|
||
**Do not remove these two settings until the admin has fixed/re-verified the
|
||
machines's P2P paths** (re-run `diag/p2p_check.py` after any reboot/repair;
|
||
all pairs must report `OK`).
|
||
|
||
## 5. Current state (end of day)
|
||
|
||
- Container `vllm`: **v0.27.1**, healthy, serving `Qwen3.6-35B-A3B` on
|
||
GPUs 0+1, port 8000. Verified: correct completions, correct chat +
|
||
reasoning output, ~146 tok/s.
|
||
- Idle watcher **armed at 15 minutes** (auto-unloads the model when nobody
|
||
is using it; `./vllmctl idle-watch off` to disable).
|
||
- GPU2 remains untouched/free (4 MiB used).
|
||
- Weights verified intact; image pinned; rollback path documented (§3).
|
||
|
||
## 6. For the system admin — GPU/IOMMU fault report
|
||
|
||
> Three NVIDIA A800-SXM4-80GB (PCI `3d:00`, `63:00`, `ab:00`), driver
|
||
> 595.71.05, host up since ~2026-06-02.
|
||
>
|
||
> **Problem:** GPU↔GPU P2P DMA corrupts data on 4 of 6 ordered pairs
|
||
> (only transfers *from* GPU1 are clean). Reproducible test:
|
||
> `/data/home/renbaibing/vllm/diag/p2p_check.py` (needs torch + GPUs, e.g.
|
||
> run inside any CUDA container). Result today: 16,777,215 of 16,777,216
|
||
> elements wrong on 0→1, 2→1, 0→2, 2→0.
|
||
>
|
||
> **Kernel evidence:**
|
||
> - Jul 6: repeated `Xid 31` CE2 MMU faults (FAULT_PDE, VIRT_WRITE) on GPU0
|
||
> (`0000:3d:00`), pids 1461777–1538048.
|
||
> - Aug 13: `DMAR: [DMA Write NO_PASID] Request device [3d:00.0] …
|
||
> fault reason 0x71: SM: Present bit in first-level paging entry is clear`
|
||
> (hundreds of faults, "122 callbacks suppressed").
|
||
> - All NVLinks report `inActive` (`nvidia-smi nvlink -e`); topology shows
|
||
> PCIe-only (`NODE`) between GPUs.
|
||
> - Aug 11: GPU2 (`ab:00`) was re-probed by the driver.
|
||
> - Jul 28: nvidia-persistenced was (re)started.
|
||
>
|
||
> **Impact:** tensor-parallel inference across GPUs is unusable without
|
||
> disabling P2P; we work around it in software (host-staged NCCL), at no
|
||
> measurable speed loss for our workload, but the underlying fault remains.
|
||
>
|
||
> **Ask:** investigate IOMMU/VT-d state and the PCIe fabric for GPU0
|
||
> (and GPU2), check whether NVLink should be active on these boards,
|
||
> consider a reboot to clear IOMMU state, then re-run the p2p_check script.
|
||
> If corruption persists after reboot, it points at hardware (PCIe path or
|
||
> GPU0 itself).
|
||
|
||
## 7. Loose ends / ideas
|
||
|
||
- If an admin adds `renbaibing` to the `docker` group, simplify `vllmctl`
|
||
(`dexec()` → plain call) and delete `.runas.py`/`.user.env`.
|
||
- The idle watcher is a plain `nohup` process; it does **not** survive a
|
||
host reboot. If wanted permanently, wrap it in a systemd user unit.
|
||
- A wake-on-request reverse proxy (port 8000 → auto-`up` on first request)
|
||
would make unloads fully transparent to clients; not built today since
|
||
requests are rare and manual `up` takes a few minutes anyway (NFS weight
|
||
load dominates).
|
||
- `Qwen3.6-35B-A3B` officially targets Hopper-class GPUs per vLLM recipes;
|
||
it works on these A800s but has no bundled A800 MoE tuning configs
|
||
("Using default MoE config" warning) — expect merely good, not peak, speed.
|
||
- `friendly_hertz` / `funny_shamir` are two ancient OnlyOffice containers —
|
||
unrelated, left untouched.
|