Initial commit: router front-door vLLM stack

Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:17:42 +00:00
commit 80eef4ce6a
35 changed files with 6506 additions and 0 deletions
--- a/NOTES-2026-08-13.md
+++ b/NOTES-2026-08-13.md
@@ -0,0 +1,210 @@
+# vLLM rework — 2026-08-13
+
+Complete record of what was changed on this box today, why, the diagnosis of
+the broken inference, and what the system admin needs to know about the GPUs.
+
+---
+
+## 1. Starting point
+
+- One compose-managed container `vllm` (image `vllm/vllm-openai:latest`,
+  actually **v0.22.0**, built 2026-05-29), serving
+  `Qwen3.6-35B-A3B` (67 GB, hybrid MoE + linear-attention "GDN" model) from
+  NFS (`/data/home/renbaibing/huggingface`) with `--tensor-parallel-size 2`
+  on GPUs 0+1, port 8000, 262144 context.
+- The compose file hardcoded that one model (volume + full command), so
+  switching models meant hand-editing YAML, and the container sat on both
+  GPUs permanently (`restart: unless-stopped`) even though requests are rare.
+- User `renbaibing` has no docker socket access; docker rights live with
+  user `x640` (sudo group). Credentials for that were provided in `.user.env`.
+
+## 2. What was built
+
+### Files
+
+| File | Purpose |
+|---|---|
+| `compose.yml` | Rewritten: model-agnostic. Mounts the whole model root `/data/home/renbaibing/huggingface` read-only at `/models`; serves `/models/${MODEL_NAME}`. Image tag, TP size, context length, GPU mem fraction and extra flags all env-driven. Adds a docker healthcheck on `/health`. Includes the P2P workaround env (`NCCL_P2P_DISABLE=1`, see §4). |
+| `.env` | All serving knobs: `MODEL_NAME`, `MODEL_ROOT`, `VLLM_PORT`, `TP_SIZE`, `MAX_MODEL_LEN`, `GPU_MEM_UTIL`, `EXTRA_ARGS`, `VLLM_VERSION` (image pin), `NCCL_P2P_DISABLE`. |
+| `vllmctl` | Management CLI (bash). Subcommands below. Runs docker as `x640` via the pty helper. |
+| `.runas.py` | Helper that runs a command as `x640` (`su` + `sudo -S`) over a pty, answering the password prompts from `.user.env`, keeping prompt text out of command output. |
+| `.user.env` | `DOCKER_USER` / `DOCKER_USER_PASSWORD` for x640. **chmod 600.** |
+| `README.md` | Usage guide. |
+| `diag/p2p_check.py` | Pairwise GPU P2P corruption test (the decisive hardware test — §4). |
+| `diag/gpu_integrity.py` | Per-GPU H2D/D2H + matmul sanity. |
+| `diag/weight_check.py` | Full model tensor finiteness scan. |
+| `idle-watch.log`, `.idle.pid`, `.idle.minutes` | Runtime state of the idle watcher. |
+
+### `vllmctl` subcommands
+
+```
+list                 models on disk + what is loaded + sizes
+up [MODEL]           load/serve MODEL (recreates container when switching)
+down                 stop container → GPU memory freed in ~1 s
+restart [MODEL]      down + up
+status               container/health, served model, GPU memory, watcher
+logs [-f] [N]        container logs
+pull HF_REPO [NAME]  download a model into MODEL_ROOT (uses HF_TOKEN from .env)
+idle-watch on [MIN]  daemon: auto-`down` after MIN min without requests (default 15)
+idle-watch off|status
+```
+
+The idle watcher polls `/metrics` every 30 s (`vllm:num_requests_running` /
+`num_requests_waiting`, falling back to its own activity tracking on older
+vLLM builds that lack `time_since_last_request_seconds`). It keeps watching
+after an unload, so later manual `up`s are guarded too.
+
+### Docker access
+
+Every docker command runs as `x640` via `sudo -S` driven through the pty
+helper (plain `sudo` refuses without a terminal; `sudo -S` works). If an
+admin ever runs `sudo usermod -aG docker renbaibing`, the helper can be
+retired by changing `dexec()` in `vllmctl` to a plain call.
+
+## 3. Upgrade v0.22.0 → v0.27.1 (pinned)
+
+- `compose.yml` now uses `vllm/vllm-openai:${VLLM_VERSION:-v0.27.1}`;
+  `.env` pins `VLLM_VERSION=v0.27.1` (image pulled 2026-08-13, 30.8 GB).
+- Reason: v0.22's vendored FLA/GDN Triton kernels for this hybrid model have
+  known bug classes upstream, and v0.27.1's hybrid-cache handling is much
+  further along. (The NaN we saw turned out to be hardware — §4 — but the
+  newer image is the right baseline anyway.)
+- Rollback: set `VLLM_VERSION=latest` (the old v0.22 image is still on disk)
+  and `./vllmctl restart`.
+- Flag note: `--enable-auto-tool-choice`, `--tool-call-parser qwen3_coder`,
+  `--reasoning-parser qwen3` all still accepted by v0.27.1.
+  `VLLM_DISABLE_CUSTOM_ALL_REDUCE` is **not** a valid env var in v0.27 —
+  use the `--disable-custom-all-reduce` flag (in `EXTRA_ARGS`) instead.
+
+## 4. The broken-inference diagnosis (root cause: GPU P2P DMA corruption)
+
+### Symptoms after recreating the container
+
+- Every request returned `!!!!!…` garbage; logits contained NaN
+  (`Out of range float values are not JSON compliant: nan`).
+- Generation crawled at ~0.25 tok/s with GPUs at "100 % util" but only
+  ~50 W draw (spin-wait, not compute).
+- On v0.27.1, startup warmup hung indefinitely; worker processes eventually
+  died silently (zombies) while their GPU kernels kept spinning.
+
+### Hypotheses ruled out
+
+| Hypothesis | Evidence against |
+|---|---|
+| My compose changes broke it | Same image sha (`0fec7ec5…`), identical final argv in logs, same model path |
+| Corrupt weights on NFS | `diag/weight_check.py`: all 26 shards, 1045 tensors, zero NaN/Inf |
+| GPU compute/H2D-D2H faults | `diag/gpu_integrity.py`: all 3 GPUs pass copies + matmul; volatile ECC counters 0; no retired pages |
+| vLLM compiled-kernel bug | `--enforce-eager` produced identical garbage |
+| Host load / NFS | Host idle (load ~1 on 128 cores), NFS read 6 GB/s cached |
+
+### The decisive test — `diag/p2p_check.py`
+
+Host→GPU→GPU→host round-trip per ordered pair (16 M floats each):
+
+| Transfer | Result |
+|---|---|
+| GPU0→GPU1 | **corrupt — 16,777,215 / 16,777,216 elements wrong** |
+| GPU1→GPU0 | clean |
+| GPU1→GPU2 | clean |
+| GPU2→GPU1 | **corrupt (all elements)** |
+| GPU0→GPU2 | **corrupt (all elements)** |
+| GPU2→GPU0 | **corrupt (all elements)** |
+
+Tensor parallel does an all-reduce across GPU0↔GPU1 on every layer — with
+that path corrupt, activations turn to NaN and collectives hang. That single
+finding explains the garbage, the 0.25 tok/s spin, and the wedged warmups.
+
+### Corroborating kernel-log evidence (`sudo dmesg -T`)
+
+- **2026-07-06**: cluster of `NVRM: Xid (PCI:0000:3d:00): 31 … MMU Fault:
+  ENGINE CE2 … FAULT_PDE ACCESS_TYPE_VIRT_WRITE` — copy-engine DMA faults on
+  **GPU0**. This coincides with when the model stopped giving good replies
+  (last good use: July).
+- **2026-08-13 09:58:30** (during vLLM warmup): `DMAR: [DMA Write NO_PASID]
+  Request device [3d:00.0] fault addr 0xccfff000 [fault reason 0x71] SM:
+  Present bit in first-level paging entry is clear` (+ repeats, "122
+  callbacks suppressed") — IOMMU rejecting GPU0 DMA writes.
+- **2026-07-28**: `nvidia-persistenced` (re)started — someone already
+  serviced the NVIDIA stack after the July faults.
+- **2026-08-11**: `nvidia 0000:ab:00.0: Using 47-bit DMA addresses` — GPU2
+  was re-probed.
+- `nvidia-smi nvlink -e`: *all NVLink links inActive* on all three A800s
+  (SXM4 boards — NVLink should normally be up).
+- `nvidia-smi topo -m`: GPU0↔GPU1 connected via `NODE` (PCIe through host
+  bridges), no `NV#` links.
+
+### Workaround in place
+
+In `compose.yml` / `.env`:
+
+```
+NCCL_P2P_DISABLE=1
+EXTRA_ARGS=... --disable-custom-all-reduce
+```
+
+i.e. NCCL is forbidden from using direct GPU↔GPU DMA (it stages reductions
+through host shared memory) and vLLM's own P2P-based custom all-reduce is
+off. Performance impact is negligible here: measured **~146 tok/s** decode
+with the workaround vs 0.25 tok/s with corruption.
+
+**Do not remove these two settings until the admin has fixed/re-verified the
+machines's P2P paths** (re-run `diag/p2p_check.py` after any reboot/repair;
+all pairs must report `OK`).
+
+## 5. Current state (end of day)
+
+- Container `vllm`: **v0.27.1**, healthy, serving `Qwen3.6-35B-A3B` on
+  GPUs 0+1, port 8000. Verified: correct completions, correct chat +
+  reasoning output, ~146 tok/s.
+- Idle watcher **armed at 15 minutes** (auto-unloads the model when nobody
+  is using it; `./vllmctl idle-watch off` to disable).
+- GPU2 remains untouched/free (4 MiB used).
+- Weights verified intact; image pinned; rollback path documented (§3).
+
+## 6. For the system admin — GPU/IOMMU fault report
+
+> Three NVIDIA A800-SXM4-80GB (PCI `3d:00`, `63:00`, `ab:00`), driver
+> 595.71.05, host up since ~2026-06-02.
+>
+> **Problem:** GPU↔GPU P2P DMA corrupts data on 4 of 6 ordered pairs
+> (only transfers *from* GPU1 are clean). Reproducible test:
+> `/data/home/renbaibing/vllm/diag/p2p_check.py` (needs torch + GPUs, e.g.
+> run inside any CUDA container). Result today: 16,777,215 of 16,777,216
+> elements wrong on 0→1, 2→1, 0→2, 2→0.
+>
+> **Kernel evidence:**
+> - Jul 6: repeated `Xid 31` CE2 MMU faults (FAULT_PDE, VIRT_WRITE) on GPU0
+>   (`0000:3d:00`), pids 1461777–1538048.
+> - Aug 13: `DMAR: [DMA Write NO_PASID] Request device [3d:00.0] …
+>   fault reason 0x71: SM: Present bit in first-level paging entry is clear`
+>   (hundreds of faults, "122 callbacks suppressed").
+> - All NVLinks report `inActive` (`nvidia-smi nvlink -e`); topology shows
+>   PCIe-only (`NODE`) between GPUs.
+> - Aug 11: GPU2 (`ab:00`) was re-probed by the driver.
+> - Jul 28: nvidia-persistenced was (re)started.
+>
+> **Impact:** tensor-parallel inference across GPUs is unusable without
+> disabling P2P; we work around it in software (host-staged NCCL), at no
+> measurable speed loss for our workload, but the underlying fault remains.
+>
+> **Ask:** investigate IOMMU/VT-d state and the PCIe fabric for GPU0
+> (and GPU2), check whether NVLink should be active on these boards,
+> consider a reboot to clear IOMMU state, then re-run the p2p_check script.
+> If corruption persists after reboot, it points at hardware (PCIe path or
+> GPU0 itself).
+
+## 7. Loose ends / ideas
+
+- If an admin adds `renbaibing` to the `docker` group, simplify `vllmctl`
+  (`dexec()` → plain call) and delete `.runas.py`/`.user.env`.
+- The idle watcher is a plain `nohup` process; it does **not** survive a
+  host reboot. If wanted permanently, wrap it in a systemd user unit.
+- A wake-on-request reverse proxy (port 8000 → auto-`up` on first request)
+  would make unloads fully transparent to clients; not built today since
+  requests are rare and manual `up` takes a few minutes anyway (NFS weight
+  load dominates).
+- `Qwen3.6-35B-A3B` officially targets Hopper-class GPUs per vLLM recipes;
+  it works on these A800s but has no bundled A800 MoE tuning configs
+  ("Using default MoE config" warning) — expect merely good, not peak, speed.
+- `friendly_hertz` / `funny_shamir` are two ancient OnlyOffice containers —
+  unrelated, left untouched.