vllm-frontdoor/NOTES-2026-08-13.md

vLLM rework — 2026-08-13

Complete record of what was changed on this box today, why, the diagnosis of the broken inference, and what the system admin needs to know about the GPUs.

---

1. Starting point

• One compose-managed container vllm (image vllm/vllm-openai:latest, actually v0.22.0, built 2026-05-29), serving Qwen3.6-35B-A3B (67 GB, hybrid MoE + linear-attention "GDN" model) from NFS (/data/home/renbaibing/huggingface) with --tensor-parallel-size 2 on GPUs 0+1, port 8000, 262144 context.
• The compose file hardcoded that one model (volume + full command), so switching models meant hand-editing YAML, and the container sat on both GPUs permanently (restart: unless-stopped) even though requests are rare.
• User renbaibing has no docker socket access; docker rights live with user x640 (sudo group). Credentials for that were provided in .user.env.

2. What was built

Files

File	Purpose
compose.yml	Rewritten: model-agnostic. Mounts the whole model root /data/home/renbaibing/huggingface read-only at /models; serves /models/${MODEL_NAME}. Image tag, TP size, context length, GPU mem fraction and extra flags all env-driven. Adds a docker healthcheck on /health. Includes the P2P workaround env (NCCL_P2P_DISABLE=1, see §4).
.env	All serving knobs: MODEL_NAME, MODEL_ROOT, VLLM_PORT, TP_SIZE, MAX_MODEL_LEN, GPU_MEM_UTIL, EXTRA_ARGS, VLLM_VERSION (image pin), NCCL_P2P_DISABLE.
vllmctl	Management CLI (bash). Subcommands below. Runs docker as x640 via the pty helper.
.runas.py	Helper that runs a command as x640 (su + sudo -S) over a pty, answering the password prompts from .user.env, keeping prompt text out of command output.
.user.env	DOCKER_USER / DOCKER_USER_PASSWORD for x640. chmod 600.
README.md	Usage guide.
diag/p2p_check.py	Pairwise GPU P2P corruption test (the decisive hardware test — §4).
diag/gpu_integrity.py	Per-GPU H2D/D2H + matmul sanity.
diag/weight_check.py	Full model tensor finiteness scan.
idle-watch.log, .idle.pid, .idle.minutes	Runtime state of the idle watcher.

vllmctl subcommands

list                 models on disk + what is loaded + sizes
up [MODEL]           load/serve MODEL (recreates container when switching)
down                 stop container → GPU memory freed in ~1 s
restart [MODEL]      down + up
status               container/health, served model, GPU memory, watcher
logs [-f] [N]        container logs
pull HF_REPO [NAME]  download a model into MODEL_ROOT (uses HF_TOKEN from .env)
idle-watch on [MIN]  daemon: auto-`down` after MIN min without requests (default 15)
idle-watch off|status

The idle watcher polls /metrics every 30 s (vllm:num_requests_running / num_requests_waiting, falling back to its own activity tracking on older vLLM builds that lack time_since_last_request_seconds). It keeps watching after an unload, so later manual ups are guarded too.

Docker access

Every docker command runs as x640 via sudo -S driven through the pty helper (plain sudo refuses without a terminal; sudo -S works). If an admin ever runs sudo usermod -aG docker renbaibing, the helper can be retired by changing dexec() in vllmctl to a plain call.

3. Upgrade v0.22.0 → v0.27.1 (pinned)

• compose.yml now uses vllm/vllm-openai:${VLLM_VERSION:-v0.27.1}; .env pins VLLM_VERSION=v0.27.1 (image pulled 2026-08-13, 30.8 GB).
• Reason: v0.22's vendored FLA/GDN Triton kernels for this hybrid model have known bug classes upstream, and v0.27.1's hybrid-cache handling is much further along. (The NaN we saw turned out to be hardware — §4 — but the newer image is the right baseline anyway.)
• Rollback: set VLLM_VERSION=latest (the old v0.22 image is still on disk) and ./vllmctl restart.
• Flag note: --enable-auto-tool-choice, --tool-call-parser qwen3_coder, --reasoning-parser qwen3 all still accepted by v0.27.1. VLLM_DISABLE_CUSTOM_ALL_REDUCE is not a valid env var in v0.27 — use the --disable-custom-all-reduce flag (in EXTRA_ARGS) instead.

4. The broken-inference diagnosis (root cause: GPU P2P DMA corruption)

Symptoms after recreating the container

• Every request returned !!!!!… garbage; logits contained NaN (Out of range float values are not JSON compliant: nan).
• Generation crawled at ~0.25 tok/s with GPUs at "100 % util" but only ~50 W draw (spin-wait, not compute).
• On v0.27.1, startup warmup hung indefinitely; worker processes eventually died silently (zombies) while their GPU kernels kept spinning.

Hypotheses ruled out

Hypothesis	Evidence against
My compose changes broke it	Same image sha (0fec7ec5…), identical final argv in logs, same model path
Corrupt weights on NFS	diag/weight_check.py: all 26 shards, 1045 tensors, zero NaN/Inf
GPU compute/H2D-D2H faults	diag/gpu_integrity.py: all 3 GPUs pass copies + matmul; volatile ECC counters 0; no retired pages
vLLM compiled-kernel bug	--enforce-eager produced identical garbage
Host load / NFS	Host idle (load ~1 on 128 cores), NFS read 6 GB/s cached

The decisive test — diag/p2p_check.py

Host→GPU→GPU→host round-trip per ordered pair (16 M floats each):

Transfer	Result
GPU0→GPU1	corrupt — 16,777,215 / 16,777,216 elements wrong
GPU1→GPU0	clean
GPU1→GPU2	clean
GPU2→GPU1	corrupt (all elements)
GPU0→GPU2	corrupt (all elements)
GPU2→GPU0	corrupt (all elements)

Tensor parallel does an all-reduce across GPU0↔GPU1 on every layer — with that path corrupt, activations turn to NaN and collectives hang. That single finding explains the garbage, the 0.25 tok/s spin, and the wedged warmups.

Corroborating kernel-log evidence (sudo dmesg -T)

• 2026-07-06: cluster of NVRM: Xid (PCI:0000:3d:00): 31 … MMU Fault: ENGINE CE2 … FAULT_PDE ACCESS_TYPE_VIRT_WRITE — copy-engine DMA faults on GPU0. This coincides with when the model stopped giving good replies (last good use: July).
• 2026-08-13 09:58:30 (during vLLM warmup): DMAR: [DMA Write NO_PASID] Request device [3d:00.0] fault addr 0xccfff000 [fault reason 0x71] SM: Present bit in first-level paging entry is clear (+ repeats, "122 callbacks suppressed") — IOMMU rejecting GPU0 DMA writes.
• 2026-07-28: nvidia-persistenced (re)started — someone already serviced the NVIDIA stack after the July faults.
• 2026-08-11: nvidia 0000:ab:00.0: Using 47-bit DMA addresses — GPU2 was re-probed.
• nvidia-smi nvlink -e: all NVLink links inActive on all three A800s (SXM4 boards — NVLink should normally be up).
• nvidia-smi topo -m: GPU0↔GPU1 connected via NODE (PCIe through host bridges), no NV# links.

Workaround in place

In compose.yml / .env:

NCCL_P2P_DISABLE=1
EXTRA_ARGS=... --disable-custom-all-reduce

i.e. NCCL is forbidden from using direct GPU↔GPU DMA (it stages reductions through host shared memory) and vLLM's own P2P-based custom all-reduce is off. Performance impact is negligible here: measured ~146 tok/s decode with the workaround vs 0.25 tok/s with corruption.

Do not remove these two settings until the admin has fixed/re-verified the machines's P2P paths (re-run diag/p2p_check.py after any reboot/repair; all pairs must report OK).

5. Current state (end of day)

• Container vllm: v0.27.1, healthy, serving Qwen3.6-35B-A3B on GPUs 0+1, port 8000. Verified: correct completions, correct chat + reasoning output, ~146 tok/s.
• Idle watcher armed at 15 minutes (auto-unloads the model when nobody is using it; ./vllmctl idle-watch off to disable).
• GPU2 remains untouched/free (4 MiB used).
• Weights verified intact; image pinned; rollback path documented (§3).

6. For the system admin — GPU/IOMMU fault report

Three NVIDIA A800-SXM4-80GB (PCI 3d:00, 63:00, ab:00), driver 595.71.05, host up since ~2026-06-02.

Problem: GPU↔GPU P2P DMA corrupts data on 4 of 6 ordered pairs (only transfers from GPU1 are clean). Reproducible test: /data/home/renbaibing/vllm/diag/p2p_check.py (needs torch + GPUs, e.g. run inside any CUDA container). Result today: 16,777,215 of 16,777,216 elements wrong on 0→1, 2→1, 0→2, 2→0.

Kernel evidence:

• Jul 6: repeated Xid 31 CE2 MMU faults (FAULT_PDE, VIRT_WRITE) on GPU0 (0000:3d:00), pids 1461777–1538048.
• Aug 13: DMAR: [DMA Write NO_PASID] Request device [3d:00.0] … fault reason 0x71: SM: Present bit in first-level paging entry is clear (hundreds of faults, "122 callbacks suppressed").
• All NVLinks report inActive (nvidia-smi nvlink -e); topology shows PCIe-only (NODE) between GPUs.
• Aug 11: GPU2 (ab:00) was re-probed by the driver.
• Jul 28: nvidia-persistenced was (re)started.

Impact: tensor-parallel inference across GPUs is unusable without disabling P2P; we work around it in software (host-staged NCCL), at no measurable speed loss for our workload, but the underlying fault remains.

Ask: investigate IOMMU/VT-d state and the PCIe fabric for GPU0 (and GPU2), check whether NVLink should be active on these boards, consider a reboot to clear IOMMU state, then re-run the p2p_check script. If corruption persists after reboot, it points at hardware (PCIe path or GPU0 itself).

7. Loose ends / ideas

• If an admin adds renbaibing to the docker group, simplify vllmctl (dexec() → plain call) and delete .runas.py/.user.env.
• The idle watcher is a plain nohup process; it does not survive a host reboot. If wanted permanently, wrap it in a systemd user unit.
• A wake-on-request reverse proxy (port 8000 → auto-up on first request) would make unloads fully transparent to clients; not built today since requests are rare and manual up takes a few minutes anyway (NFS weight load dominates).
• Qwen3.6-35B-A3B officially targets Hopper-class GPUs per vLLM recipes; it works on these A800s but has no bundled A800 MoE tuning configs ("Using default MoE config" warning) — expect merely good, not peak, speed.
• friendly_hertz / funny_shamir are two ancient OnlyOffice containers — unrelated, left untouched.