# vLLM rework — 2026-08-13

Complete record of what was changed on this box today, why, the diagnosis of
the broken inference, and what the system admin needs to know about the GPUs.

---

## 1. Starting point

- One compose-managed container `vllm` (image `vllm/vllm-openai:latest`,
  actually **v0.22.0**, built 2026-05-29), serving
  `Qwen3.6-35B-A3B` (67 GB, hybrid MoE + linear-attention "GDN" model) from
  NFS (`/data/home/renbaibing/huggingface`) with `--tensor-parallel-size 2`
  on GPUs 0+1, port 8000, 262144 context.
- The compose file hardcoded that one model (volume + full command), so
  switching models meant hand-editing YAML, and the container sat on both
  GPUs permanently (`restart: unless-stopped`) even though requests are rare.
- User `renbaibing` has no docker socket access; docker rights live with
  user `x640` (sudo group). Credentials for that were provided in `.user.env`.

## 2. What was built

### Files

| File | Purpose |
|---|---|
| `compose.yml` | Rewritten: model-agnostic. Mounts the whole model root `/data/home/renbaibing/huggingface` read-only at `/models`; serves `/models/${MODEL_NAME}`. Image tag, TP size, context length, GPU mem fraction and extra flags all env-driven. Adds a docker healthcheck on `/health`. Includes the P2P workaround env (`NCCL_P2P_DISABLE=1`, see §4). |
| `.env` | All serving knobs: `MODEL_NAME`, `MODEL_ROOT`, `VLLM_PORT`, `TP_SIZE`, `MAX_MODEL_LEN`, `GPU_MEM_UTIL`, `EXTRA_ARGS`, `VLLM_VERSION` (image pin), `NCCL_P2P_DISABLE`. |
| `vllmctl` | Management CLI (bash). Subcommands below. Runs docker as `x640` via the pty helper. |
| `.runas.py` | Helper that runs a command as `x640` (`su` + `sudo -S`) over a pty, answering the password prompts from `.user.env`, keeping prompt text out of command output. |
| `.user.env` | `DOCKER_USER` / `DOCKER_USER_PASSWORD` for x640. **chmod 600.** |
| `README.md` | Usage guide. |
| `diag/p2p_check.py` | Pairwise GPU P2P corruption test (the decisive hardware test — §4). |
| `diag/gpu_integrity.py` | Per-GPU H2D/D2H + matmul sanity. |
| `diag/weight_check.py` | Full model tensor finiteness scan. |
| `idle-watch.log`, `.idle.pid`, `.idle.minutes` | Runtime state of the idle watcher. |

### `vllmctl` subcommands

```
list                 models on disk + what is loaded + sizes
up [MODEL]           load/serve MODEL (recreates container when switching)
down                 stop container → GPU memory freed in ~1 s
restart [MODEL]      down + up
status               container/health, served model, GPU memory, watcher
logs [-f] [N]        container logs
pull HF_REPO [NAME]  download a model into MODEL_ROOT (uses HF_TOKEN from .env)
idle-watch on [MIN]  daemon: auto-`down` after MIN min without requests (default 15)
idle-watch off|status
```

The idle watcher polls `/metrics` every 30 s (`vllm:num_requests_running` /
`num_requests_waiting`, falling back to its own activity tracking on older
vLLM builds that lack `time_since_last_request_seconds`). It keeps watching
after an unload, so later manual `up`s are guarded too.

### Docker access

Every docker command runs as `x640` via `sudo -S` driven through the pty
helper (plain `sudo` refuses without a terminal; `sudo -S` works). If an
admin ever runs `sudo usermod -aG docker renbaibing`, the helper can be
retired by changing `dexec()` in `vllmctl` to a plain call.

## 3. Upgrade v0.22.0 → v0.27.1 (pinned)

- `compose.yml` now uses `vllm/vllm-openai:${VLLM_VERSION:-v0.27.1}`;
  `.env` pins `VLLM_VERSION=v0.27.1` (image pulled 2026-08-13, 30.8 GB).
- Reason: v0.22's vendored FLA/GDN Triton kernels for this hybrid model have
  known bug classes upstream, and v0.27.1's hybrid-cache handling is much
  further along. (The NaN we saw turned out to be hardware — §4 — but the
  newer image is the right baseline anyway.)
- Rollback: set `VLLM_VERSION=latest` (the old v0.22 image is still on disk)
  and `./vllmctl restart`.
- Flag note: `--enable-auto-tool-choice`, `--tool-call-parser qwen3_coder`,
  `--reasoning-parser qwen3` all still accepted by v0.27.1.
  `VLLM_DISABLE_CUSTOM_ALL_REDUCE` is **not** a valid env var in v0.27 —
  use the `--disable-custom-all-reduce` flag (in `EXTRA_ARGS`) instead.

## 4. The broken-inference diagnosis (root cause: GPU P2P DMA corruption)

### Symptoms after recreating the container

- Every request returned `!!!!!…` garbage; logits contained NaN
  (`Out of range float values are not JSON compliant: nan`).
- Generation crawled at ~0.25 tok/s with GPUs at "100 % util" but only
  ~50 W draw (spin-wait, not compute).
- On v0.27.1, startup warmup hung indefinitely; worker processes eventually
  died silently (zombies) while their GPU kernels kept spinning.

### Hypotheses ruled out

| Hypothesis | Evidence against |
|---|---|
| My compose changes broke it | Same image sha (`0fec7ec5…`), identical final argv in logs, same model path |
| Corrupt weights on NFS | `diag/weight_check.py`: all 26 shards, 1045 tensors, zero NaN/Inf |
| GPU compute/H2D-D2H faults | `diag/gpu_integrity.py`: all 3 GPUs pass copies + matmul; volatile ECC counters 0; no retired pages |
| vLLM compiled-kernel bug | `--enforce-eager` produced identical garbage |
| Host load / NFS | Host idle (load ~1 on 128 cores), NFS read 6 GB/s cached |

### The decisive test — `diag/p2p_check.py`

Host→GPU→GPU→host round-trip per ordered pair (16 M floats each):

| Transfer | Result |
|---|---|
| GPU0→GPU1 | **corrupt — 16,777,215 / 16,777,216 elements wrong** |
| GPU1→GPU0 | clean |
| GPU1→GPU2 | clean |
| GPU2→GPU1 | **corrupt (all elements)** |
| GPU0→GPU2 | **corrupt (all elements)** |
| GPU2→GPU0 | **corrupt (all elements)** |

Tensor parallel does an all-reduce across GPU0↔GPU1 on every layer — with
that path corrupt, activations turn to NaN and collectives hang. That single
finding explains the garbage, the 0.25 tok/s spin, and the wedged warmups.

### Corroborating kernel-log evidence (`sudo dmesg -T`)

- **2026-07-06**: cluster of `NVRM: Xid (PCI:0000:3d:00): 31 … MMU Fault:
  ENGINE CE2 … FAULT_PDE ACCESS_TYPE_VIRT_WRITE` — copy-engine DMA faults on
  **GPU0**. This coincides with when the model stopped giving good replies
  (last good use: July).
- **2026-08-13 09:58:30** (during vLLM warmup): `DMAR: [DMA Write NO_PASID]
  Request device [3d:00.0] fault addr 0xccfff000 [fault reason 0x71] SM:
  Present bit in first-level paging entry is clear` (+ repeats, "122
  callbacks suppressed") — IOMMU rejecting GPU0 DMA writes.
- **2026-07-28**: `nvidia-persistenced` (re)started — someone already
  serviced the NVIDIA stack after the July faults.
- **2026-08-11**: `nvidia 0000:ab:00.0: Using 47-bit DMA addresses` — GPU2
  was re-probed.
- `nvidia-smi nvlink -e`: *all NVLink links inActive* on all three A800s
  (SXM4 boards — NVLink should normally be up).
- `nvidia-smi topo -m`: GPU0↔GPU1 connected via `NODE` (PCIe through host
  bridges), no `NV#` links.

### Workaround in place

In `compose.yml` / `.env`:

```
NCCL_P2P_DISABLE=1
EXTRA_ARGS=... --disable-custom-all-reduce
```

i.e. NCCL is forbidden from using direct GPU↔GPU DMA (it stages reductions
through host shared memory) and vLLM's own P2P-based custom all-reduce is
off. Performance impact is negligible here: measured **~146 tok/s** decode
with the workaround vs 0.25 tok/s with corruption.

**Do not remove these two settings until the admin has fixed/re-verified the
machines's P2P paths** (re-run `diag/p2p_check.py` after any reboot/repair;
all pairs must report `OK`).

## 5. Current state (end of day)

- Container `vllm`: **v0.27.1**, healthy, serving `Qwen3.6-35B-A3B` on
  GPUs 0+1, port 8000. Verified: correct completions, correct chat +
  reasoning output, ~146 tok/s.
- Idle watcher **armed at 15 minutes** (auto-unloads the model when nobody
  is using it; `./vllmctl idle-watch off` to disable).
- GPU2 remains untouched/free (4 MiB used).
- Weights verified intact; image pinned; rollback path documented (§3).

## 6. For the system admin — GPU/IOMMU fault report

> Three NVIDIA A800-SXM4-80GB (PCI `3d:00`, `63:00`, `ab:00`), driver
> 595.71.05, host up since ~2026-06-02.
>
> **Problem:** GPU↔GPU P2P DMA corrupts data on 4 of 6 ordered pairs
> (only transfers *from* GPU1 are clean). Reproducible test:
> `/data/home/renbaibing/vllm/diag/p2p_check.py` (needs torch + GPUs, e.g.
> run inside any CUDA container). Result today: 16,777,215 of 16,777,216
> elements wrong on 0→1, 2→1, 0→2, 2→0.
>
> **Kernel evidence:**
> - Jul 6: repeated `Xid 31` CE2 MMU faults (FAULT_PDE, VIRT_WRITE) on GPU0
>   (`0000:3d:00`), pids 1461777–1538048.
> - Aug 13: `DMAR: [DMA Write NO_PASID] Request device [3d:00.0] …
>   fault reason 0x71: SM: Present bit in first-level paging entry is clear`
>   (hundreds of faults, "122 callbacks suppressed").
> - All NVLinks report `inActive` (`nvidia-smi nvlink -e`); topology shows
>   PCIe-only (`NODE`) between GPUs.
> - Aug 11: GPU2 (`ab:00`) was re-probed by the driver.
> - Jul 28: nvidia-persistenced was (re)started.
>
> **Impact:** tensor-parallel inference across GPUs is unusable without
> disabling P2P; we work around it in software (host-staged NCCL), at no
> measurable speed loss for our workload, but the underlying fault remains.
>
> **Ask:** investigate IOMMU/VT-d state and the PCIe fabric for GPU0
> (and GPU2), check whether NVLink should be active on these boards,
> consider a reboot to clear IOMMU state, then re-run the p2p_check script.
> If corruption persists after reboot, it points at hardware (PCIe path or
> GPU0 itself).

## 7. Loose ends / ideas

- If an admin adds `renbaibing` to the `docker` group, simplify `vllmctl`
  (`dexec()` → plain call) and delete `.runas.py`/`.user.env`.
- The idle watcher is a plain `nohup` process; it does **not** survive a
  host reboot. If wanted permanently, wrap it in a systemd user unit.
- A wake-on-request reverse proxy (port 8000 → auto-`up` on first request)
  would make unloads fully transparent to clients; not built today since
  requests are rare and manual `up` takes a few minutes anyway (NFS weight
  load dominates).
- `Qwen3.6-35B-A3B` officially targets Hopper-class GPUs per vLLM recipes;
  it works on these A800s but has no bundled A800 MoE tuning configs
  ("Using default MoE config" warning) — expect merely good, not peak, speed.
- `friendly_hertz` / `funny_shamir` are two ancient OnlyOffice containers —
  unrelated, left untouched.