bing
80eef4ce6a
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
70 lines
3.5 KiB
Plaintext
70 lines
3.5 KiB
Plaintext
GPU/IOMMU fault report — v2, 2026-08-17
|
|
(supersedes the 2026-08-14 report; the fault has MOVED, not healed)
|
|
|
|
> Three NVIDIA A800-SXM4-80GB on CPU root ports (no PCIe switch):
|
|
> GPU0 = 0000:3d:00.0 serial 1324522063353
|
|
> GPU1 = 0000:63:00.0 serial 1324522063362
|
|
> GPU2 = 0000:ab:00.0 serial 1324522063359
|
|
> Driver 595.71.05, kernel 6.8.0-137-generic (changed from 6.8.0-117 at the
|
|
> Aug-14 reboot), VBIOS 92.00.A4.00.02 on all three.
|
|
|
|
PROBLEM
|
|
|
|
GPU<->GPU P2P DMA corrupts data. Corruption always follows the SOURCE GPU:
|
|
|
|
Source GPU | 2026-08-13 | 2026-08-17 (after reboot + kernel change)
|
|
-----------+------------+-------------
|
|
GPU0 | CORRUPT | CORRUPT
|
|
GPU1 | clean | CORRUPT
|
|
GPU2 | CORRUPT | clean
|
|
|
|
The Aug-14 reboot did not fix anything: GPU0 has been a corrupt source both
|
|
times, and the failure relocated from GPU2's DMA path to GPU1's. Reproducible
|
|
test: /data/home/renbaibing/vllm/diag/p2p_check.py (run inside any CUDA
|
|
container, e.g. vllm/vllm-openai:v0.27.1). Result: 16,777,215 of 16,777,216
|
|
elements wrong on the corrupt paths, bit-stable across runs.
|
|
|
|
ROOT-CAUSE ASSESSMENT (platform translation path, NOT bad GPUs)
|
|
|
|
- All hardware counters clean on all three GPUs: ECC volatile AND aggregate
|
|
zero, zero remapped rows, zero retired pages, links x16 Gen4.
|
|
- Per-GPU H2D/D2H copies and matmul PASS on all three GPUs. Only
|
|
*peer*-addressed DMA corrupts.
|
|
- Deterministic all-elements-wrong corruption = data written to wrong
|
|
addresses (translation fault), not random bit errors from failing silicon.
|
|
- VT-d is ENABLED: 14 active DMAR units, /proc/cmdline has no "iommu=pt" —
|
|
every peer write crosses VT-d translation. There are no PCIe switches, so
|
|
all P2P funnels through the root complex + IOMMU.
|
|
- Kernel evidence, all pointing at GPU0 (3d:00) as the initiator:
|
|
- Jul 6: Xid 31 CE2 MMU faults (FAULT_PDE, VIRT_WRITE) on 3d:00
|
|
- Aug 13: DMAR: [DMA Write NO_PASID] device [3d:00.0] fault reason 0x71
|
|
"Present bit in first-level paging entry is clear" (hundreds)
|
|
- All NVLink links report inActive on all three SXM4 boards (topo shows
|
|
PCIe-only: NODE/SYS). SXM4 boards exist for NVLink — links being down at
|
|
3 GPUs populated is itself worth investigating.
|
|
|
|
ASK (in priority order)
|
|
|
|
1. Decisive test: add "intel_iommu=on iommu=pt" to the kernel command line,
|
|
reboot, re-run diag/p2p_check.py. All 6 ordered pairs must print OK.
|
|
Clean result = root cause proven (VT-d peer-DMA translation).
|
|
2. Investigate why NVLink is inactive (SBIOS setting / baseboard population /
|
|
link training). If NVLink comes up, NCCL stops using the PCIe P2P path
|
|
entirely and this bug stops mattering for our workload.
|
|
3. Check ACS on the three GPU root ports; consider SBIOS/BMC firmware and a
|
|
driver branch matched to kernel 6.8.0-137 (driver + kernel both changed at
|
|
the same reboot that relocated the fault).
|
|
4. Only if corruption survives iommu=pt: slot-swap two GPUs. Corruption
|
|
following the serial = bad board; following the slot = bad fabric.
|
|
Serials are recorded above for that comparison.
|
|
|
|
IMPACT
|
|
|
|
Tensor-parallel inference is unusable across GPUs without the software
|
|
workaround (NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce, i.e. NCCL
|
|
stages through host memory). Measured cost at our TP=2 workload: none
|
|
(~146 tok/s). Single-GPU workloads are unaffected and verified clean.
|
|
Please re-run diag/p2p_check.py after ANY reboot or admin change and record
|
|
the matrix; also re-record "nvidia-smi --query-gpu=index,serial,pci.bus_id"
|
|
to track board vs slot.
|