GPU/IOMMU fault report — v2, 2026-08-17
(supersedes the 2026-08-14 report; the fault has MOVED, not healed)

> Three NVIDIA A800-SXM4-80GB on CPU root ports (no PCIe switch):
>   GPU0 = 0000:3d:00.0  serial 1324522063353
>   GPU1 = 0000:63:00.0  serial 1324522063362
>   GPU2 = 0000:ab:00.0  serial 1324522063359
> Driver 595.71.05, kernel 6.8.0-137-generic (changed from 6.8.0-117 at the
> Aug-14 reboot), VBIOS 92.00.A4.00.02 on all three.

PROBLEM

GPU<->GPU P2P DMA corrupts data. Corruption always follows the SOURCE GPU:

    Source GPU | 2026-08-13 | 2026-08-17 (after reboot + kernel change)
    -----------+------------+-------------
    GPU0       | CORRUPT    | CORRUPT
    GPU1       | clean      | CORRUPT
    GPU2       | CORRUPT    | clean

The Aug-14 reboot did not fix anything: GPU0 has been a corrupt source both
times, and the failure relocated from GPU2's DMA path to GPU1's. Reproducible
test: /data/home/renbaibing/vllm/diag/p2p_check.py (run inside any CUDA
container, e.g. vllm/vllm-openai:v0.27.1). Result: 16,777,215 of 16,777,216
elements wrong on the corrupt paths, bit-stable across runs.

ROOT-CAUSE ASSESSMENT (platform translation path, NOT bad GPUs)

- All hardware counters clean on all three GPUs: ECC volatile AND aggregate
  zero, zero remapped rows, zero retired pages, links x16 Gen4.
- Per-GPU H2D/D2H copies and matmul PASS on all three GPUs. Only
  *peer*-addressed DMA corrupts.
- Deterministic all-elements-wrong corruption = data written to wrong
  addresses (translation fault), not random bit errors from failing silicon.
- VT-d is ENABLED: 14 active DMAR units, /proc/cmdline has no "iommu=pt" —
  every peer write crosses VT-d translation. There are no PCIe switches, so
  all P2P funnels through the root complex + IOMMU.
- Kernel evidence, all pointing at GPU0 (3d:00) as the initiator:
  - Jul 6:  Xid 31 CE2 MMU faults (FAULT_PDE, VIRT_WRITE) on 3d:00
  - Aug 13: DMAR: [DMA Write NO_PASID] device [3d:00.0] fault reason 0x71
    "Present bit in first-level paging entry is clear" (hundreds)
- All NVLink links report inActive on all three SXM4 boards (topo shows
  PCIe-only: NODE/SYS). SXM4 boards exist for NVLink — links being down at
  3 GPUs populated is itself worth investigating.

ASK (in priority order)

1. Decisive test: add "intel_iommu=on iommu=pt" to the kernel command line,
   reboot, re-run diag/p2p_check.py. All 6 ordered pairs must print OK.
   Clean result = root cause proven (VT-d peer-DMA translation).
2. Investigate why NVLink is inactive (SBIOS setting / baseboard population /
   link training). If NVLink comes up, NCCL stops using the PCIe P2P path
   entirely and this bug stops mattering for our workload.
3. Check ACS on the three GPU root ports; consider SBIOS/BMC firmware and a
   driver branch matched to kernel 6.8.0-137 (driver + kernel both changed at
   the same reboot that relocated the fault).
4. Only if corruption survives iommu=pt: slot-swap two GPUs. Corruption
   following the serial = bad board; following the slot = bad fabric.
   Serials are recorded above for that comparison.

IMPACT

Tensor-parallel inference is unusable across GPUs without the software
workaround (NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce, i.e. NCCL
stages through host memory). Measured cost at our TP=2 workload: none
(~146 tok/s). Single-GPU workloads are unaffected and verified clean.
Please re-run diag/p2p_check.py after ANY reboot or admin change and record
the matrix; also re-record "nvidia-smi --query-gpu=index,serial,pci.bus_id"
to track board vs slot.