vllm-frontdoor/NOTES-2026-08-17-gpu2-recheck.md

GPU re-check — 2026-08-17

Follow-up to NOTES-2026-08-13.md. The admin reportedly fixed GPU2; re-ran the same diagnostics (diag/p2p_check.py, diag/gpu_integrity.py, inside vllm/vllm-openai:v0.27.1).

Result

Per-GPU integrity: all three GPUs PASS (H2D/D2H copy fidelity, matmul vs double-precision reference, 20× random-copy stress).

P2P matrix: still 4/6 paths corrupt — the fault MOVED, it did not heal.

Path	2026-08-13	2026-08-17 (2 runs, identical)
0→1	CORRUPT	CORRUPT
0→2	CORRUPT	CORRUPT
1→0	clean	CORRUPT
1→2	clean	CORRUPT
2→0	CORRUPT	OK
2→1	CORRUPT	OK

Before: only GPU1-sourced transfers were clean. Now: only GPU2-sourced transfers are clean; everything leaving GPU0/GPU1 corrupts (~16.7M/16.7M elements wrong, stable across runs).

Interpretation

• GPU2 in isolation is healthy, and its DMA engine is now the most trustworthy on the box.
• The PCIe/IOMMU P2P fault persists host-wide; whatever was changed relocated the corrupting paths instead of fixing them.
• NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce remains mandatory for any multi-GPU (TP>1) workload. Host-staged copies (D2H→H2D) verify clean on all GPUs, which is why TP=2 with P2P disabled has been working.

Consequences for the vLLM stack

• Single-GPU services (OCR, embedding) may run on GPU2 safely — they never issue P2P transfers; weights load H2D and outputs return D2H, both verified clean.
• Recommended placement (plan v3.3): text TP=2 on GPU0+GPU1 (exclusive), OCR + embed on GPU2.

For the admin

The 2026-08-13 report asked to investigate IOMMU/VT-d and the PCIe fabric. The new matrix shows the corrupting paths moved from {0→1, 0→2, 2→1, 2→0} to {0→1, 0→2, 1→0, 1→2} — i.e. all transfers sourced from GPU0 and GPU1 now corrupt, while GPU2-sourced transfers became clean. Reproducible: two consecutive runs of diag/p2p_check.py, identical results.

---

Root-cause analysis (later on 2026-08-17)

The "GPU2 fixed, others broken" reading is an illusion. The GPUs are fine; the host's peer-DMA translation path is what's broken.

The two matrices re-sorted by SOURCE GPU (corruption always follows the source — p2p_check.py uses torch .to(), which issues a source-side copy-engine DMA):

Source GPU	2026-08-13	2026-08-17
GPU0 (3d:00)	CORRUPT	CORRUPT
GPU1 (63:00)	clean	CORRUPT
GPU2 (ab:00)	CORRUPT	clean

• GPU0 is a corrupt source BOTH times. Nothing healed; the Aug-14 reboot (which also bumped kernel 6.8.0-117 → 6.8.0-137) rebuilt the platform state and relocated the failure from GPU2's DMA path to GPU1's. Restarts will keep "moving" the fault like this — that is the signature of a software/platform state problem, not dying silicon.
• Corruption signature is deterministic mistranslation, not hardware noise: 16,777,215 / 16,777,216 elements wrong, bit-stable across runs. Flaky hardware gives random bit errors; wrong-everywhere means wrong addresses.
• Re-verified 2026-08-17: zero ECC (volatile AND aggregate), zero remapped rows, zero retired pages on all three GPUs; all links x16 Gen4; per-GPU H2D/D2H + matmul pass on all three.
• VT-d confirmed ENABLED: 14 active DMAR units under /sys/class/iommu, and /proc/cmdline has NO iommu=pt — every peer write crosses VT-d translation.
• No PLX switches (lspci -t): each GPU sits on its own CPU root port, so all P2P funnels through the root complex + IOMMU.
• The kernel evidence from 08-13 points the same way, and only ever at GPU0: Jul 6 Xid 31 CE2 MMU faults (VIRT_WRITE) on 3d:00; Aug 13 DMAR [DMA Write] fault reason 0x71 on 3d:00.

Conclusion: platform fault in peer-DMA translation — VT-d peer-to-peer mappings interacting with driver 595.71.05 / kernel 6.8.0-137 / SBIOS ACS state. Reboots rebuild per-device translation contexts, which is why the bad set moves between GPUs.

GPU serials (baseline for board-vs-slot tracking)

Bus	Serial	VBIOS
0000:3d:00.0	1324522063353	92.00.A4.00.02
0000:63:00.0	1324522063362	92.00.A4.00.02
0000:ab:00.0	1324522063359	92.00.A4.00.02

If a slot-swap test is ever done: corruption following the serial = bad board; following the bus/slot = bad fabric/root port.

Interconnect primer: NVLink vs PCIe P2P (why this box is stuck on fallback #3)

NVLink is dedicated GPU-to-GPU interconnect hardware (~400 GB/s per GPU on A800), built into the SXM4 module design — in multi-GPU HGX servers it is wired to NVSwitch chips on the baseboard. PCIe P2P DMA (~25 GB/s practical, Gen4 x16) is the standard fallback and is supposed to be fully correct — it works on the vast majority of GPU servers.

This box is unusual twice over:

1. All NVLink links inActive on all three A800-SXM4s. SXM4 exists for NVLink, so "inActive" means not-up, not absent. Candidate causes: only 3 GPUs populated in a 4/8-GPU baseboard (incomplete NVLink topology), NVLink disabled in SBIOS, links not training (bridge/baseboard seating), or a driver reporting quirk. If NVLink can be brought up, NCCL uses it and the corrupt PCIe path becomes irrelevant.
2. The PCIe P2P path itself corrupts — the actual fault diagnosed above.

So NCCL on this box sits on the third fallback:

Path	Bandwidth	Status on this box
NVLink	~400 GB/s	links down
PCIe P2P DMA	~25 GB/s	corrupts data (the fault)
Host-staged D2H→CPU→H2D	~12 GB/s effective	✅ forced by NCCL_P2P_DISABLE=1

Host-staged works because per-GPU H2D/D2H is verified clean on all three GPUs — only peer-addressed DMA mistranslates. Cost at TP=2 is negligible here (~146 tok/s measured).

Fix ladder for the admin (decisive test first)

1. GRUB: intel_iommu=on iommu=pt → reboot → re-run diag/p2p_check.py; all 6 ordered pairs must print OK. A clean result proves the root cause is VT-d peer-DMA translation. This is the decisive experiment.
2. Bring NVLink up (SBIOS/baseboard investigation) — sidesteps PCIe P2P entirely even if (1) cannot fully fix it.
3. Check ACS on the three root ports; SBIOS/BMC firmware updates; driver branch matched to kernel 6.8.0-137.
4. Only if corruption survives iommu=pt: suspect PCIe fabric hardware → slot-swap two GPUs and compare against the serial table above.