# GPU re-check — 2026-08-17

Follow-up to `NOTES-2026-08-13.md`. The admin reportedly fixed GPU2; re-ran
the same diagnostics (`diag/p2p_check.py`, `diag/gpu_integrity.py`, inside
`vllm/vllm-openai:v0.27.1`).

## Result

**Per-GPU integrity: all three GPUs PASS** (H2D/D2H copy fidelity, matmul
vs double-precision reference, 20× random-copy stress).

**P2P matrix: still 4/6 paths corrupt — the fault MOVED, it did not heal.**

| Path | 2026-08-13 | 2026-08-17 (2 runs, identical) |
|---|---|---|
| 0→1 | CORRUPT | CORRUPT |
| 0→2 | CORRUPT | CORRUPT |
| 1→0 | clean | **CORRUPT** |
| 1→2 | clean | **CORRUPT** |
| 2→0 | CORRUPT | **OK** |
| 2→1 | CORRUPT | **OK** |

Before: only GPU1-sourced transfers were clean. Now: only GPU2-sourced
transfers are clean; everything leaving GPU0/GPU1 corrupts
(~16.7M/16.7M elements wrong, stable across runs).

## Interpretation

- GPU2 in isolation is healthy, and its DMA engine is now the most
  trustworthy on the box.
- The PCIe/IOMMU P2P fault persists host-wide; whatever was changed
  relocated the corrupting paths instead of fixing them.
- `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce` remains mandatory for
  any multi-GPU (TP>1) workload. Host-staged copies (D2H→H2D) verify clean
  on all GPUs, which is why TP=2 with P2P disabled has been working.

## Consequences for the vLLM stack

- Single-GPU services (OCR, embedding) may run on **GPU2** safely — they
  never issue P2P transfers; weights load H2D and outputs return D2H, both
  verified clean.
- Recommended placement (plan v3.3): text TP=2 on GPU0+GPU1 (exclusive),
  OCR + embed on GPU2.

## For the admin

The 2026-08-13 report asked to investigate IOMMU/VT-d and the PCIe fabric.
The new matrix shows the corrupting paths moved from {0→1, 0→2, 2→1, 2→0}
to {0→1, 0→2, 1→0, 1→2} — i.e. **all transfers sourced from GPU0 and GPU1
now corrupt, while GPU2-sourced transfers became clean**. Reproducible:
two consecutive runs of `diag/p2p_check.py`, identical results.

---

## Root-cause analysis (later on 2026-08-17)

**The "GPU2 fixed, others broken" reading is an illusion. The GPUs are fine;
the host's peer-DMA translation path is what's broken.**

The two matrices re-sorted by SOURCE GPU (corruption always follows the
source — `p2p_check.py` uses torch `.to()`, which issues a source-side
copy-engine DMA):

| Source GPU | 2026-08-13 | 2026-08-17 |
|---|---|---|
| GPU0 (`3d:00`) | CORRUPT | CORRUPT |
| GPU1 (`63:00`) | clean | **CORRUPT** |
| GPU2 (`ab:00`) | CORRUPT | **clean** |

- GPU0 is a corrupt source BOTH times. Nothing healed; the Aug-14 reboot
  (which also bumped kernel 6.8.0-117 → 6.8.0-137) rebuilt the platform state
  and relocated the failure from GPU2's DMA path to GPU1's. Restarts will keep
  "moving" the fault like this — that is the signature of a software/platform
  state problem, not dying silicon.
- Corruption signature is deterministic mistranslation, not hardware noise:
  16,777,215 / 16,777,216 elements wrong, bit-stable across runs. Flaky
  hardware gives random bit errors; wrong-everywhere means wrong addresses.
- Re-verified 2026-08-17: zero ECC (volatile AND aggregate), zero remapped
  rows, zero retired pages on all three GPUs; all links x16 Gen4; per-GPU
  H2D/D2H + matmul pass on all three.
- **VT-d confirmed ENABLED**: 14 active DMAR units under `/sys/class/iommu`,
  and `/proc/cmdline` has NO `iommu=pt` — every peer write crosses VT-d
  translation.
- No PLX switches (`lspci -t`): each GPU sits on its own CPU root port, so
  all P2P funnels through the root complex + IOMMU.
- The kernel evidence from 08-13 points the same way, and only ever at GPU0:
  Jul 6 `Xid 31` CE2 MMU faults (VIRT_WRITE) on `3d:00`; Aug 13 `DMAR
  [DMA Write] fault reason 0x71` on `3d:00`.

**Conclusion:** platform fault in peer-DMA translation — VT-d peer-to-peer
mappings interacting with driver 595.71.05 / kernel 6.8.0-137 / SBIOS ACS
state. Reboots rebuild per-device translation contexts, which is why the bad
set moves between GPUs.

### GPU serials (baseline for board-vs-slot tracking)

| Bus | Serial | VBIOS |
|---|---|---|
| `0000:3d:00.0` | 1324522063353 | 92.00.A4.00.02 |
| `0000:63:00.0` | 1324522063362 | 92.00.A4.00.02 |
| `0000:ab:00.0` | 1324522063359 | 92.00.A4.00.02 |

If a slot-swap test is ever done: corruption following the **serial** = bad
board; following the **bus/slot** = bad fabric/root port.

## Interconnect primer: NVLink vs PCIe P2P (why this box is stuck on fallback #3)

NVLink is dedicated GPU-to-GPU interconnect hardware (~400 GB/s per GPU on
A800), built into the SXM4 module design — in multi-GPU HGX servers it is
wired to NVSwitch chips on the baseboard. PCIe P2P DMA (~25 GB/s practical,
Gen4 x16) is the standard fallback and is *supposed to be fully correct* —
it works on the vast majority of GPU servers.

This box is unusual twice over:

1. **All NVLink links `inActive` on all three A800-SXM4s.** SXM4 exists for
   NVLink, so "inActive" means not-up, not absent. Candidate causes: only 3
   GPUs populated in a 4/8-GPU baseboard (incomplete NVLink topology), NVLink
   disabled in SBIOS, links not training (bridge/baseboard seating), or a
   driver reporting quirk. **If NVLink can be brought up, NCCL uses it and
   the corrupt PCIe path becomes irrelevant.**
2. **The PCIe P2P path itself corrupts** — the actual fault diagnosed above.

So NCCL on this box sits on the third fallback:

| Path | Bandwidth | Status on this box |
|---|---|---|
| NVLink | ~400 GB/s | links down |
| PCIe P2P DMA | ~25 GB/s | corrupts data (the fault) |
| Host-staged D2H→CPU→H2D | ~12 GB/s effective | ✅ forced by `NCCL_P2P_DISABLE=1` |

Host-staged works because per-GPU H2D/D2H is verified clean on all three
GPUs — only *peer*-addressed DMA mistranslates. Cost at TP=2 is negligible
here (~146 tok/s measured).

## Fix ladder for the admin (decisive test first)

1. **GRUB: `intel_iommu=on iommu=pt`** → reboot → re-run `diag/p2p_check.py`;
   all 6 ordered pairs must print `OK`. A clean result proves the root cause
   is VT-d peer-DMA translation. This is the decisive experiment.
2. **Bring NVLink up** (SBIOS/baseboard investigation) — sidesteps PCIe P2P
   entirely even if (1) cannot fully fix it.
3. Check ACS on the three root ports; SBIOS/BMC firmware updates; driver
   branch matched to kernel 6.8.0-137.
4. Only if corruption survives `iommu=pt`: suspect PCIe fabric hardware →
   slot-swap two GPUs and compare against the serial table above.