bing
80eef4ce6a
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
147 lines
6.4 KiB
Markdown
147 lines
6.4 KiB
Markdown
# GPU re-check — 2026-08-17
|
||
|
||
Follow-up to `NOTES-2026-08-13.md`. The admin reportedly fixed GPU2; re-ran
|
||
the same diagnostics (`diag/p2p_check.py`, `diag/gpu_integrity.py`, inside
|
||
`vllm/vllm-openai:v0.27.1`).
|
||
|
||
## Result
|
||
|
||
**Per-GPU integrity: all three GPUs PASS** (H2D/D2H copy fidelity, matmul
|
||
vs double-precision reference, 20× random-copy stress).
|
||
|
||
**P2P matrix: still 4/6 paths corrupt — the fault MOVED, it did not heal.**
|
||
|
||
| Path | 2026-08-13 | 2026-08-17 (2 runs, identical) |
|
||
|---|---|---|
|
||
| 0→1 | CORRUPT | CORRUPT |
|
||
| 0→2 | CORRUPT | CORRUPT |
|
||
| 1→0 | clean | **CORRUPT** |
|
||
| 1→2 | clean | **CORRUPT** |
|
||
| 2→0 | CORRUPT | **OK** |
|
||
| 2→1 | CORRUPT | **OK** |
|
||
|
||
Before: only GPU1-sourced transfers were clean. Now: only GPU2-sourced
|
||
transfers are clean; everything leaving GPU0/GPU1 corrupts
|
||
(~16.7M/16.7M elements wrong, stable across runs).
|
||
|
||
## Interpretation
|
||
|
||
- GPU2 in isolation is healthy, and its DMA engine is now the most
|
||
trustworthy on the box.
|
||
- The PCIe/IOMMU P2P fault persists host-wide; whatever was changed
|
||
relocated the corrupting paths instead of fixing them.
|
||
- `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce` remains mandatory for
|
||
any multi-GPU (TP>1) workload. Host-staged copies (D2H→H2D) verify clean
|
||
on all GPUs, which is why TP=2 with P2P disabled has been working.
|
||
|
||
## Consequences for the vLLM stack
|
||
|
||
- Single-GPU services (OCR, embedding) may run on **GPU2** safely — they
|
||
never issue P2P transfers; weights load H2D and outputs return D2H, both
|
||
verified clean.
|
||
- Recommended placement (plan v3.3): text TP=2 on GPU0+GPU1 (exclusive),
|
||
OCR + embed on GPU2.
|
||
|
||
## For the admin
|
||
|
||
The 2026-08-13 report asked to investigate IOMMU/VT-d and the PCIe fabric.
|
||
The new matrix shows the corrupting paths moved from {0→1, 0→2, 2→1, 2→0}
|
||
to {0→1, 0→2, 1→0, 1→2} — i.e. **all transfers sourced from GPU0 and GPU1
|
||
now corrupt, while GPU2-sourced transfers became clean**. Reproducible:
|
||
two consecutive runs of `diag/p2p_check.py`, identical results.
|
||
|
||
---
|
||
|
||
## Root-cause analysis (later on 2026-08-17)
|
||
|
||
**The "GPU2 fixed, others broken" reading is an illusion. The GPUs are fine;
|
||
the host's peer-DMA translation path is what's broken.**
|
||
|
||
The two matrices re-sorted by SOURCE GPU (corruption always follows the
|
||
source — `p2p_check.py` uses torch `.to()`, which issues a source-side
|
||
copy-engine DMA):
|
||
|
||
| Source GPU | 2026-08-13 | 2026-08-17 |
|
||
|---|---|---|
|
||
| GPU0 (`3d:00`) | CORRUPT | CORRUPT |
|
||
| GPU1 (`63:00`) | clean | **CORRUPT** |
|
||
| GPU2 (`ab:00`) | CORRUPT | **clean** |
|
||
|
||
- GPU0 is a corrupt source BOTH times. Nothing healed; the Aug-14 reboot
|
||
(which also bumped kernel 6.8.0-117 → 6.8.0-137) rebuilt the platform state
|
||
and relocated the failure from GPU2's DMA path to GPU1's. Restarts will keep
|
||
"moving" the fault like this — that is the signature of a software/platform
|
||
state problem, not dying silicon.
|
||
- Corruption signature is deterministic mistranslation, not hardware noise:
|
||
16,777,215 / 16,777,216 elements wrong, bit-stable across runs. Flaky
|
||
hardware gives random bit errors; wrong-everywhere means wrong addresses.
|
||
- Re-verified 2026-08-17: zero ECC (volatile AND aggregate), zero remapped
|
||
rows, zero retired pages on all three GPUs; all links x16 Gen4; per-GPU
|
||
H2D/D2H + matmul pass on all three.
|
||
- **VT-d confirmed ENABLED**: 14 active DMAR units under `/sys/class/iommu`,
|
||
and `/proc/cmdline` has NO `iommu=pt` — every peer write crosses VT-d
|
||
translation.
|
||
- No PLX switches (`lspci -t`): each GPU sits on its own CPU root port, so
|
||
all P2P funnels through the root complex + IOMMU.
|
||
- The kernel evidence from 08-13 points the same way, and only ever at GPU0:
|
||
Jul 6 `Xid 31` CE2 MMU faults (VIRT_WRITE) on `3d:00`; Aug 13 `DMAR
|
||
[DMA Write] fault reason 0x71` on `3d:00`.
|
||
|
||
**Conclusion:** platform fault in peer-DMA translation — VT-d peer-to-peer
|
||
mappings interacting with driver 595.71.05 / kernel 6.8.0-137 / SBIOS ACS
|
||
state. Reboots rebuild per-device translation contexts, which is why the bad
|
||
set moves between GPUs.
|
||
|
||
### GPU serials (baseline for board-vs-slot tracking)
|
||
|
||
| Bus | Serial | VBIOS |
|
||
|---|---|---|
|
||
| `0000:3d:00.0` | 1324522063353 | 92.00.A4.00.02 |
|
||
| `0000:63:00.0` | 1324522063362 | 92.00.A4.00.02 |
|
||
| `0000:ab:00.0` | 1324522063359 | 92.00.A4.00.02 |
|
||
|
||
If a slot-swap test is ever done: corruption following the **serial** = bad
|
||
board; following the **bus/slot** = bad fabric/root port.
|
||
|
||
## Interconnect primer: NVLink vs PCIe P2P (why this box is stuck on fallback #3)
|
||
|
||
NVLink is dedicated GPU-to-GPU interconnect hardware (~400 GB/s per GPU on
|
||
A800), built into the SXM4 module design — in multi-GPU HGX servers it is
|
||
wired to NVSwitch chips on the baseboard. PCIe P2P DMA (~25 GB/s practical,
|
||
Gen4 x16) is the standard fallback and is *supposed to be fully correct* —
|
||
it works on the vast majority of GPU servers.
|
||
|
||
This box is unusual twice over:
|
||
|
||
1. **All NVLink links `inActive` on all three A800-SXM4s.** SXM4 exists for
|
||
NVLink, so "inActive" means not-up, not absent. Candidate causes: only 3
|
||
GPUs populated in a 4/8-GPU baseboard (incomplete NVLink topology), NVLink
|
||
disabled in SBIOS, links not training (bridge/baseboard seating), or a
|
||
driver reporting quirk. **If NVLink can be brought up, NCCL uses it and
|
||
the corrupt PCIe path becomes irrelevant.**
|
||
2. **The PCIe P2P path itself corrupts** — the actual fault diagnosed above.
|
||
|
||
So NCCL on this box sits on the third fallback:
|
||
|
||
| Path | Bandwidth | Status on this box |
|
||
|---|---|---|
|
||
| NVLink | ~400 GB/s | links down |
|
||
| PCIe P2P DMA | ~25 GB/s | corrupts data (the fault) |
|
||
| Host-staged D2H→CPU→H2D | ~12 GB/s effective | ✅ forced by `NCCL_P2P_DISABLE=1` |
|
||
|
||
Host-staged works because per-GPU H2D/D2H is verified clean on all three
|
||
GPUs — only *peer*-addressed DMA mistranslates. Cost at TP=2 is negligible
|
||
here (~146 tok/s measured).
|
||
|
||
## Fix ladder for the admin (decisive test first)
|
||
|
||
1. **GRUB: `intel_iommu=on iommu=pt`** → reboot → re-run `diag/p2p_check.py`;
|
||
all 6 ordered pairs must print `OK`. A clean result proves the root cause
|
||
is VT-d peer-DMA translation. This is the decisive experiment.
|
||
2. **Bring NVLink up** (SBIOS/baseboard investigation) — sidesteps PCIe P2P
|
||
entirely even if (1) cannot fully fix it.
|
||
3. Check ACS on the three root ports; SBIOS/BMC firmware updates; driver
|
||
branch matched to kernel 6.8.0-137.
|
||
4. Only if corruption survives `iommu=pt`: suspect PCIe fabric hardware →
|
||
slot-swap two GPUs and compare against the serial table above.
|