Initial commit: router front-door vLLM stack
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
146
NOTES-2026-08-17-gpu2-recheck.md
Normal file
146
NOTES-2026-08-17-gpu2-recheck.md
Normal file
@@ -0,0 +1,146 @@
|
||||
# GPU re-check — 2026-08-17
|
||||
|
||||
Follow-up to `NOTES-2026-08-13.md`. The admin reportedly fixed GPU2; re-ran
|
||||
the same diagnostics (`diag/p2p_check.py`, `diag/gpu_integrity.py`, inside
|
||||
`vllm/vllm-openai:v0.27.1`).
|
||||
|
||||
## Result
|
||||
|
||||
**Per-GPU integrity: all three GPUs PASS** (H2D/D2H copy fidelity, matmul
|
||||
vs double-precision reference, 20× random-copy stress).
|
||||
|
||||
**P2P matrix: still 4/6 paths corrupt — the fault MOVED, it did not heal.**
|
||||
|
||||
| Path | 2026-08-13 | 2026-08-17 (2 runs, identical) |
|
||||
|---|---|---|
|
||||
| 0→1 | CORRUPT | CORRUPT |
|
||||
| 0→2 | CORRUPT | CORRUPT |
|
||||
| 1→0 | clean | **CORRUPT** |
|
||||
| 1→2 | clean | **CORRUPT** |
|
||||
| 2→0 | CORRUPT | **OK** |
|
||||
| 2→1 | CORRUPT | **OK** |
|
||||
|
||||
Before: only GPU1-sourced transfers were clean. Now: only GPU2-sourced
|
||||
transfers are clean; everything leaving GPU0/GPU1 corrupts
|
||||
(~16.7M/16.7M elements wrong, stable across runs).
|
||||
|
||||
## Interpretation
|
||||
|
||||
- GPU2 in isolation is healthy, and its DMA engine is now the most
|
||||
trustworthy on the box.
|
||||
- The PCIe/IOMMU P2P fault persists host-wide; whatever was changed
|
||||
relocated the corrupting paths instead of fixing them.
|
||||
- `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce` remains mandatory for
|
||||
any multi-GPU (TP>1) workload. Host-staged copies (D2H→H2D) verify clean
|
||||
on all GPUs, which is why TP=2 with P2P disabled has been working.
|
||||
|
||||
## Consequences for the vLLM stack
|
||||
|
||||
- Single-GPU services (OCR, embedding) may run on **GPU2** safely — they
|
||||
never issue P2P transfers; weights load H2D and outputs return D2H, both
|
||||
verified clean.
|
||||
- Recommended placement (plan v3.3): text TP=2 on GPU0+GPU1 (exclusive),
|
||||
OCR + embed on GPU2.
|
||||
|
||||
## For the admin
|
||||
|
||||
The 2026-08-13 report asked to investigate IOMMU/VT-d and the PCIe fabric.
|
||||
The new matrix shows the corrupting paths moved from {0→1, 0→2, 2→1, 2→0}
|
||||
to {0→1, 0→2, 1→0, 1→2} — i.e. **all transfers sourced from GPU0 and GPU1
|
||||
now corrupt, while GPU2-sourced transfers became clean**. Reproducible:
|
||||
two consecutive runs of `diag/p2p_check.py`, identical results.
|
||||
|
||||
---
|
||||
|
||||
## Root-cause analysis (later on 2026-08-17)
|
||||
|
||||
**The "GPU2 fixed, others broken" reading is an illusion. The GPUs are fine;
|
||||
the host's peer-DMA translation path is what's broken.**
|
||||
|
||||
The two matrices re-sorted by SOURCE GPU (corruption always follows the
|
||||
source — `p2p_check.py` uses torch `.to()`, which issues a source-side
|
||||
copy-engine DMA):
|
||||
|
||||
| Source GPU | 2026-08-13 | 2026-08-17 |
|
||||
|---|---|---|
|
||||
| GPU0 (`3d:00`) | CORRUPT | CORRUPT |
|
||||
| GPU1 (`63:00`) | clean | **CORRUPT** |
|
||||
| GPU2 (`ab:00`) | CORRUPT | **clean** |
|
||||
|
||||
- GPU0 is a corrupt source BOTH times. Nothing healed; the Aug-14 reboot
|
||||
(which also bumped kernel 6.8.0-117 → 6.8.0-137) rebuilt the platform state
|
||||
and relocated the failure from GPU2's DMA path to GPU1's. Restarts will keep
|
||||
"moving" the fault like this — that is the signature of a software/platform
|
||||
state problem, not dying silicon.
|
||||
- Corruption signature is deterministic mistranslation, not hardware noise:
|
||||
16,777,215 / 16,777,216 elements wrong, bit-stable across runs. Flaky
|
||||
hardware gives random bit errors; wrong-everywhere means wrong addresses.
|
||||
- Re-verified 2026-08-17: zero ECC (volatile AND aggregate), zero remapped
|
||||
rows, zero retired pages on all three GPUs; all links x16 Gen4; per-GPU
|
||||
H2D/D2H + matmul pass on all three.
|
||||
- **VT-d confirmed ENABLED**: 14 active DMAR units under `/sys/class/iommu`,
|
||||
and `/proc/cmdline` has NO `iommu=pt` — every peer write crosses VT-d
|
||||
translation.
|
||||
- No PLX switches (`lspci -t`): each GPU sits on its own CPU root port, so
|
||||
all P2P funnels through the root complex + IOMMU.
|
||||
- The kernel evidence from 08-13 points the same way, and only ever at GPU0:
|
||||
Jul 6 `Xid 31` CE2 MMU faults (VIRT_WRITE) on `3d:00`; Aug 13 `DMAR
|
||||
[DMA Write] fault reason 0x71` on `3d:00`.
|
||||
|
||||
**Conclusion:** platform fault in peer-DMA translation — VT-d peer-to-peer
|
||||
mappings interacting with driver 595.71.05 / kernel 6.8.0-137 / SBIOS ACS
|
||||
state. Reboots rebuild per-device translation contexts, which is why the bad
|
||||
set moves between GPUs.
|
||||
|
||||
### GPU serials (baseline for board-vs-slot tracking)
|
||||
|
||||
| Bus | Serial | VBIOS |
|
||||
|---|---|---|
|
||||
| `0000:3d:00.0` | 1324522063353 | 92.00.A4.00.02 |
|
||||
| `0000:63:00.0` | 1324522063362 | 92.00.A4.00.02 |
|
||||
| `0000:ab:00.0` | 1324522063359 | 92.00.A4.00.02 |
|
||||
|
||||
If a slot-swap test is ever done: corruption following the **serial** = bad
|
||||
board; following the **bus/slot** = bad fabric/root port.
|
||||
|
||||
## Interconnect primer: NVLink vs PCIe P2P (why this box is stuck on fallback #3)
|
||||
|
||||
NVLink is dedicated GPU-to-GPU interconnect hardware (~400 GB/s per GPU on
|
||||
A800), built into the SXM4 module design — in multi-GPU HGX servers it is
|
||||
wired to NVSwitch chips on the baseboard. PCIe P2P DMA (~25 GB/s practical,
|
||||
Gen4 x16) is the standard fallback and is *supposed to be fully correct* —
|
||||
it works on the vast majority of GPU servers.
|
||||
|
||||
This box is unusual twice over:
|
||||
|
||||
1. **All NVLink links `inActive` on all three A800-SXM4s.** SXM4 exists for
|
||||
NVLink, so "inActive" means not-up, not absent. Candidate causes: only 3
|
||||
GPUs populated in a 4/8-GPU baseboard (incomplete NVLink topology), NVLink
|
||||
disabled in SBIOS, links not training (bridge/baseboard seating), or a
|
||||
driver reporting quirk. **If NVLink can be brought up, NCCL uses it and
|
||||
the corrupt PCIe path becomes irrelevant.**
|
||||
2. **The PCIe P2P path itself corrupts** — the actual fault diagnosed above.
|
||||
|
||||
So NCCL on this box sits on the third fallback:
|
||||
|
||||
| Path | Bandwidth | Status on this box |
|
||||
|---|---|---|
|
||||
| NVLink | ~400 GB/s | links down |
|
||||
| PCIe P2P DMA | ~25 GB/s | corrupts data (the fault) |
|
||||
| Host-staged D2H→CPU→H2D | ~12 GB/s effective | ✅ forced by `NCCL_P2P_DISABLE=1` |
|
||||
|
||||
Host-staged works because per-GPU H2D/D2H is verified clean on all three
|
||||
GPUs — only *peer*-addressed DMA mistranslates. Cost at TP=2 is negligible
|
||||
here (~146 tok/s measured).
|
||||
|
||||
## Fix ladder for the admin (decisive test first)
|
||||
|
||||
1. **GRUB: `intel_iommu=on iommu=pt`** → reboot → re-run `diag/p2p_check.py`;
|
||||
all 6 ordered pairs must print `OK`. A clean result proves the root cause
|
||||
is VT-d peer-DMA translation. This is the decisive experiment.
|
||||
2. **Bring NVLink up** (SBIOS/baseboard investigation) — sidesteps PCIe P2P
|
||||
entirely even if (1) cannot fully fix it.
|
||||
3. Check ACS on the three root ports; SBIOS/BMC firmware updates; driver
|
||||
branch matched to kernel 6.8.0-137.
|
||||
4. Only if corruption survives `iommu=pt`: suspect PCIe fabric hardware →
|
||||
slot-swap two GPUs and compare against the serial table above.
|
||||
Reference in New Issue
Block a user