Initial commit: router front-door vLLM stack

Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2,
sleep mode) behind a FastAPI router that auto-wakes models on request.
Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with
Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010.
Routine control via vllmctl is pure HTTP — no docker on the request path.

Verified: 91 router unit tests + 15-test E2E on real hardware
(measurements in CALIBRATION.md; design record in
.claude/memory/router-front-door-plan.md).

Old nginx stack files removed before git init; design survives in
.claude/memory/sleep-mode-implementation-plan.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-17 10:17:42 +00:00
commit 80eef4ce6a
35 changed files with 6506 additions and 0 deletions

33
diag/gpu_integrity.py Normal file
View File

@@ -0,0 +1,33 @@
#!/usr/bin/env python3
"""Per-GPU basic integrity: H2D/D2H copy fidelity, matmul correctness,
repeated-copy stress. Does NOT test P2P — use p2p_check.py for that.
Run: docker exec vllm python3 /path/to/gpu_integrity.py
"""
import torch
print("torch", torch.__version__, "cuda", torch.version.cuda, "devs", torch.cuda.device_count())
for dev in range(torch.cuda.device_count()):
try:
torch.cuda.set_device(dev)
x = torch.randint(0, 2**31 - 1, (128 * 1024 * 1024,), dtype=torch.int32)
y = x.to(f"cuda:{dev}")
z = y.cpu()
copy_ok = torch.equal(x, z)
a = torch.randn(2048, 2048, dtype=torch.float32)
b = torch.randn(2048, 2048, dtype=torch.float32)
ref = (a.double() @ b.double()).float()
c = (a.to(f"cuda:{dev}") @ b.to(f"cuda:{dev}")).cpu()
err = (c - ref).abs().max().item()
stress_ok = True
for _ in range(20):
s = torch.randint(0, 2**31 - 1, (16 * 1024 * 1024,), dtype=torch.int32)
if not torch.equal(s, s.to(f"cuda:{dev}").cpu()):
stress_ok = False
break
print(f"GPU{dev}: copy_ok={copy_ok} matmul_max_err={err:.3e} "
f"stress_ok={stress_ok} name={torch.cuda.get_device_name(dev)}")
del x, y, z, a, b, c
torch.cuda.empty_cache()
except Exception as e:
print(f"GPU{dev}: ERROR {type(e).__name__}: {e}")