Initial commit: router front-door vLLM stack
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
33
diag/gpu_integrity.py
Normal file
33
diag/gpu_integrity.py
Normal file
@@ -0,0 +1,33 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Per-GPU basic integrity: H2D/D2H copy fidelity, matmul correctness,
|
||||
repeated-copy stress. Does NOT test P2P — use p2p_check.py for that.
|
||||
|
||||
Run: docker exec vllm python3 /path/to/gpu_integrity.py
|
||||
"""
|
||||
import torch
|
||||
|
||||
print("torch", torch.__version__, "cuda", torch.version.cuda, "devs", torch.cuda.device_count())
|
||||
for dev in range(torch.cuda.device_count()):
|
||||
try:
|
||||
torch.cuda.set_device(dev)
|
||||
x = torch.randint(0, 2**31 - 1, (128 * 1024 * 1024,), dtype=torch.int32)
|
||||
y = x.to(f"cuda:{dev}")
|
||||
z = y.cpu()
|
||||
copy_ok = torch.equal(x, z)
|
||||
a = torch.randn(2048, 2048, dtype=torch.float32)
|
||||
b = torch.randn(2048, 2048, dtype=torch.float32)
|
||||
ref = (a.double() @ b.double()).float()
|
||||
c = (a.to(f"cuda:{dev}") @ b.to(f"cuda:{dev}")).cpu()
|
||||
err = (c - ref).abs().max().item()
|
||||
stress_ok = True
|
||||
for _ in range(20):
|
||||
s = torch.randint(0, 2**31 - 1, (16 * 1024 * 1024,), dtype=torch.int32)
|
||||
if not torch.equal(s, s.to(f"cuda:{dev}").cpu()):
|
||||
stress_ok = False
|
||||
break
|
||||
print(f"GPU{dev}: copy_ok={copy_ok} matmul_max_err={err:.3e} "
|
||||
f"stress_ok={stress_ok} name={torch.cuda.get_device_name(dev)}")
|
||||
del x, y, z, a, b, c
|
||||
torch.cuda.empty_cache()
|
||||
except Exception as e:
|
||||
print(f"GPU{dev}: ERROR {type(e).__name__}: {e}")
|
||||
Reference in New Issue
Block a user