vllm-frontdoor/NOTES-2026-08-20-wake-cuda-failure.md

vllm-text wake failure + 503 Retry-After verification — 2026-08-20

First level-2 wake failure since the stack went live (2026-08-17), found during remote-access testing by a calling service. No router code was changed; this is the incident record + two client-facing findings.

Timeline (all times local)

Time	Event
03:17 / 03:28	embed / text wake from offloaded normally (2.9 s / 22.7 s) — morning remote-access tests
09:36	4 requests to text → every POST /wake_up returns HTTP 500: CUDA Error: invalid argument at /workspace/csrc/csrc/cumem_allocator.cpp:169 (python_create_and_map)
09:36–09:51	text hard-down: router 503 model_waking (reason=wake_failed) for all requests; GPU memory drifts asymmetric (GPU0 77 GB / GPU1 44 GB)
09:52	./vllmctl restart text (logs survive: same container)
09:54	Cold boot done in ~2 min (NFS warm); E2E chat verified, GPU0/1 symmetric ~70 GB

Diagnosis

• Same morning an offloaded wake succeeded (03:28) and all wakes 08-17 → 08-20 were 200s, so the failure is intermittent, not a config regression. cuMemCreate/cuMemMap path failing with invalid argument smells like the same platform/IOMMU flakiness as the P2P corruption trail (NOTES-2026-08-13.md, NOTES-2026-08-17-gpu2-recheck.md) — unproven; vLLM sleep-mode bug not ruled out either.
• Host RAM was fine (894 GB available), so not memory exhaustion.

503 Retry-After header: verified present server-side

A caller reported the 503 body carried retry guidance but the Retry-After header was missing. Verified on the exact path they hit (reason=wake_failed, offloaded depth):

Path	Response
127.0.0.1:8000 direct (--noproxy)	retry-after: 60 present
10.11.19.24:8000 via local 7890 proxy	retry-after: 60 present (proxy adds Proxy-Connection, strips nothing)
All 5 503 sites in router/ (waking, shutdown ×2, admin ×2)	header set in every one

Conclusion: the drop is on the caller's side (their self-built service's HTTP client or an egress proxy). Body also carries estimated_wake_seconds as a fallback.

Client-facing quirks

• Use full model names, not service keys/aliases. The router resolves model: "text" but does not rewrite the body; an awake vLLM answers its own 404 The model 'text' does not exist. Valid names: Qwen3.6-35B-A3B-FP8, OvisOCR2, Qwen3-Embedding-8B.
• First remote request after a long idle should allow ~60 s (offload wake); today showed wake can also fail — retry per Retry-After but give up + alert after a few rounds (router retries internally 2× per request already).

Follow-ups

• Watch for recurrence; if it reappears, capture docker logs vllm-text around the failure + dmesg DMAR faults before restarting (this time the pre-restart state was not imaged).
• If it recurs >once/week: raise IDLE_OFFLOAD_MIN for text so it parks at level 1 (L1 wake never showed the failure and is ~2.5–4 s).
• Router could rewrite model to the canonical backend name when proxying — low priority, documented quirk instead.