Notes: 2026-08-20 text wake_up CUDA failure + Retry-After verification

Incident record for the first level-2 wake failure (cumem allocator, text hard-down ~16 min, container restart restored it) plus the caller-side findings from remote-access testing: 503 Retry-After verified present on every server path, and the model-alias 404 quirk. Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-20 10:16:07 +00:00
parent 80eef4ce6a
commit 2526d7b203
2 changed files with 66 additions and 0 deletions
@@ -0,0 +1,63 @@
+# vllm-text wake failure + 503 Retry-After verification — 2026-08-20
+
+First level-2 wake failure since the stack went live (2026-08-17), found
+during remote-access testing by a calling service. No router code was
+changed; this is the incident record + two client-facing findings.
+
+## Timeline (all times local)
+
+| Time | Event |
+|---|---|
+| 03:17 / 03:28 | embed / text wake from offloaded normally (2.9 s / 22.7 s) — morning remote-access tests |
+| 09:36 | 4 requests to text → every `POST /wake_up` returns **HTTP 500**: `CUDA Error: invalid argument at /workspace/csrc/csrc/cumem_allocator.cpp:169` (`python_create_and_map`) |
+| 09:36–09:51 | text hard-down: router 503 `model_waking` (`reason=wake_failed`) for all requests; GPU memory drifts asymmetric (GPU0 77 GB / GPU1 44 GB) |
+| 09:52 | `./vllmctl restart text` (logs survive: same container) |
+| 09:54 | Cold boot done in ~2 min (NFS warm); E2E chat verified, GPU0/1 symmetric ~70 GB |
+
+## Diagnosis
+
+- Same morning an offloaded wake succeeded (03:28) and all wakes
+  08-17 → 08-20 were 200s, so the failure is **intermittent**, not a
+  config regression. cuMemCreate/cuMemMap path failing with
+  `invalid argument` smells like the same platform/IOMMU flakiness as
+  the P2P corruption trail (`NOTES-2026-08-13.md`,
+  `NOTES-2026-08-17-gpu2-recheck.md`) — unproven; vLLM sleep-mode bug
+  not ruled out either.
+- Host RAM was fine (894 GB available), so not memory exhaustion.
+
+## 503 `Retry-After` header: verified present server-side
+
+A caller reported the 503 body carried retry guidance but the
+`Retry-After` header was missing. Verified on the exact path they hit
+(`reason=wake_failed`, offloaded depth):
+
+| Path | Response |
+|---|---|
+| `127.0.0.1:8000` direct (`--noproxy`) | `retry-after: 60` present |
+| `10.11.19.24:8000` via local 7890 proxy | `retry-after: 60` present (proxy adds `Proxy-Connection`, strips nothing) |
+| All 5 503 sites in `router/` (waking, shutdown ×2, admin ×2) | header set in every one |
+
+Conclusion: the drop is on the caller's side (their self-built service's
+HTTP client or an egress proxy). Body also carries
+`estimated_wake_seconds` as a fallback.
+
+## Client-facing quirks
+
+- **Use full model names, not service keys/aliases.** The router
+  resolves `model: "text"` but does not rewrite the body; an awake vLLM
+  answers its own 404 `The model 'text' does not exist`. Valid names:
+  `Qwen3.6-35B-A3B-FP8`, `OvisOCR2`, `Qwen3-Embedding-8B`.
+- First remote request after a long idle should allow ~60 s (offload
+  wake); today showed wake can also *fail* — retry per `Retry-After`
+  but give up + alert after a few rounds (router retries internally
+  2× per request already).
+
+## Follow-ups
+
+- [ ] Watch for recurrence; if it reappears, capture `docker logs
+      vllm-text` around the failure + `dmesg` DMAR faults before
+      restarting (this time the pre-restart state was not imaged).
+- [ ] If it recurs >once/week: raise `IDLE_OFFLOAD_MIN` for text so it
+      parks at level 1 (L1 wake never showed the failure and is ~2.5–4 s).
+- [ ] Router could rewrite `model` to the canonical backend name when
+      proxying — low priority, documented quirk instead.