From 2526d7b20316cb5eb70247e243f8b4863ccc26b8 Mon Sep 17 00:00:00 2001 From: bing Date: Thu, 20 Aug 2026 10:16:07 +0000 Subject: [PATCH] Notes: 2026-08-20 text wake_up CUDA failure + Retry-After verification Incident record for the first level-2 wake failure (cumem allocator, text hard-down ~16 min, container restart restored it) plus the caller-side findings from remote-access testing: 503 Retry-After verified present on every server path, and the model-alias 404 quirk. Co-Authored-By: Claude --- NOTES-2026-08-20-wake-cuda-failure.md | 63 +++++++++++++++++++++++++++ TODO.md | 3 ++ 2 files changed, 66 insertions(+) create mode 100644 NOTES-2026-08-20-wake-cuda-failure.md diff --git a/NOTES-2026-08-20-wake-cuda-failure.md b/NOTES-2026-08-20-wake-cuda-failure.md new file mode 100644 index 0000000..33fa9dc --- /dev/null +++ b/NOTES-2026-08-20-wake-cuda-failure.md @@ -0,0 +1,63 @@ +# vllm-text wake failure + 503 Retry-After verification — 2026-08-20 + +First level-2 wake failure since the stack went live (2026-08-17), found +during remote-access testing by a calling service. No router code was +changed; this is the incident record + two client-facing findings. + +## Timeline (all times local) + +| Time | Event | +|---|---| +| 03:17 / 03:28 | embed / text wake from offloaded normally (2.9 s / 22.7 s) — morning remote-access tests | +| 09:36 | 4 requests to text → every `POST /wake_up` returns **HTTP 500**: `CUDA Error: invalid argument at /workspace/csrc/csrc/cumem_allocator.cpp:169` (`python_create_and_map`) | +| 09:36–09:51 | text hard-down: router 503 `model_waking` (`reason=wake_failed`) for all requests; GPU memory drifts asymmetric (GPU0 77 GB / GPU1 44 GB) | +| 09:52 | `./vllmctl restart text` (logs survive: same container) | +| 09:54 | Cold boot done in ~2 min (NFS warm); E2E chat verified, GPU0/1 symmetric ~70 GB | + +## Diagnosis + +- Same morning an offloaded wake succeeded (03:28) and all wakes + 08-17 → 08-20 were 200s, so the failure is **intermittent**, not a + config regression. cuMemCreate/cuMemMap path failing with + `invalid argument` smells like the same platform/IOMMU flakiness as + the P2P corruption trail (`NOTES-2026-08-13.md`, + `NOTES-2026-08-17-gpu2-recheck.md`) — unproven; vLLM sleep-mode bug + not ruled out either. +- Host RAM was fine (894 GB available), so not memory exhaustion. + +## 503 `Retry-After` header: verified present server-side + +A caller reported the 503 body carried retry guidance but the +`Retry-After` header was missing. Verified on the exact path they hit +(`reason=wake_failed`, offloaded depth): + +| Path | Response | +|---|---| +| `127.0.0.1:8000` direct (`--noproxy`) | `retry-after: 60` present | +| `10.11.19.24:8000` via local 7890 proxy | `retry-after: 60` present (proxy adds `Proxy-Connection`, strips nothing) | +| All 5 503 sites in `router/` (waking, shutdown ×2, admin ×2) | header set in every one | + +Conclusion: the drop is on the caller's side (their self-built service's +HTTP client or an egress proxy). Body also carries +`estimated_wake_seconds` as a fallback. + +## Client-facing quirks + +- **Use full model names, not service keys/aliases.** The router + resolves `model: "text"` but does not rewrite the body; an awake vLLM + answers its own 404 `The model 'text' does not exist`. Valid names: + `Qwen3.6-35B-A3B-FP8`, `OvisOCR2`, `Qwen3-Embedding-8B`. +- First remote request after a long idle should allow ~60 s (offload + wake); today showed wake can also *fail* — retry per `Retry-After` + but give up + alert after a few rounds (router retries internally + 2× per request already). + +## Follow-ups + +- [ ] Watch for recurrence; if it reappears, capture `docker logs + vllm-text` around the failure + `dmesg` DMAR faults before + restarting (this time the pre-restart state was not imaged). +- [ ] If it recurs >once/week: raise `IDLE_OFFLOAD_MIN` for text so it + parks at level 1 (L1 wake never showed the failure and is ~2.5–4 s). +- [ ] Router could rewrite `model` to the canonical backend name when + proxying — low priority, documented quirk instead. diff --git a/TODO.md b/TODO.md index 0aedd23..fb3669c 100644 --- a/TODO.md +++ b/TODO.md @@ -42,6 +42,9 @@ but did not wake models on request; the front door's real job. Design record: ## Housekeeping +- [ ] 2026-08-20: first L2 wake failure (CUDA cumem error, text hard-down + ~16 min, restart fixed) — incident + follow-ups in + `NOTES-2026-08-20-wake-cuda-failure.md`. - [ ] Delete `Qwen3.6-35B-A3B-FP8.corrupt-20260817` (43 GB, in MODEL_ROOT) once the repaired FP8 copy is trusted. - [x] Old-stack leftovers removed 2026-08-17 before `git init`: nginx.conf,