bingandClaude
2526d7b203
Incident record for the first level-2 wake failure (cumem allocator, text hard-down ~16 min, container restart restored it) plus the caller-side findings from remote-access testing: 503 Retry-After verified present on every server path, and the model-alias 404 quirk. Co-Authored-By: Claude <noreply@anthropic.com>
64 lines
3.1 KiB
Markdown
64 lines
3.1 KiB
Markdown
# vllm-text wake failure + 503 Retry-After verification — 2026-08-20
|
||
|
||
First level-2 wake failure since the stack went live (2026-08-17), found
|
||
during remote-access testing by a calling service. No router code was
|
||
changed; this is the incident record + two client-facing findings.
|
||
|
||
## Timeline (all times local)
|
||
|
||
| Time | Event |
|
||
|---|---|
|
||
| 03:17 / 03:28 | embed / text wake from offloaded normally (2.9 s / 22.7 s) — morning remote-access tests |
|
||
| 09:36 | 4 requests to text → every `POST /wake_up` returns **HTTP 500**: `CUDA Error: invalid argument at /workspace/csrc/csrc/cumem_allocator.cpp:169` (`python_create_and_map`) |
|
||
| 09:36–09:51 | text hard-down: router 503 `model_waking` (`reason=wake_failed`) for all requests; GPU memory drifts asymmetric (GPU0 77 GB / GPU1 44 GB) |
|
||
| 09:52 | `./vllmctl restart text` (logs survive: same container) |
|
||
| 09:54 | Cold boot done in ~2 min (NFS warm); E2E chat verified, GPU0/1 symmetric ~70 GB |
|
||
|
||
## Diagnosis
|
||
|
||
- Same morning an offloaded wake succeeded (03:28) and all wakes
|
||
08-17 → 08-20 were 200s, so the failure is **intermittent**, not a
|
||
config regression. cuMemCreate/cuMemMap path failing with
|
||
`invalid argument` smells like the same platform/IOMMU flakiness as
|
||
the P2P corruption trail (`NOTES-2026-08-13.md`,
|
||
`NOTES-2026-08-17-gpu2-recheck.md`) — unproven; vLLM sleep-mode bug
|
||
not ruled out either.
|
||
- Host RAM was fine (894 GB available), so not memory exhaustion.
|
||
|
||
## 503 `Retry-After` header: verified present server-side
|
||
|
||
A caller reported the 503 body carried retry guidance but the
|
||
`Retry-After` header was missing. Verified on the exact path they hit
|
||
(`reason=wake_failed`, offloaded depth):
|
||
|
||
| Path | Response |
|
||
|---|---|
|
||
| `127.0.0.1:8000` direct (`--noproxy`) | `retry-after: 60` present |
|
||
| `10.11.19.24:8000` via local 7890 proxy | `retry-after: 60` present (proxy adds `Proxy-Connection`, strips nothing) |
|
||
| All 5 503 sites in `router/` (waking, shutdown ×2, admin ×2) | header set in every one |
|
||
|
||
Conclusion: the drop is on the caller's side (their self-built service's
|
||
HTTP client or an egress proxy). Body also carries
|
||
`estimated_wake_seconds` as a fallback.
|
||
|
||
## Client-facing quirks
|
||
|
||
- **Use full model names, not service keys/aliases.** The router
|
||
resolves `model: "text"` but does not rewrite the body; an awake vLLM
|
||
answers its own 404 `The model 'text' does not exist`. Valid names:
|
||
`Qwen3.6-35B-A3B-FP8`, `OvisOCR2`, `Qwen3-Embedding-8B`.
|
||
- First remote request after a long idle should allow ~60 s (offload
|
||
wake); today showed wake can also *fail* — retry per `Retry-After`
|
||
but give up + alert after a few rounds (router retries internally
|
||
2× per request already).
|
||
|
||
## Follow-ups
|
||
|
||
- [ ] Watch for recurrence; if it reappears, capture `docker logs
|
||
vllm-text` around the failure + `dmesg` DMAR faults before
|
||
restarting (this time the pre-restart state was not imaged).
|
||
- [ ] If it recurs >once/week: raise `IDLE_OFFLOAD_MIN` for text so it
|
||
parks at level 1 (L1 wake never showed the failure and is ~2.5–4 s).
|
||
- [ ] Router could rewrite `model` to the canonical backend name when
|
||
proxying — low priority, documented quirk instead.
|