# vllm-text wake failure + 503 Retry-After verification — 2026-08-20

First level-2 wake failure since the stack went live (2026-08-17), found
during remote-access testing by a calling service. No router code was
changed; this is the incident record + two client-facing findings.

## Timeline (all times local)

| Time | Event |
|---|---|
| 03:17 / 03:28 | embed / text wake from offloaded normally (2.9 s / 22.7 s) — morning remote-access tests |
| 09:36 | 4 requests to text → every `POST /wake_up` returns **HTTP 500**: `CUDA Error: invalid argument at /workspace/csrc/csrc/cumem_allocator.cpp:169` (`python_create_and_map`) |
| 09:36–09:51 | text hard-down: router 503 `model_waking` (`reason=wake_failed`) for all requests; GPU memory drifts asymmetric (GPU0 77 GB / GPU1 44 GB) |
| 09:52 | `./vllmctl restart text` (logs survive: same container) |
| 09:54 | Cold boot done in ~2 min (NFS warm); E2E chat verified, GPU0/1 symmetric ~70 GB |

## Diagnosis

- Same morning an offloaded wake succeeded (03:28) and all wakes
  08-17 → 08-20 were 200s, so the failure is **intermittent**, not a
  config regression. cuMemCreate/cuMemMap path failing with
  `invalid argument` smells like the same platform/IOMMU flakiness as
  the P2P corruption trail (`NOTES-2026-08-13.md`,
  `NOTES-2026-08-17-gpu2-recheck.md`) — unproven; vLLM sleep-mode bug
  not ruled out either.
- Host RAM was fine (894 GB available), so not memory exhaustion.

## 503 `Retry-After` header: verified present server-side

A caller reported the 503 body carried retry guidance but the
`Retry-After` header was missing. Verified on the exact path they hit
(`reason=wake_failed`, offloaded depth):

| Path | Response |
|---|---|
| `127.0.0.1:8000` direct (`--noproxy`) | `retry-after: 60` present |
| `10.11.19.24:8000` via local 7890 proxy | `retry-after: 60` present (proxy adds `Proxy-Connection`, strips nothing) |
| All 5 503 sites in `router/` (waking, shutdown ×2, admin ×2) | header set in every one |

Conclusion: the drop is on the caller's side (their self-built service's
HTTP client or an egress proxy). Body also carries
`estimated_wake_seconds` as a fallback.

## Client-facing quirks

- **Use full model names, not service keys/aliases.** The router
  resolves `model: "text"` but does not rewrite the body; an awake vLLM
  answers its own 404 `The model 'text' does not exist`. Valid names:
  `Qwen3.6-35B-A3B-FP8`, `OvisOCR2`, `Qwen3-Embedding-8B`.
- First remote request after a long idle should allow ~60 s (offload
  wake); today showed wake can also *fail* — retry per `Retry-After`
  but give up + alert after a few rounds (router retries internally
  2× per request already).

## Follow-ups

- [ ] Watch for recurrence; if it reappears, capture `docker logs
      vllm-text` around the failure + `dmesg` DMAR faults before
      restarting (this time the pre-restart state was not imaged).
- [ ] If it recurs >once/week: raise `IDLE_OFFLOAD_MIN` for text so it
      parks at level 1 (L1 wake never showed the failure and is ~2.5–4 s).
- [ ] Router could rewrite `model` to the canonical backend name when
      proxying — low priority, documented quirk instead.