bing
80eef4ce6a
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
58 lines
3.2 KiB
Markdown
58 lines
3.2 KiB
Markdown
# vLLM Project Memory
|
|
|
|
Index of memory files for this project.
|
|
|
|
## Architecture & Design
|
|
|
|
- [Router Front Door Plan v3.3](router-front-door-plan.md) — **IMPLEMENTED & VERIFIED 2026-08-17**: per-model vLLM services (all sleeping) + FastAPI router that auto-wakes on request. Pure-HTTP load/unload, no docker on the request path, replaced nginx. Two review rounds + full E2E.
|
|
|
|
- [Front Door Proxy Design](front-door-proxy-design.md) — original research (vLLM Sleep Mode, Triton, vllm-proxy) that led here.
|
|
|
|
- [Sleep Mode Implementation Plan v2](sleep-mode-implementation-plan.md) — the earlier Sleep Mode + nginx stack (implemented 2026-08-14, then superseded by the router plan).
|
|
|
|
## Project Documentation
|
|
|
|
- [`../README.md`](../README.md) — architecture, request behavior (503/Retry-After semantics), vllmctl, router tunables
|
|
- [`../TODO.md`](../TODO.md) — implementation record, verification trail, known limitations, housekeeping
|
|
- [`../CALIBRATION.md`](../CALIBRATION.md) — measured GPU footprints, wake latencies, backend quirks
|
|
- [`../NOTES-2026-08-13.md`](../NOTES-2026-08-13.md) — 2026-08-13 rework + GPU P2P diagnosis
|
|
- [`../NOTES-2026-08-17-gpu2-recheck.md`](../NOTES-2026-08-17-gpu2-recheck.md) — P2P fault moved (not fixed); root-cause analysis (VT-d peer-DMA translation); fix ladder for admin
|
|
|
|
## Model Configuration (live)
|
|
|
|
| Key | Served model | Service / GPUs | util |
|
|
|-----|--------------|----------------|------|
|
|
| text | `Qwen3.6-35B-A3B-FP8` | vllm-text, TP=2, GPU0+1 | 0.85, 262K ctx |
|
|
| ocr | `OvisOCR2` | vllm-ocr, GPU2 | 0.10 |
|
|
| embed | `Qwen3-Embedding-8B` | vllm-embed, GPU2 | 0.25 |
|
|
|
|
Public API `:8000/v1` (all three always available, auto-wake); admin API
|
|
`127.0.0.1:8010`; debug ports `127.0.0.1:8001-8003`.
|
|
|
|
## Hard-won facts (do not regress)
|
|
|
|
- **P2P corrupts data host-wide** (moves between GPUs across reboots; VT-d
|
|
peer-DMA platform fault). `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce`
|
|
mandatory everywhere. GPU2 is fine for single-GPU services.
|
|
- **vLLM backend quirks**: `/health` returns 200 while asleep (gate on
|
|
`/is_sleeping`); requests to a sleeping backend hang; requests between
|
|
`wake_up` and `reload_weights` return garbage; L1 wake needs `wake_up`
|
|
only; L1→L2 re-sleep is a no-op (wake-then-sleep to actually offload).
|
|
- **`gpu_memory_utilization` counts TOTAL GPU memory per process**; keep
|
|
per-GPU sums ≤ 0.88 including CUDA contexts.
|
|
- **Router state volume must be a named volume** (`router-state`), not a
|
|
bind mount — the non-root container user (10001) can't write a host-uid
|
|
dir, and the state store fails OPEN (fix silently inert).
|
|
- Docker: `renbaibing` in docker group since 2026-08-14; shells older than
|
|
that need `sg docker -c "…"`.
|
|
- OCR needs `--max-num-seqs 256` (GDN block limit at util 0.10).
|
|
- FP8 model dir was corrupted by racing `hf download`s on 2026-08-14;
|
|
repaired + sha256-verified 2026-08-17 (corrupt copy kept, see TODO).
|
|
|
|
## Housekeeping reminders
|
|
|
|
- Delete `Qwen3.6-35B-A3B-FP8.corrupt-20260817` (43 GB, in MODEL_ROOT) once trusted.
|
|
- Repo under git since 2026-08-17 (initial commit = router stack). The v2
|
|
nginx stack's runnable files were removed before git init; its design
|
|
survives in `sleep-mode-implementation-plan.md`.
|