# vLLM Project Memory

Index of memory files for this project.

## Architecture & Design

- [Router Front Door Plan v3.3](router-front-door-plan.md) — **IMPLEMENTED & VERIFIED 2026-08-17**: per-model vLLM services (all sleeping) + FastAPI router that auto-wakes on request. Pure-HTTP load/unload, no docker on the request path, replaced nginx. Two review rounds + full E2E.

- [Front Door Proxy Design](front-door-proxy-design.md) — original research (vLLM Sleep Mode, Triton, vllm-proxy) that led here.

- [Sleep Mode Implementation Plan v2](sleep-mode-implementation-plan.md) — the earlier Sleep Mode + nginx stack (implemented 2026-08-14, then superseded by the router plan).

## Project Documentation

- [`../README.md`](../README.md) — architecture, request behavior (503/Retry-After semantics), vllmctl, router tunables
- [`../TODO.md`](../TODO.md) — implementation record, verification trail, known limitations, housekeeping
- [`../CALIBRATION.md`](../CALIBRATION.md) — measured GPU footprints, wake latencies, backend quirks
- [`../NOTES-2026-08-13.md`](../NOTES-2026-08-13.md) — 2026-08-13 rework + GPU P2P diagnosis
- [`../NOTES-2026-08-17-gpu2-recheck.md`](../NOTES-2026-08-17-gpu2-recheck.md) — P2P fault moved (not fixed); root-cause analysis (VT-d peer-DMA translation); fix ladder for admin

## Model Configuration (live)

| Key | Served model | Service / GPUs | util |
|-----|--------------|----------------|------|
| text | `Qwen3.6-35B-A3B-FP8` | vllm-text, TP=2, GPU0+1 | 0.85, 262K ctx |
| ocr | `OvisOCR2` | vllm-ocr, GPU2 | 0.10 |
| embed | `Qwen3-Embedding-8B` | vllm-embed, GPU2 | 0.25 |

Public API `:8000/v1` (all three always available, auto-wake); admin API
`127.0.0.1:8010`; debug ports `127.0.0.1:8001-8003`.

## Hard-won facts (do not regress)

- **P2P corrupts data host-wide** (moves between GPUs across reboots; VT-d
  peer-DMA platform fault). `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce`
  mandatory everywhere. GPU2 is fine for single-GPU services.
- **vLLM backend quirks**: `/health` returns 200 while asleep (gate on
  `/is_sleeping`); requests to a sleeping backend hang; requests between
  `wake_up` and `reload_weights` return garbage; L1 wake needs `wake_up`
  only; L1→L2 re-sleep is a no-op (wake-then-sleep to actually offload).
- **`gpu_memory_utilization` counts TOTAL GPU memory per process**; keep
  per-GPU sums ≤ 0.88 including CUDA contexts.
- **Router state volume must be a named volume** (`router-state`), not a
  bind mount — the non-root container user (10001) can't write a host-uid
  dir, and the state store fails OPEN (fix silently inert).
- Docker: `renbaibing` in docker group since 2026-08-14; shells older than
  that need `sg docker -c "…"`.
- OCR needs `--max-num-seqs 256` (GDN block limit at util 0.10).
- FP8 model dir was corrupted by racing `hf download`s on 2026-08-14;
  repaired + sha256-verified 2026-08-17 (corrupt copy kept, see TODO).

## Housekeeping reminders

- Delete `Qwen3.6-35B-A3B-FP8.corrupt-20260817` (43 GB, in MODEL_ROOT) once trusted.
- Repo under git since 2026-08-17 (initial commit = router stack). The v2
  nginx stack's runnable files were removed before git init; its design
  survives in `sleep-mode-implementation-plan.md`.