bing/vllm-frontdoor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
vllm-frontdoor/.claude/memory/MEMORY.md
bing 80eef4ce6a Initial commit: router front-door vLLM stack 
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2,
sleep mode) behind a FastAPI router that auto-wakes models on request.
Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with
Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010.
Routine control via vllmctl is pure HTTP — no docker on the request path.

Verified: 91 router unit tests + 15-test E2E on real hardware
(measurements in CALIBRATION.md; design record in
.claude/memory/router-front-door-plan.md).

Old nginx stack files removed before git init; design survives in
.claude/memory/sleep-mode-implementation-plan.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:17:42 +00:00

3.2 KiB
Raw Permalink Blame History

vLLM Project Memory

Index of memory files for this project.

Architecture & Design

  • Router Front Door Plan v3.3 — IMPLEMENTED & VERIFIED 2026-08-17: per-model vLLM services (all sleeping) + FastAPI router that auto-wakes on request. Pure-HTTP load/unload, no docker on the request path, replaced nginx. Two review rounds + full E2E.

  • Front Door Proxy Design — original research (vLLM Sleep Mode, Triton, vllm-proxy) that led here.

  • Sleep Mode Implementation Plan v2 — the earlier Sleep Mode + nginx stack (implemented 2026-08-14, then superseded by the router plan).

Project Documentation

Model Configuration (live)

Key Served model Service / GPUs util
text Qwen3.6-35B-A3B-FP8 vllm-text, TP=2, GPU0+1 0.85, 262K ctx
ocr OvisOCR2 vllm-ocr, GPU2 0.10
embed Qwen3-Embedding-8B vllm-embed, GPU2 0.25

Public API :8000/v1 (all three always available, auto-wake); admin API 127.0.0.1:8010; debug ports 127.0.0.1:8001-8003.

Hard-won facts (do not regress)

  • P2P corrupts data host-wide (moves between GPUs across reboots; VT-d peer-DMA platform fault). NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce mandatory everywhere. GPU2 is fine for single-GPU services.
  • vLLM backend quirks: /health returns 200 while asleep (gate on /is_sleeping); requests to a sleeping backend hang; requests between wake_up and reload_weights return garbage; L1 wake needs wake_up only; L1→L2 re-sleep is a no-op (wake-then-sleep to actually offload).
  • gpu_memory_utilization counts TOTAL GPU memory per process; keep per-GPU sums ≤ 0.88 including CUDA contexts.
  • Router state volume must be a named volume (router-state), not a bind mount — the non-root container user (10001) can't write a host-uid dir, and the state store fails OPEN (fix silently inert).
  • Docker: renbaibing in docker group since 2026-08-14; shells older than that need sg docker -c "…".
  • OCR needs --max-num-seqs 256 (GDN block limit at util 0.10).
  • FP8 model dir was corrupted by racing hf downloads on 2026-08-14; repaired + sha256-verified 2026-08-17 (corrupt copy kept, see TODO).

Housekeeping reminders

  • Delete Qwen3.6-35B-A3B-FP8.corrupt-20260817 (43 GB, in MODEL_ROOT) once trusted.
  • Repo under git since 2026-08-17 (initial commit = router stack). The v2 nginx stack's runnable files were removed before git init; its design survives in sleep-mode-implementation-plan.md.