Initial commit: router front-door vLLM stack
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
57
.claude/memory/MEMORY.md
Normal file
57
.claude/memory/MEMORY.md
Normal file
@@ -0,0 +1,57 @@
|
||||
# vLLM Project Memory
|
||||
|
||||
Index of memory files for this project.
|
||||
|
||||
## Architecture & Design
|
||||
|
||||
- [Router Front Door Plan v3.3](router-front-door-plan.md) — **IMPLEMENTED & VERIFIED 2026-08-17**: per-model vLLM services (all sleeping) + FastAPI router that auto-wakes on request. Pure-HTTP load/unload, no docker on the request path, replaced nginx. Two review rounds + full E2E.
|
||||
|
||||
- [Front Door Proxy Design](front-door-proxy-design.md) — original research (vLLM Sleep Mode, Triton, vllm-proxy) that led here.
|
||||
|
||||
- [Sleep Mode Implementation Plan v2](sleep-mode-implementation-plan.md) — the earlier Sleep Mode + nginx stack (implemented 2026-08-14, then superseded by the router plan).
|
||||
|
||||
## Project Documentation
|
||||
|
||||
- [`../README.md`](../README.md) — architecture, request behavior (503/Retry-After semantics), vllmctl, router tunables
|
||||
- [`../TODO.md`](../TODO.md) — implementation record, verification trail, known limitations, housekeeping
|
||||
- [`../CALIBRATION.md`](../CALIBRATION.md) — measured GPU footprints, wake latencies, backend quirks
|
||||
- [`../NOTES-2026-08-13.md`](../NOTES-2026-08-13.md) — 2026-08-13 rework + GPU P2P diagnosis
|
||||
- [`../NOTES-2026-08-17-gpu2-recheck.md`](../NOTES-2026-08-17-gpu2-recheck.md) — P2P fault moved (not fixed); root-cause analysis (VT-d peer-DMA translation); fix ladder for admin
|
||||
|
||||
## Model Configuration (live)
|
||||
|
||||
| Key | Served model | Service / GPUs | util |
|
||||
|-----|--------------|----------------|------|
|
||||
| text | `Qwen3.6-35B-A3B-FP8` | vllm-text, TP=2, GPU0+1 | 0.85, 262K ctx |
|
||||
| ocr | `OvisOCR2` | vllm-ocr, GPU2 | 0.10 |
|
||||
| embed | `Qwen3-Embedding-8B` | vllm-embed, GPU2 | 0.25 |
|
||||
|
||||
Public API `:8000/v1` (all three always available, auto-wake); admin API
|
||||
`127.0.0.1:8010`; debug ports `127.0.0.1:8001-8003`.
|
||||
|
||||
## Hard-won facts (do not regress)
|
||||
|
||||
- **P2P corrupts data host-wide** (moves between GPUs across reboots; VT-d
|
||||
peer-DMA platform fault). `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce`
|
||||
mandatory everywhere. GPU2 is fine for single-GPU services.
|
||||
- **vLLM backend quirks**: `/health` returns 200 while asleep (gate on
|
||||
`/is_sleeping`); requests to a sleeping backend hang; requests between
|
||||
`wake_up` and `reload_weights` return garbage; L1 wake needs `wake_up`
|
||||
only; L1→L2 re-sleep is a no-op (wake-then-sleep to actually offload).
|
||||
- **`gpu_memory_utilization` counts TOTAL GPU memory per process**; keep
|
||||
per-GPU sums ≤ 0.88 including CUDA contexts.
|
||||
- **Router state volume must be a named volume** (`router-state`), not a
|
||||
bind mount — the non-root container user (10001) can't write a host-uid
|
||||
dir, and the state store fails OPEN (fix silently inert).
|
||||
- Docker: `renbaibing` in docker group since 2026-08-14; shells older than
|
||||
that need `sg docker -c "…"`.
|
||||
- OCR needs `--max-num-seqs 256` (GDN block limit at util 0.10).
|
||||
- FP8 model dir was corrupted by racing `hf download`s on 2026-08-14;
|
||||
repaired + sha256-verified 2026-08-17 (corrupt copy kept, see TODO).
|
||||
|
||||
## Housekeeping reminders
|
||||
|
||||
- Delete `Qwen3.6-35B-A3B-FP8.corrupt-20260817` (43 GB, in MODEL_ROOT) once trusted.
|
||||
- Repo under git since 2026-08-17 (initial commit = router stack). The v2
|
||||
nginx stack's runnable files were removed before git init; its design
|
||||
survives in `sleep-mode-implementation-plan.md`.
|
||||
164
.claude/memory/front-door-proxy-design.md
Normal file
164
.claude/memory/front-door-proxy-design.md
Normal file
@@ -0,0 +1,164 @@
|
||||
# Front Door Proxy Design — Handling Requests to Unloaded Models
|
||||
|
||||
**Date:** 2026-08-13
|
||||
**Status:** Design exploration — vLLM Sleep Mode available as better alternative
|
||||
|
||||
> **TL;DR:** vLLM 0.11.0+ includes **Sleep Mode** — a built-in feature that enables fast model switching (18-200x faster than full reload) by hibernating weights while preserving process state. **This is likely the solution we need instead of building a custom front door.**
|
||||
|
||||
## Current Behavior: Request to Unloaded Model
|
||||
|
||||
When a client makes a request to `http://<host>:8000/v1/...` and no model is loaded:
|
||||
|
||||
1. **Container is down** (auto-unload via `idle-watch` or manual `down`)
|
||||
2. **Immediate connection failure** — client gets `Connection refused` or similar
|
||||
3. **No retry guidance** — client doesn't know if the service is down, moving, or loading
|
||||
4. **Manual recovery required** — someone must run `./vllmctl up <model>`
|
||||
|
||||
### What the client sees
|
||||
|
||||
```bash
|
||||
$ curl http://host:8000/v1/models
|
||||
curl: (7) Failed to connect to host port 8000: Connection refused
|
||||
```
|
||||
|
||||
No indication of:
|
||||
- Whether the service is permanently down
|
||||
- Which model should be loaded
|
||||
- How long loading would take (~2-10 min for large models from NFS)
|
||||
|
||||
## The Shared Front Door Concept
|
||||
|
||||
A reverse proxy that sits in front of vLLM and handles the "unloaded model" case gracefully.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
Client → Front Door (port 8000) → vLLM Container (port 8001 or similar)
|
||||
↓
|
||||
[Model unloaded?]
|
||||
↓
|
||||
Trigger load / Queue / Inform
|
||||
```
|
||||
|
||||
### Design Options
|
||||
|
||||
#### Option 1: Wake-on-Request (Auto-Load)
|
||||
|
||||
On receiving a request when no model is loaded:
|
||||
|
||||
1. **Spawn the container** with the requested model (from URL path or header)
|
||||
2. **Queue the request** (or return 503 with `Retry-After`)
|
||||
3. **Client retries** after model is ready (or proxy forwards queued request)
|
||||
|
||||
**Pros:** Transparent to clients, no manual intervention
|
||||
**Cons:** Long wait times (2-10 min), cold start, need to decide which model to load
|
||||
**Complexity:** Need request queue, loading state tracking, model selection logic
|
||||
|
||||
#### Option 2: Friendly Error + Status Endpoint
|
||||
|
||||
On receiving a request when no model is loaded:
|
||||
|
||||
1. **Return 503 Service Unavailable** with clear JSON body
|
||||
2. **Include status info:** "No model loaded. Load with `vllmctl up <model>`"
|
||||
3. **Provide a status endpoint** clients can poll: `GET /front-door/status`
|
||||
- Shows: models available, currently loaded, loading state (if any)
|
||||
|
||||
**Pros:** Simple to implement, clear client guidance
|
||||
**Cons:** Still requires manual load or client-side logic
|
||||
**Complexity:** Low — just a small nginx/envoy/python proxy
|
||||
|
||||
#### Option 3: Request Queue + Load Notification
|
||||
|
||||
On receiving a request when no model is loaded:
|
||||
|
||||
1. **Return 202 Accepted** with a `Location` header pointing to a job status URL
|
||||
2. **Trigger model load** in background
|
||||
3. **Client polls status** or gets notified (webhook) when ready
|
||||
4. **When ready, client retries** original request
|
||||
|
||||
**Pros:** Clear async flow, client knows what's happening
|
||||
**Cons:** More complex, needs job tracking, notification or polling
|
||||
**Complexity:** Medium — need job store, status API
|
||||
|
||||
## Implementation Sketch (Option 2: Friendly Error)
|
||||
|
||||
### Simple Python Front Door
|
||||
|
||||
```python
|
||||
# front-door.py
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
import subprocess
|
||||
import requests
|
||||
import json
|
||||
|
||||
VLLM_PORT = 8001
|
||||
FRONT_PORT = 8000
|
||||
MODELS_ROOT = "/data/home/renbaibing/huggingface"
|
||||
|
||||
class FrontDoorHandler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
# Check if vLLM is healthy
|
||||
try:
|
||||
resp = requests.get(f"http://127.0.0.1:{VLLM_PORT}/health", timeout=1)
|
||||
# Proxy the request
|
||||
proxy_resp = requests.get(f"http://127.0.0.1:{VLLM_PORT}{self.path}")
|
||||
self.wfile.write(proxy_resp.content)
|
||||
except requests.exceptions.RequestException:
|
||||
# vLLM is not ready
|
||||
self.send_response(503)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.end_headers()
|
||||
response = {
|
||||
"error": "No model currently loaded",
|
||||
"message": "Load a model with: ./vllmctl up <model>",
|
||||
"available_models": self.list_models(),
|
||||
"status_url": f"http://{self.server.server_port}/status"
|
||||
}
|
||||
self.wfile.write(json.dumps(response).encode())
|
||||
|
||||
def list_models(self):
|
||||
# Scan MODEL_ROOT for model directories
|
||||
# Return list of available model names
|
||||
pass
|
||||
|
||||
HTTPServer(("", FRONT_PORT), FrontDoorHandler).serve_forever()
|
||||
```
|
||||
|
||||
### Integration with existing setup
|
||||
|
||||
1. Move vLLM to port 8001 (`VLLM_PORT=8001` in .env)
|
||||
2. Run front door on port 8000 (or via compose as a separate service)
|
||||
3. Front door checks vLLM health before proxying
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Model selection:** If auto-loading, how does the proxy know which model to load?
|
||||
- From URL path? (`/v1/models/NAME/completions`?)
|
||||
- From header? (`X-Model-Name: Qwen3.6-35B-A3B`)
|
||||
- Configured default?
|
||||
|
||||
2. **Cold start cost:** Large models take 2-10 minutes to load from NFS
|
||||
- Is this acceptable for clients?
|
||||
- Should we cache recently-used models?
|
||||
|
||||
3. **Multi-model concurrent serving:** Does vLLM support serving multiple models simultaneously?
|
||||
- If yes, the proxy could load models on-demand and keep them warm
|
||||
- If no, we need model switching logic (potentially disruptive to existing requests)
|
||||
|
||||
4. **Authentication:** Should the front door handle API keys?
|
||||
- Current setup has no auth
|
||||
- Adding auth at the front door would be natural
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Confirm requirements:** Decide between Option 1 (auto-load), 2 (friendly error), or 3 (async queue)
|
||||
2. **Prototype:** Build a minimal front door for the chosen option
|
||||
3. **Integration:** Wire into existing `compose.yml` and `vllmctl` workflow
|
||||
4. **Testing:** Simulate various failure modes and client behaviors
|
||||
|
||||
## References
|
||||
|
||||
- Current setup: `README.md`, `NOTES-2026-08-13.md`
|
||||
- vLLM health endpoint: `http://127.0.0.1:8000/health`
|
||||
- vLLM metrics: `http://127.0.0.1:8000/metrics` (used by idle-watch)
|
||||
- OpenAI API compatibility: `http://127.0.0.1:8000/v1/...`
|
||||
339
.claude/memory/router-front-door-plan.md
Normal file
339
.claude/memory/router-front-door-plan.md
Normal file
@@ -0,0 +1,339 @@
|
||||
# Plan v3.3 — Router Front Door (auto wake-on-request)
|
||||
|
||||
**Date:** 2026-08-14, revised 2026-08-17 (v3.3)
|
||||
**Status:** **IMPLEMENTED & VERIFIED 2026-08-17** — v3.2 passed two review rounds (5 critical + 6 major fixed and confirmed; round-2 minors applied). v3.3 = hardware-driven placement change (GPU2 cleared for single-GPU services → OCR+embed on GPU2, text exclusive on GPU0/1). Implementation record and verification trail: `../../TODO.md`; measured numbers: `../../CALIBRATION.md`. One post-plan correctness fix: persisted wake-intent + startup recovery (E2E case 12), see TODO "Verification trail".
|
||||
**Decisions made with user:** multiple models may be awake concurrently; router = Python FastAPI; tiered idle (level-1 sleep after 15 min, level-2 offload after 3 hr); depth-aware 503 errors.
|
||||
|
||||
## 1. Problem & constraints
|
||||
|
||||
Calling services must be able to hit `http://<host>:8000/v1/...` with any of our
|
||||
three models at any time, with no knowledge of load state. Hard constraints:
|
||||
|
||||
1. **Native to the Docker stack** — the entire request path runs inside
|
||||
containers; no host-side cron, watchers, or helpers.
|
||||
2. **No docker permissions on the request path** — model load/unload/switch is
|
||||
pure HTTP (vLLM Sleep Mode). Docker is only needed for rare manual ops
|
||||
(image upgrade, pulling new models).
|
||||
3. **Highly reliable** — failures degrade to depth-aware 503 + `Retry-After`,
|
||||
never a hang, never a proxy to a half-awake backend.
|
||||
|
||||
The Plan v2 nginx front door hid dev endpoints but did **not** wake models on
|
||||
request; that inverted priority is what this plan fixes.
|
||||
|
||||
## 2. Hardware reality (re-verified 2026-08-17 — see NOTES-2026-08-17-gpu2-recheck.md)
|
||||
|
||||
- **3× A800-SXM4-80GB**. P2P is **still broken host-wide, 4/6 paths corrupt**,
|
||||
but the fault MOVED: now every transfer **sourced from GPU0 or GPU1**
|
||||
corrupts; GPU2-sourced transfers are the only clean ones (was the inverse
|
||||
on 2026-08-13). Per-GPU integrity (H2D/D2H, matmul, stress) passes on all
|
||||
three, twice.
|
||||
- Consequences: `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce` stay
|
||||
mandatory for any TP>1 workload (host-staged copies verify clean — which is
|
||||
why TP=2 on GPU0+1 with P2P off works). **GPU2 is usable for single-GPU
|
||||
services** (no P2P involved; H2D/D2H verified clean).
|
||||
- On-disk models (verified): `Qwen3.6-35B-A3B-FP8` (43GB dir), `OvisOCR2`
|
||||
(1.8GB), `Qwen3-Embedding-8B` (12GB). **User confirmed 2026-08-14: the 8B
|
||||
embedding model is the intended one** (earlier docs mentioning a 0.6B were
|
||||
wrong; slices are sized for the 8B).
|
||||
|
||||
## 3. Architecture
|
||||
|
||||
```
|
||||
┌──────────────────── Docker network ────────────────────┐
|
||||
callers ─► :8000 ─► ┌─────────┐ ┌──────────────────────┐ ┌───────────────────┐
|
||||
│ router │ ─►│ vllm-text (TP=2) │ │ vllm-ocr GPU2 │
|
||||
│ FastAPI │ │ GPU0 + GPU1 only │ │ vllm-embed GPU2 │
|
||||
└─────────┘ └──────────────────────┘ └───────────────────┘
|
||||
127.0.0.1:8010 (admin)
|
||||
```
|
||||
|
||||
- **One always-running vLLM service per model**, each with
|
||||
`--enable-sleep-mode` + `VLLM_SERVER_DEV_MODE=1`, each pinned to a GPU
|
||||
memory slice so all three can be awake simultaneously.
|
||||
- **Router** is the only public ingress (port 8000). It owns: model-name →
|
||||
service mapping, wake-on-request, streaming proxy, tiered idle management,
|
||||
and dev-endpoint hiding (404 for anything not an allowed path, matched on
|
||||
the **normalized** path to defeat `/v1/../sleep` traversal).
|
||||
- **Admin API is a separate listener** published only on `127.0.0.1:8010` —
|
||||
NOT paths on the public port (review C5: a public `/admin/sleep` would be a
|
||||
trivial remote DoS).
|
||||
- **nginx is removed.** The router replaces it.
|
||||
- **Placement (v3.3, user-approved 2026-08-17):** text TP=2 gets GPU0+GPU1
|
||||
**exclusively** (0.85 slice, full 262K context); OCR and embed run TP=1
|
||||
**on GPU2** — single-GPU services never issue P2P transfers, and GPU2's
|
||||
H2D/D2H paths verify clean (§2). `CUDA_VISIBLE_DEVICES` is set explicitly
|
||||
per service. Rollback if GPU2 ever misbehaves: change one env var to move
|
||||
a service back to GPU0/1 (slices would need re-tightening).
|
||||
- All services keep `restart: unless-stopped` and `ipc: host` (NCCL shm).
|
||||
Each service's TP TCP store is isolated by its own network namespace — no
|
||||
distributed-init port collision (would only become a risk if someone later
|
||||
sets `network_mode: host`; do not).
|
||||
|
||||
## 4. Port layout
|
||||
|
||||
| Port | Binding | Purpose |
|
||||
|---|---|---|
|
||||
| 8000 | 0.0.0.0 (host) → router | Public API (only public ingress) |
|
||||
| 8010 | 127.0.0.1 (host) → router admin listener | `vllmctl` / debugging |
|
||||
| 8001 | 127.0.0.1 (host) → vllm-text :8000 | Direct debug access |
|
||||
| 8002 | 127.0.0.1 (host) → vllm-ocr :8000 | Direct debug access |
|
||||
| 8003 | 127.0.0.1 (host) → vllm-embed :8000 | Direct debug access |
|
||||
| — | Docker net only | router → vllm-* |
|
||||
|
||||
**Security boundary, stated explicitly:** the loopback debug ports 8001–8003
|
||||
expose each service's dev endpoints (`/sleep`, `/wake_up`, …) to **any local
|
||||
user** on this multi-user host, bypassing the router's 404s. Same exposure as
|
||||
the current stack's port 8001; accepted, but "dev endpoints unreachable"
|
||||
means "unreachable from the network," not "unreachable locally."
|
||||
|
||||
## 5. GPU memory budget (3× 80GB in use; calibration gates final numbers)
|
||||
|
||||
`gpu_memory_utilization` is a fraction of **total** GPU memory, and each
|
||||
process accounts only for itself. Budget per GPU:
|
||||
|
||||
Σ(slices on that GPU) + (CUDA ctx per process ~1–2GB) + boot/wake transients ≤ 0.88
|
||||
|
||||
Initial slices (recalibrated in Step 1):
|
||||
|
||||
| Service | Weights on disk | TP / GPU | util slice | Notes |
|
||||
|---|---|---|---|---|
|
||||
| vllm-text | 43GB (FP8) | 2 / GPU0+1 (exclusive) | 0.85 | full 262K context; KV ~25GB — no 131K fallback needed |
|
||||
| vllm-ocr | 1.8GB | 1 / GPU2 | 0.10 | small vision model |
|
||||
| vllm-embed | 12GB (8B) | 1 / GPU2 | 0.25 | `--max-model-len 8192`; raisable at calibration |
|
||||
|
||||
Per-GPU: GPU0 = GPU1 = 0.85 + 1 ctx ≈ 0.87; GPU2 = 0.10 + 0.25 = 0.35 + 2 ctx.
|
||||
Disjoint pools — text never competes with the small services, and the small
|
||||
services have an entire idle GPU of headroom.
|
||||
|
||||
Sleeping (level 2) residual is ~2.5GB per **process** (E2E measured ~5GB
|
||||
total for one TP=2 instance = 1 proc/GPU). Under v3.3 placement: GPU0/GPU1
|
||||
host 1 process each (text) → ~2.5GB residual; GPU2 hosts 2 (ocr + embed)
|
||||
→ ~5GB. Measured in Step 1.
|
||||
|
||||
## 6. Router design
|
||||
|
||||
Python 3.12-slim container, **pinned** fastapi/httpx/uvicorn versions,
|
||||
`restart: unless-stopped`. ~300 lines.
|
||||
|
||||
**Single process, single event loop, two listening sockets** (public :8000 +
|
||||
admin :8010): run two `uvicorn.Server` instances as coroutines on **one**
|
||||
asyncio loop (uvicorn can't bind two ports in one worker, and two separate
|
||||
processes would break the in-process wake locks — the admin listener MUST
|
||||
share the same locks and depth tracking as the request path).
|
||||
|
||||
### 6.1 Model registry (env or `models.json`)
|
||||
|
||||
```
|
||||
text : { service: "vllm-text", model: "Qwen3.6-35B-A3B-FP8", aliases: ["qwen3.6-35b-a3b-fp8", ...] }
|
||||
ocr : { service: "vllm-ocr", model: "OvisOCR2" }
|
||||
embed: { service: "vllm-embed", model: "Qwen3-Embedding-8B" }
|
||||
```
|
||||
|
||||
(No 0.6B alias — the 8B is the confirmed model (2026-08-14); an alias would
|
||||
mislead clients into thinking they got a different model than they did.)
|
||||
|
||||
Matching is case-insensitive. Unknown `model` → OpenAI-style 404
|
||||
(`model_not_found`), **no wake triggered**.
|
||||
|
||||
### 6.2 Request path
|
||||
|
||||
1. Resolve target service:
|
||||
- Content-Type `application/json` → parse body, use `model` field;
|
||||
missing `model` defaults to text **only for** `/v1/chat/completions`
|
||||
and `/v1/completions`; other endpoints require an explicit model.
|
||||
- Non-JSON bodies (e.g. `multipart/form-data` image uploads for OCR —
|
||||
review C2) → **never fully parse**; extract only the small `model`
|
||||
text field from the multipart parts (without reading file parts) and
|
||||
forward the raw buffered body unchanged. A multipart request on
|
||||
`/v1/chat/completions` with no resolvable `model` **defaults to OCR**
|
||||
(that's the image-bearing path).
|
||||
- `/v1/embeddings` → embed service regardless of body.
|
||||
2. Check service state (cached briefly — don't add a backend round trip to
|
||||
every request):
|
||||
- **awake** → proxy immediately (httpx streaming passthrough, no buffering).
|
||||
- **sleeping/offloading** → single-flight per service under an asyncio
|
||||
lock; concurrent callers await the same event. Re-check `is_sleeping`
|
||||
after acquiring the lock (another path may have just woken or slept it).
|
||||
3. Wake sequence (proven in E2E on 2026-08-14):
|
||||
`POST /wake_up` → `POST /collective_rpc {"method":"reload_weights"}`
|
||||
→ `POST /reset_prefix_cache` → poll `GET /health` until 200.
|
||||
4. Proxy the held request.
|
||||
|
||||
**Proxy timeouts (review M4):** httpx connect timeout 5s; read timeout
|
||||
**900s** (matching today's nginx — capped, never disabled, so a wedged
|
||||
backend stream can't hang forever) for both normal and streaming paths;
|
||||
connection pool sized for ≥16 concurrent requests. A 5s default read timeout
|
||||
would kill nearly every generation.
|
||||
|
||||
**Other public endpoints:**
|
||||
- `/v1/models` → aggregated list of all three registered models (clients
|
||||
commonly list before calling).
|
||||
- `/health` → router liveness + per-service state summary (uptime checks).
|
||||
- `/metrics` → pass through from all services, or omit; decide at
|
||||
implementation (was unauthenticated under nginx anyway).
|
||||
|
||||
### 6.2.1 Depth-aware error semantics
|
||||
|
||||
The router tracks each service's sleep depth from its own actions.
|
||||
**Depth loss after router restart or direct `vllmctl` admin calls:** if
|
||||
sleeping but depth unknown → treat as `offloaded` (conservative — worst case
|
||||
the client waits slightly longer than needed, never gets a too-early retry).
|
||||
|
||||
Hold-first policy: requests queue during wake and only error if the
|
||||
depth-dependent hold deadline is exceeded.
|
||||
|
||||
| Situation | Hold deadline | then Status | `Retry-After` | body `sleep_depth` |
|
||||
|---|---|---|---|---|
|
||||
| Wake from level 1 (sleep, ~1–6s) | 30s | 503 | `10` | `"sleeping"` |
|
||||
| Wake from level 2 (offload, ~15–60s+) | 180s | 503 | `60` | `"offloaded"` |
|
||||
| Container restarting (cold start = 2–10 min NFS load) | 300s | 503 | `600` | `"restarting"` |
|
||||
| Unknown model | — | 404 | — | `model_not_found` |
|
||||
|
||||
503 body (OpenAI-style, parseable):
|
||||
|
||||
```json
|
||||
{"error": {"type": "model_waking", "code": "model_waking",
|
||||
"message": "Model 'OvisOCR2' is waking from offload; retry shortly",
|
||||
"sleep_depth": "offloaded", "estimated_wake_seconds": 45}}
|
||||
```
|
||||
|
||||
**Wake-sequence failure:** retry the sequence once, then the depth-aware 503.
|
||||
Never proxy to a half-awake backend. All deadlines/estimates get re-tuned
|
||||
from latencies logged during calibration (Step 1).
|
||||
|
||||
### 6.3 Tiered idle management (replaces `vllmctl idle-watch`)
|
||||
|
||||
| Tier | Trigger (idle) | Action | State after | Wake cost |
|
||||
|---|---|---|---|---|
|
||||
| Sleep | 15 min (`IDLE_SLEEP_MIN`) | `POST /sleep?level=1` | weights → host RAM | ~1–6s |
|
||||
| Offload | 3 hr (`IDLE_OFFLOAD_MIN`) | `POST /sleep?level=2` | RAM freed; ~5GB/GPU ctx | ~15–60s (NFS) |
|
||||
|
||||
Race-safety (review M1): the idle manager takes the **same per-service
|
||||
asyncio lock** as the wake path and re-checks under the lock that
|
||||
(a) active-request count == 0 and (b) last-activity is still past the
|
||||
threshold. Last-activity is refreshed at request **completion** as well as
|
||||
arrival, and an active-request counter guards long streaming generations —
|
||||
a 20-minute stream must not trigger sleep mid-flight.
|
||||
|
||||
**Verify at implementation:** `/sleep?level=2` on a service already sleeping
|
||||
at level 1 must discard the offloaded weights (not no-op/error). Fallback:
|
||||
wake → re-sleep at level 2.
|
||||
|
||||
**Verify at implementation:** host RAM — level 1 holds a full weights copy
|
||||
(text 43GB + ocr + embed ≈ 57GB total if all three nap simultaneously).
|
||||
Check `free -g` headroom.
|
||||
|
||||
**NFS risk (accepted, documented):** level-2 wake requires a healthy NFS
|
||||
mount. If NFS is down, the model is unwakeable and the router 503s (with
|
||||
`restarting`-grade Retry-After) until NFS returns. Mitigation option: cap the
|
||||
text model at level 1 (weights stay in RAM). Decide after calibration.
|
||||
|
||||
### 6.4 Router restart mid-wake (review M3)
|
||||
|
||||
The wake lock is in-process and dies with the router. The new instance must
|
||||
not fire a second concurrent wake at a backend already waking. Mitigation:
|
||||
the wake sequence is effectively idempotent from the router's view (re-POST
|
||||
`/wake_up` on an already-waking backend is safe; vLLM serializes it), and the
|
||||
request path re-checks `is_sleeping` under the lock before acting.
|
||||
|
||||
### 6.5 Admin API — separate listener, `127.0.0.1:8010` only
|
||||
|
||||
- `GET /admin/status` — per-service: running?, sleeping?, depth, last activity, wake in progress
|
||||
- `POST /admin/wake/{key}` / `POST /admin/sleep/{key}[?level=1|2]` — manual control (updates the router's depth tracking)
|
||||
|
||||
### 6.6 Failure modes
|
||||
|
||||
| Failure | Router behavior |
|
||||
|---|---|
|
||||
| vLLM container down/crashing | Detect unhealthy; hold up to 300s (`restart: unless-stopped` recovers it — but note cold start is 2–10 min for text); then 503 `Retry-After: 600` |
|
||||
| Wake sequence fails | Retry once, then depth-aware 503 (6.2.1); never proxy half-awake |
|
||||
| Router crash | `restart: unless-stopped`; vLLM services unaffected; at most in-flight requests lost; depth resets to conservative `offloaded` |
|
||||
| Host reboot | **All three vLLM containers cold-start AWAKE** (vLLM has no boot-asleep) — see §7; text on GPU0/1 and small services on GPU2 initialize in parallel without racing; recovery takes minutes, router serves `restarting` 503s meanwhile |
|
||||
| NFS outage | Level-2 unwakeable → 503s until NFS returns (§6.3) |
|
||||
| Request without `model` field | Defaults to text **only** on chat/completions; elsewhere 400 |
|
||||
|
||||
## 7. Compose changes
|
||||
|
||||
- Add `router` service (build `./router/`): public `8000:8000`, admin
|
||||
`127.0.0.1:8010:8010`, `depends_on` all vllm services
|
||||
(`condition: service_started`).
|
||||
- **Boot ordering (simplified in v3.3):** no staggering needed — text
|
||||
initializes on GPU0/1 while OCR and embed initialize on GPU2, disjoint
|
||||
memory pools, no race. All services use plain `depends_on:
|
||||
service_started` (sleep makes healthchecks flappy, so never
|
||||
`service_healthy`). The earlier wait-for-healthy entrypoint choreography
|
||||
(review-2 issue #1) is obsolete under this placement.
|
||||
- Split `vllm` into `vllm-text` / `vllm-ocr` / `vllm-embed` with per-model
|
||||
args baked in; `MODEL_NAME`/`EXTRA_ARGS`/`TP_SIZE`/`MAX_MODEL_LEN` in
|
||||
`.env` become obsolete for serving (kept only for rollback).
|
||||
- Remove `nginx` service and `nginx.conf`.
|
||||
- All services: `restart: unless-stopped`, `ipc: host`,
|
||||
`NCCL_P2P_DISABLE=1`, explicit `CUDA_VISIBLE_DEVICES`.
|
||||
|
||||
Per-model serving args (all include `--disable-custom-all-reduce
|
||||
--enable-sleep-mode`, **explicit `--max-model-len` each** — review M2):
|
||||
|
||||
- **text:** `--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.85`
|
||||
- **ocr:** `--max-model-len 32768 --gpu-memory-utilization 0.10` (+ vision args from OvisOCR2 model card, e.g. `--limit-mm-per-prompt`)
|
||||
- **embed:** `--max-model-len 8192 --gpu-memory-utilization 0.25`
|
||||
|
||||
## 8. `vllmctl` changes
|
||||
|
||||
- `up [MODEL]` / `down [MODEL]` → HTTP to `127.0.0.1:8010` admin API (no
|
||||
docker needed for routine control).
|
||||
- `status` → `GET 127.0.0.1:8010/admin/status` pretty-printed.
|
||||
- `pull` unchanged (docker, rare, manual).
|
||||
- `idle-watch` removed (router owns idle management).
|
||||
|
||||
## 9. Implementation order
|
||||
|
||||
0. **Backups + cutover plan (review M5):**
|
||||
`cp compose.yml compose.yml.v2.bak; cp vllmctl vllmctl.v2.bak`.
|
||||
**Status 2026-08-14: `renbaibing` is now in the docker group** (works
|
||||
directly in fresh login shells; use `sg docker -c "…"` in sessions
|
||||
started before the change — `.runas.py`/`.user.env` are obsolete).
|
||||
Old stack already brought down by user; GPUs idle. Rollback: restore
|
||||
`.v2.bak` files, `docker compose up -d`.
|
||||
1. **Calibration (gates everything):** bring up the three services with
|
||||
initial slices (old stack stopped); measure awake footprints, sleeping
|
||||
residuals, level-1 and level-2 wake latencies, host RAM cost; adjust
|
||||
slices so each GPU ≤ 0.88 including contexts. Log latencies → tune the
|
||||
§6.2.1 deadlines from data.
|
||||
2. `router/` app (registry, wake single-flight, raw-body-safe routing,
|
||||
streaming proxy with proper timeouts, tiered idle with lock, admin
|
||||
listener) + compose rewiring; nginx removed.
|
||||
3. `vllmctl` rewrite of up/down/status.
|
||||
4. E2E verification (below).
|
||||
5. Docs: README, TODO, memory.
|
||||
|
||||
## 10. E2E test matrix
|
||||
|
||||
1. All asleep → `POST /v1/chat/completions` (text) → 200 after wake; latency logged.
|
||||
2. While text awake → `POST /v1/embeddings` → embed wakes independently; both serve.
|
||||
3. OCR request with **`multipart/form-data`** body (not just JSON base64) → routes correctly, no body parsing error.
|
||||
4. Idle tiers (shortened timers): 1 min → level-1 sleep (host RAM grows by weights); 3 min → level-2 offload (RAM freed); auto-wake from both.
|
||||
5. Streaming request during wake → first chunk after wake, no buffering.
|
||||
6. Unknown model name → 404 `model_not_found`, no wake triggered.
|
||||
7. Dev endpoints (`/sleep`, `/wake_up`, `/collective_rpc`, `/reset_prefix_cache`) **and** `/admin/*` on public :8000 → 404; admin reachable only via 127.0.0.1:8010.
|
||||
8. Kill vllm-text container → request → `restarting` 503 (`Retry-After: 600`) during recovery → 200 after restart completes.
|
||||
9. Concurrent 10× requests while sleeping → exactly one wake sequence (router logs).
|
||||
10. Depth-aware errors (hold deadline forced to 1s): level-1 wake → 503 `Retry-After: 10`, `"sleeping"`; level-2 → `Retry-After: 60`, `"offloaded"`; body parses with `estimated_wake_seconds`.
|
||||
11. Idle-race: fire a request at the moment the idle timer expires → request served, no sleep mid-request (lock + active-counter verified).
|
||||
12. Router restart mid-wake → no double-wake crash; request eventually served.
|
||||
13. Two services waking concurrently (text + embed requested simultaneously) → both complete, no lock cross-talk.
|
||||
14. Path traversal: `/v1/../sleep`, `/v1%2f..%2fsleep` → 404.
|
||||
15. Host reboot (if feasible to test): parallel boot (text on GPU0/1, small services on GPU2 — no race), router serves `restarting` 503s, all three become available.
|
||||
|
||||
## 11. Risks / open items
|
||||
|
||||
- **GPU slice calibration** gates final numbers (§5, Step 1). v3.3 placement
|
||||
(text exclusive on GPU0/1 at 0.85) removes the KV-starvation concern;
|
||||
calibration confirms.
|
||||
- **GPU2 trust:** single-GPU tests pass (2026-08-17, twice), but it has a
|
||||
fault history (Aug 11 driver re-probe). Mitigation: E2E test 3+4 exercise
|
||||
it before rollout; if GPU2 degrades in production, move OCR/embed back to
|
||||
GPU0/1 with one env-var change each (slices re-tighten to v3.2 values).
|
||||
- ~~Which embedding model~~ — **resolved 2026-08-14: Qwen3-Embedding-8B.**
|
||||
- **Level 1→2 escalation** and **host RAM** (~57GB if all three nap) need verification (§6.3).
|
||||
- **NFS outage strands level-2 models** — accepted risk or cap text at level 1 (§6.3).
|
||||
- Wake latency must fit calling services' client timeouts — confirm their settings before rollout.
|
||||
783
.claude/memory/sleep-mode-implementation-plan.md
Normal file
783
.claude/memory/sleep-mode-implementation-plan.md
Normal file
@@ -0,0 +1,783 @@
|
||||
# Implementation Plan: vLLM Sleep Mode + nginx Front Door
|
||||
|
||||
**Date:** 2026-08-14
|
||||
**Status:** Revised (v2) — Critical issues from review addressed
|
||||
|
||||
## Changelog (v2)
|
||||
|
||||
**Fixed Critical Issues:**
|
||||
1. ✅ Healthcheck dependency — changed to `service_started` (doesn't require healthy)
|
||||
2. ✅ nginx upstream host — changed from `127.0.0.1:8001` to `vllm:8000`
|
||||
3. ✅ Port binding clarified — vLLM: `127.0.0.1:8001:8000` (localhost only), nginx: external 8000
|
||||
4. ✅ Added `--enable-sleep-mode` flag to EXTRA_ARGS
|
||||
5. ✅ Implemented idle watcher update with `/sleep` calls
|
||||
|
||||
**Important Concerns Addressed:**
|
||||
- Added note about `/metrics` exposure
|
||||
- Clarified in-flight request handling during `/sleep`
|
||||
- Updated wake time estimate to be more realistic
|
||||
- Removed redundant `return 403` after `deny all`
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Transform the current vLLM setup from full container teardown (`docker compose down`) to vLLM Sleep Mode, with nginx as a security front door.
|
||||
|
||||
### Current Architecture
|
||||
|
||||
```
|
||||
Client → [vLLM Container :8000]
|
||||
↓
|
||||
down = docker compose down (full teardown)
|
||||
up = docker compose up (cold start, 2-10 min)
|
||||
```
|
||||
|
||||
### Target Architecture
|
||||
|
||||
```
|
||||
Internet → [nginx :8000] → [vLLM :8000 (Docker network) ]
|
||||
↓
|
||||
[vLLM :8001 (localhost only) ]
|
||||
↓
|
||||
sleep = POST /sleep (model hibernates, GPU freed)
|
||||
wake = POST /wake_up (model resumes, ~5-15s for 35B)
|
||||
```
|
||||
|
||||
### Port Binding Clarification
|
||||
|
||||
| Port | Access | Purpose |
|
||||
|------|--------|---------|
|
||||
| `8000` (nginx) | External (0.0.0.0:8000) | Public API, clients connect here |
|
||||
| `8001` (vLLM) | Localhost only (127.0.0.1:8001) | Admin access for `vllmctl`, sleep/wake endpoints |
|
||||
| `8000` (Docker network) | Internal | nginx → vLLM communication within Docker network |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Enable vLLM Sleep Mode
|
||||
|
||||
### 1.1 Update `compose.yml`
|
||||
|
||||
**Critical changes:**
|
||||
|
||||
1. Add `VLLM_SERVER_DEV_MODE=1` to environment (required for Sleep Mode endpoints)
|
||||
2. Add `--enable-sleep-mode` to EXTRA_ARGS (this was missing in v1!)
|
||||
3. Expose vLLM on localhost only: `127.0.0.1:8001:8000`
|
||||
4. Keep vLLM accessible on Docker network at port 8000
|
||||
|
||||
**Before:**
|
||||
```yaml
|
||||
services:
|
||||
vllm:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION:-v0.27.1}
|
||||
container_name: vllm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${VLLM_PORT:-8000}:8000"
|
||||
volumes:
|
||||
- ${MODEL_ROOT:-/data/home/renbaibing/huggingface}:/models:ro
|
||||
ipc: host
|
||||
gpus: all
|
||||
environment:
|
||||
- HF_TOKEN=${HF_TOKEN:-}
|
||||
- MODEL_NAME=${MODEL_NAME}
|
||||
- TP_SIZE=${TP_SIZE:-2}
|
||||
- MAX_MODEL_LEN=${MAX_MODEL_LEN:-32768}
|
||||
- GPU_MEM_UTIL=${GPU_MEM_UTIL:-}
|
||||
- EXTRA_ARGS=${EXTRA_ARGS:-}
|
||||
- NCCL_P2P_DISABLE=${NCCL_P2P_DISABLE:-1}
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)\" || exit 1"]
|
||||
interval: 15s
|
||||
timeout: 10s
|
||||
retries: 8
|
||||
start_period: 1200s
|
||||
```
|
||||
|
||||
**After:**
|
||||
```yaml
|
||||
services:
|
||||
vllm:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION:-v0.27.1}
|
||||
container_name: vllm
|
||||
restart: unless-stopped
|
||||
|
||||
# Two port bindings:
|
||||
# 1. Docker network port 8000 (internal, for nginx)
|
||||
# 2. Localhost port 8001 (admin access for vllmctl)
|
||||
ports:
|
||||
- "127.0.0.1:8001:8000" # Admin access - localhost only
|
||||
expose:
|
||||
- "8000" # Docker network - for nginx
|
||||
|
||||
volumes:
|
||||
- ${MODEL_ROOT:-/data/home/renbaibing/huggingface}:/models:ro
|
||||
ipc: host
|
||||
gpus: all
|
||||
|
||||
environment:
|
||||
- VLLM_SERVER_DEV_MODE=1 # Required for Sleep Mode endpoints
|
||||
- HF_TOKEN=${HF_TOKEN:-}
|
||||
- MODEL_NAME=${MODEL_NAME}
|
||||
- TP_SIZE=${TP_SIZE:-2}
|
||||
- MAX_MODEL_LEN=${MAX_MODEL_LEN:-32768}
|
||||
- GPU_MEM_UTIL=${GPU_MEM_UTIL:-}
|
||||
# EXTRA_ARGS MUST include --enable-sleep-mode
|
||||
- EXTRA_ARGS=${EXTRA_ARGS:- --enable-sleep-mode}
|
||||
- NCCL_P2P_DISABLE=${NCCL_P2P_DISABLE:-1}
|
||||
|
||||
# Healthcheck stays the same - container will be "unhealthy" during sleep
|
||||
# This is OK - nginx uses service_started, not service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)\" || exit 1"]
|
||||
interval: 15s
|
||||
timeout: 10s
|
||||
retries: 8
|
||||
start_period: 1200s
|
||||
```
|
||||
|
||||
**Rationale:**
|
||||
- `VLLM_SERVER_DEV_MODE=1` enables dev endpoints including `/sleep`, `/wake_up`, `/is_sleeping`
|
||||
- `--enable-sleep-mode` in EXTRA_ARGS is REQUIRED for sleep functionality (this was the critical missing piece in v1)
|
||||
- Port `127.0.0.1:8001` gives `vllmctl` localhost-only access to admin endpoints
|
||||
- `expose: - "8000"` makes port 8000 available on Docker network for nginx
|
||||
- Healthcheck will fail during sleep, but that's OK — nginx uses `service_started` condition
|
||||
|
||||
### 1.2 Update `vllmctl`
|
||||
|
||||
**First, update the API URL:**
|
||||
|
||||
```bash
|
||||
# Change at the top of vllmctl:
|
||||
VLLM_PORT="$(env_get VLLM_PORT 8000)"
|
||||
API="http://127.0.0.1:8001" # Direct to vLLM localhost port, bypasses nginx
|
||||
```
|
||||
|
||||
**Modify `cmd_down()`:**
|
||||
|
||||
```bash
|
||||
cmd_down() {
|
||||
say 'Putting vLLM to sleep (frees GPU memory, keeps server alive)...'
|
||||
|
||||
# Try to sleep the model (level 2 = discard weights, minimal RAM)
|
||||
if curl -fs -X POST "$API/sleep?level=2" >/dev/null 2>&1; then
|
||||
ok "Model is sleeping. Server still running."
|
||||
|
||||
# Wait a moment for sleep to complete
|
||||
sleep 2
|
||||
|
||||
# Verify sleep state
|
||||
local sleeping
|
||||
sleeping="$(curl -fs "$API/is_sleeping" 2>/dev/null)"
|
||||
if [ "$sleeping" = "true" ]; then
|
||||
ok "Confirmed: Model is in sleep state."
|
||||
else
|
||||
warn "Sleep state unclear - check with: curl $API/is_sleeping"
|
||||
fi
|
||||
else
|
||||
warn "Sleep request failed — server may not be ready. Falling back to full stop."
|
||||
dcompose down -t 20 || return 1
|
||||
fi
|
||||
|
||||
# Show GPU memory after sleep
|
||||
local used
|
||||
used="$(nvidia-smi --query-gpu=index,memory.used --format=csv,noheader,nounits 2>/dev/null | awk -F', *' '{printf "GPU%s:%sMiB ", $1, $2}')"
|
||||
say "GPU memory now: $used"
|
||||
}
|
||||
```
|
||||
|
||||
**Rationale:**
|
||||
- `level=2` discards weights entirely (minimal RAM usage)
|
||||
- Wake time for 35B model: ~5-15s (more realistic than the 0.8-2.6s estimate)
|
||||
- Adds sleep verification to confirm the operation worked
|
||||
- Falls back to full stop if sleep fails
|
||||
|
||||
**Modify `cmd_up()`:**
|
||||
|
||||
```bash
|
||||
cmd_up() {
|
||||
local model="${1:-}"
|
||||
if [ -n "$model" ]; then
|
||||
if [ ! -d "$MODEL_ROOT/$model" ]; then
|
||||
err "Model '$model' not found in $MODEL_ROOT"
|
||||
err "Available:"; cmd_list
|
||||
return 1
|
||||
fi
|
||||
env_set MODEL_NAME "$model"
|
||||
fi
|
||||
model="$(env_get MODEL_NAME)"
|
||||
[ -n "$model" ] || { err 'No MODEL_NAME set in .env'; return 1; }
|
||||
|
||||
local st
|
||||
st="$(container_state)"
|
||||
|
||||
# Case 1: Container not running at all
|
||||
case "$st" in
|
||||
absent*|exited*|dead*)
|
||||
say "Starting vLLM with model '${C_CYAN}$model${C_OFF}' (TP=$(env_get TP_SIZE 2), Sleep Mode enabled)…"
|
||||
dcompose up -d || return 1
|
||||
wait_for_ready "$model"
|
||||
return 0
|
||||
;;
|
||||
esac
|
||||
|
||||
# Case 2: Container running - check if sleeping or different model
|
||||
local sleeping loaded
|
||||
sleeping="$(curl -fs "$API/is_sleeping" 2>/dev/null)"
|
||||
loaded="$(loaded_model)"
|
||||
|
||||
# Model already loaded and serving
|
||||
if [ "$sleeping" != "true" ] && [ "$loaded" = "$model" ]; then
|
||||
ok "Already serving '$model'."
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Model is sleeping - wake it up
|
||||
if [ "$sleeping" = "true" ]; then
|
||||
say "Model is sleeping. Waking up '${C_CYAN}$model${C_OFF}'…"
|
||||
# Level 2 sleep requires reload_weights after wake_up
|
||||
curl -fs -X POST "$API/wake_up" >/dev/null 2>&1 || {
|
||||
err "Wake-up request failed."
|
||||
return 1
|
||||
}
|
||||
# Reload weights (required for level 2 sleep)
|
||||
curl -fs -X POST "$API/collective_rpc" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"method":"reload_weights"}' >/dev/null 2>&1 || {
|
||||
warn "reload_weights request failed - model may still wake up"
|
||||
}
|
||||
# Reset prefix cache (required for level 2 sleep)
|
||||
curl -fs -X POST "$API/reset_prefix_cache" >/dev/null 2>&1 || {
|
||||
warn "reset_prefix_cache failed - non-critical"
|
||||
}
|
||||
# Wait for model to be ready
|
||||
wait_for_ready "$model"
|
||||
ok "Model '$model' is awake and ready."
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Different model loaded - need to switch
|
||||
if [ -n "$loaded" ] && [ "$loaded" != "$model" ]; then
|
||||
say "Switching from '${C_YELLOW}$loaded${C_OFF}' to '${C_CYAN}$model${C_OFF}'…"
|
||||
say "Restarting container with new model (this will take a few minutes)…"
|
||||
dcompose down -t 20
|
||||
sleep 2
|
||||
dcompose up -d || return 1
|
||||
wait_for_ready "$model"
|
||||
ok "Now serving '$model'."
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Shouldn't reach here, but handle gracefully
|
||||
err "Unexpected state - container running but no model loaded. Try: ./vllmctl restart"
|
||||
return 1
|
||||
}
|
||||
```
|
||||
|
||||
**Rationale:**
|
||||
- Now handles the sleeping state properly
|
||||
- For level 2 sleep, calls the required sequence: `/wake_up` → `reload_weights` → `reset_prefix_cache`
|
||||
- Keeps container restart for model switching (cleaner than in-place reload)
|
||||
- Better error messages and state handling
|
||||
|
||||
**Modify `cmd_status()`:**
|
||||
|
||||
```bash
|
||||
cmd_status() {
|
||||
local st loaded cfg sleeping
|
||||
st="$(container_state)"
|
||||
cfg="$(env_get MODEL_NAME)"
|
||||
|
||||
say "${C_CYAN}Container:${C_OFF} $st"
|
||||
|
||||
# Check if sleeping first (loaded_model won't work when sleeping)
|
||||
sleeping="$(curl -fs "$API/is_sleeping" 2>/dev/null)"
|
||||
|
||||
if [ "$sleeping" = "true" ]; then
|
||||
say "${C_CYAN}Serving:${C_OFF} ${C_DIM}model sleeping${C_OFF} (configured: $cfg)"
|
||||
else
|
||||
loaded="$(loaded_model)"
|
||||
if [ -n "$loaded" ]; then
|
||||
say "${C_CYAN}Serving:${C_OFF} ${C_GREEN}$loaded${C_OFF} at http://127.0.0.1:${VLLM_PORT}/v1"
|
||||
else
|
||||
say "${C_CYAN}Serving:${C_OFF} ${C_DIM}nothing loaded${C_OFF} (configured: $cfg)"
|
||||
fi
|
||||
fi
|
||||
|
||||
say "${C_CYAN}GPU:${C_OFF}"
|
||||
nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu \
|
||||
--format=csv,noheader 2>/dev/null | sed 's/^/ /'
|
||||
|
||||
# External access (nginx)
|
||||
say "${C_CYAN}Public API:${C_OFF} http://$(hostname -f | head -1):${VLLM_PORT}/v1"
|
||||
say "${C_CYAN}Admin API:${C_OFF} $API (localhost only)"
|
||||
|
||||
# Idle watcher status
|
||||
local pidfile="$ROOT/.idle.pid"
|
||||
if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile" 2>/dev/null)" 2>/dev/null; then
|
||||
say "${C_CYAN}Idle-watch:${C_OFF} active (pid $(cat "$pidfile), $(cat "$ROOT/.idle.minutes" 2>/dev/null) min timeout)"
|
||||
else
|
||||
say "${C_CYAN}Idle-watch:${C_OFF} ${C_DIM}off${C_OFF} (enable: ./vllmctl idle-watch on [minutes])"
|
||||
fi
|
||||
}
|
||||
```
|
||||
|
||||
**Modify `idle_loop()` — CRITICAL FIX:**
|
||||
|
||||
```bash
|
||||
idle_loop() {
|
||||
local timeout_min="$1"
|
||||
local timeout_s=$(( timeout_min * 60 ))
|
||||
local last_active=0
|
||||
say "[idle-watch] started: will auto-sleep after ${timeout_min} min without requests (poll 30s)"
|
||||
|
||||
# Use internal API for sleep calls
|
||||
local SLEEP_API="http://127.0.0.1:8001"
|
||||
|
||||
while true; do
|
||||
sleep 30
|
||||
|
||||
# Check if container is running at all
|
||||
local st
|
||||
st="$(container_state)"
|
||||
case "$st" in
|
||||
absent*|exited*|dead*)
|
||||
# Container not running - nothing to do
|
||||
last_active=0
|
||||
continue
|
||||
;;
|
||||
esac
|
||||
|
||||
# Check if already sleeping
|
||||
local sleeping
|
||||
sleeping="$(curl -fs "$SLEEP_API/is_sleeping" 2>/dev/null)"
|
||||
if [ "$sleeping" = "true" ]; then
|
||||
# Already sleeping, nothing to do
|
||||
continue
|
||||
fi
|
||||
|
||||
# Check for activity using metrics
|
||||
local idle_s='' run_wait=0
|
||||
idle_s="$(curl -fs -m 5 "$SLEEP_API/metrics" 2>/dev/null \
|
||||
| awk '/^vllm:time_since_last_request_seconds/ {print $2; exit}')"
|
||||
|
||||
if [ -z "$idle_s" ] || [ "$idle_s" = "+Inf" ] || [ "$idle_s" = "NaN" ]; then
|
||||
# Fall back: count running/waiting requests
|
||||
run_wait="$(curl -fs -m 5 "$SLEEP_API/metrics" 2>/dev/null \
|
||||
| awk '/^vllm:num_requests_(running|waiting)/ {s+=$2} END {print s+0}')"
|
||||
if [ "${run_wait:-0}" != "0" ]; then
|
||||
last_active="$(date +%s)"
|
||||
continue
|
||||
fi
|
||||
[ "$last_active" = 0 ] && last_active="$(date +%s)"
|
||||
idle_s=$(( $(date +%s) - last_active ))
|
||||
fi
|
||||
|
||||
# Integer compare (strip possible decimals)
|
||||
local idle_i=${idle_s%%.*}
|
||||
if [ -n "$idle_i" ] && [ "$idle_i" -ge "$timeout_s" ] 2>/dev/null; then
|
||||
say "[idle-watch] idle for ${idle_i}s ≥ ${timeout_s}s → putting model to sleep"
|
||||
if curl -fs -X POST "$SLEEP_API/sleep?level=2" >/dev/null 2>&1; then
|
||||
say "[idle-watch] model is now sleeping at $(date '+%F %T')"
|
||||
else
|
||||
warn '[idle-watch] sleep request failed - check if server is responsive'
|
||||
fi
|
||||
last_active=0
|
||||
fi
|
||||
done
|
||||
}
|
||||
```
|
||||
|
||||
**Rationale:**
|
||||
- Uses internal API (`:8001`) for all calls
|
||||
- Checks if container is running before attempting API calls
|
||||
- Checks if already sleeping (avoid redundant sleep calls)
|
||||
- Calls `/sleep?level=2` instead of `dcompose down`
|
||||
- Graceful error handling if server is unresponsive
|
||||
|
||||
### 1.3 Update `.env`
|
||||
|
||||
No changes needed — everything is in `compose.yml` and `vllmctl`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Add nginx Front Door
|
||||
|
||||
### 2.1 Create `nginx.conf`
|
||||
|
||||
```nginx
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
# Rate limiting (optional, can be commented out)
|
||||
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
|
||||
|
||||
# Upstream using Docker service name - CRITICAL FIX
|
||||
upstream vllm {
|
||||
server vllm:8000; # Docker service name, not 127.0.0.1
|
||||
keepalive 32;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8000;
|
||||
server_name _;
|
||||
|
||||
# Block dev endpoints — deny all is sufficient
|
||||
location /sleep {
|
||||
deny all;
|
||||
}
|
||||
location /wake_up {
|
||||
deny all;
|
||||
}
|
||||
location /is_sleeping {
|
||||
deny all;
|
||||
}
|
||||
location /collective_rpc {
|
||||
deny all;
|
||||
}
|
||||
location /reset_prefix_cache {
|
||||
deny all;
|
||||
}
|
||||
|
||||
# Main API proxy
|
||||
location / {
|
||||
limit_req zone=api_limit burst=20 nodelay;
|
||||
|
||||
proxy_pass http://vllm;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
# Headers for WebSocket/streaming support
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Timeouts for LLM inference (increased from v1)
|
||||
proxy_read_timeout 900s;
|
||||
proxy_send_timeout 900s;
|
||||
proxy_connect_timeout 10s;
|
||||
|
||||
# Disable buffering for streaming
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# Health endpoint (for health checks, allows unhealthy during sleep)
|
||||
location /health {
|
||||
proxy_pass http://vllm/health;
|
||||
access_log off;
|
||||
# Don't fail if unhealthy - vLLM can be sleeping
|
||||
proxy_next_upstream error timeout http_502 http_503 http_504;
|
||||
}
|
||||
|
||||
# Metrics endpoint (WARNING: exposed without auth)
|
||||
# Consider adding authentication if this becomes publicly accessible
|
||||
location /metrics {
|
||||
proxy_pass http://vllm/metrics;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Rationale:**
|
||||
- Fixed: `server vllm:8000` uses Docker service name (was `127.0.0.1:8001` in v1)
|
||||
- Removed redundant `return 403` after `deny all`
|
||||
- Increased timeouts to 900s for long-running inference
|
||||
- Added `proxy_next_upstream` for health endpoint tolerance
|
||||
- Added warning comment about `/metrics` exposure
|
||||
|
||||
### 2.2 Update `compose.yml`
|
||||
|
||||
Add nginx service with CRITICAL FIX to dependency:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: vllm-nginx
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${VLLM_PORT:-8000}:8000"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
# CRITICAL FIX: Use service_started, not service_healthy
|
||||
# vLLM healthcheck fails during sleep, but that's OK
|
||||
depends_on:
|
||||
vllm:
|
||||
condition: service_started # Changed from service_healthy
|
||||
networks:
|
||||
- default
|
||||
|
||||
vllm:
|
||||
# ... (as shown in Phase 1.1)
|
||||
```
|
||||
|
||||
**Rationale:**
|
||||
- **CRITICAL FIX**: Changed from `service_healthy` to `service_started`
|
||||
- This allows nginx to start even when vLLM healthcheck fails (during sleep)
|
||||
- nginx will still return 502 if vLLM is completely down, which is correct behavior
|
||||
|
||||
### 2.3 Port Binding Summary
|
||||
|
||||
After all changes, the port layout is:
|
||||
|
||||
| From | To | Who Can Access | Purpose |
|
||||
|-----|-----|----------------|---------|
|
||||
| `0.0.0.0:8000` | nginx:8000 | Everyone | Public API |
|
||||
| nginx | vllm:8000 | Docker only | nginx→vLLM |
|
||||
| `127.0.0.1:8001` | vllm:8000 | Localhost only | Admin (`vllmctl`) |
|
||||
|
||||
**Security Model:**
|
||||
- External clients hit nginx on port 8000
|
||||
- nginx blocks `/sleep`, `/wake_up`, `/is_sleeping`, etc.
|
||||
- `vllmctl` uses localhost:8001 to bypass nginx and access admin endpoints
|
||||
- vLLM containers can talk to each other on Docker network at vllm:8000
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Testing & Validation
|
||||
|
||||
### 3.1 Test Sleep Mode
|
||||
|
||||
```bash
|
||||
# 1. Start model
|
||||
./vllmctl up Qwen3.6-35B-A3B
|
||||
|
||||
# 2. Verify serving through nginx
|
||||
curl http://localhost:8000/v1/models
|
||||
# Should show: {"object":"list","data":[{"id":"Qwen3.6-35B-A3B",...}]}
|
||||
|
||||
# 3. Put to sleep
|
||||
./vllmctl down
|
||||
# Should show: "Model is sleeping. Server still running."
|
||||
|
||||
# 4. Check GPU memory freed
|
||||
nvidia-smi
|
||||
# GPU memory should be significantly lower
|
||||
|
||||
# 5. Verify external API blocked - dev endpoints
|
||||
curl http://localhost:8000/sleep
|
||||
# Should return: 403 Forbidden
|
||||
|
||||
# 6. Verify internal API works
|
||||
curl http://localhost:8001/is_sleeping
|
||||
# Should return: true
|
||||
|
||||
# 7. Test public API during sleep
|
||||
curl http://localhost:8000/v1/models
|
||||
# Should return error or timeout (model is sleeping)
|
||||
|
||||
# 8. Wake up
|
||||
./vllmctl up
|
||||
# Should wake up the existing model
|
||||
|
||||
# 9. Verify serving again
|
||||
curl http://localhost:8000/v1/models
|
||||
# Should work again
|
||||
```
|
||||
|
||||
### 3.2 Test Idle Watcher
|
||||
|
||||
```bash
|
||||
# Enable short idle timeout for testing
|
||||
./vllmctl idle-watch on 1
|
||||
|
||||
# Wait 1 minute, then check status
|
||||
./vllmctl status
|
||||
# Should show: "model sleeping"
|
||||
|
||||
# Test that wake-up works
|
||||
./vllmctl up
|
||||
./vllmctl status
|
||||
# Should show: serving the model
|
||||
|
||||
# Turn off when done
|
||||
./vllmctl idle-watch off
|
||||
```
|
||||
|
||||
### 3.3 Test Error Handling
|
||||
|
||||
```bash
|
||||
# 1. Stop vLLM container
|
||||
docker compose stop vllm
|
||||
|
||||
# 2. Try request through nginx
|
||||
curl http://localhost:8000/v1/models
|
||||
# Should get: 502 Bad Gateway
|
||||
|
||||
# 3. Start vLLM again
|
||||
docker compose start vllm
|
||||
|
||||
# 4. Verify recovery
|
||||
curl http://localhost:8000/v1/models
|
||||
# Should work again
|
||||
```
|
||||
|
||||
### 3.4 Test In-Flight Request Handling
|
||||
|
||||
**Note:** vLLM Sleep Mode will allow in-flight requests to complete before sleeping. Test:
|
||||
|
||||
```bash
|
||||
# Start a long-running request in background
|
||||
curl -N http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"Qwen3.6-35B-A3B","prompt":"Tell me a long story","max_tokens":500}' &
|
||||
CURL_PID=$!
|
||||
|
||||
# Immediately put to sleep
|
||||
./vllmctl down
|
||||
|
||||
# The request should complete (may take a moment)
|
||||
wait $CURL_PID
|
||||
echo "Request completed with exit code: $?"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Optional Enhancements
|
||||
|
||||
### 4.1 Protect /metrics Endpoint
|
||||
|
||||
If `/metrics` exposure is a concern:
|
||||
|
||||
```nginx
|
||||
location /metrics {
|
||||
# Optional: basic auth
|
||||
# auth_basic "Metrics";
|
||||
# auth_basic_user_file /etc/nginx/.htpasswd;
|
||||
|
||||
# Or restrict to localhost
|
||||
# allow 127.0.0.1;
|
||||
# deny all;
|
||||
|
||||
proxy_pass http://vllm/metrics;
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 Friendly Error When Model Is Sleeping
|
||||
|
||||
Add Lua-based checking (requires nginx with Lua module):
|
||||
|
||||
```nginx
|
||||
# This requires nginx-lua or OpenResty - can add later if needed
|
||||
```
|
||||
|
||||
### 4.3 Request Logging
|
||||
|
||||
Enable access logging for debugging:
|
||||
|
||||
```nginx
|
||||
http {
|
||||
access_log /var/log/nginx/access.log;
|
||||
error_log /var/log/nginx/error.log;
|
||||
# ...
|
||||
}
|
||||
```
|
||||
|
||||
And mount log volume in compose.yml:
|
||||
|
||||
```yaml
|
||||
nginx:
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./logs/nginx:/var/log/nginx
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Steps
|
||||
|
||||
1. **Backup current setup:**
|
||||
```bash
|
||||
cp compose.yml compose.yml.bak
|
||||
cp vllmctl vllmctl.bak
|
||||
```
|
||||
|
||||
2. **Update `compose.yml`:**
|
||||
- Add nginx service
|
||||
- Update vLLM service with new ports and environment
|
||||
- Add `--enable-sleep-mode` to EXTRA_ARGS
|
||||
|
||||
3. **Create `nginx.conf`:**
|
||||
- Copy the config above
|
||||
- Adjust as needed
|
||||
|
||||
4. **Update `vllmctl`:**
|
||||
- Change API URL to `:8001`
|
||||
- Modify `cmd_down()`, `cmd_up()`, `cmd_status()`, `idle_loop()`
|
||||
- Use the code blocks from Phase 1.2
|
||||
|
||||
5. **Test:**
|
||||
```bash
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
./vllmctl status
|
||||
```
|
||||
|
||||
6. **Rollback if needed:**
|
||||
```bash
|
||||
docker compose down
|
||||
cp compose.yml.bak compose.yml
|
||||
cp vllmctl.bak vllmctl
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Dev endpoints are protected** — nginx blocks `/sleep`, `/wake_up`, `/is_sleeping`, `/collective_rpc`, `/reset_prefix_cache`
|
||||
2. **Internal access only** — `vllmctl` uses port 8001 which is bound to localhost only
|
||||
3. **Container isolation** — vLLM and nginx are in the same Docker network
|
||||
4. **WARNING: /metrics is exposed** — Consider adding authentication if deployment becomes public
|
||||
5. **VLLM_SERVER_DEV_MODE=1** — This enables dev endpoints; ensure nginx properly blocks them
|
||||
|
||||
---
|
||||
|
||||
## Performance Impact
|
||||
|
||||
- **nginx overhead**: Minimal (~1-2ms latency, sub-1% CPU)
|
||||
- **Sleep Mode wake**: ~5-15s for level 2 with 35B model (realistic estimate)
|
||||
- **Memory**: Level 2 sleep uses minimal CPU RAM, frees ~90% GPU memory
|
||||
- **In-flight requests**: vLLM allows completion before sleep
|
||||
|
||||
---
|
||||
|
||||
## Open Questions (Resolved)
|
||||
|
||||
1. ✅ **Should idle watcher call `/sleep` directly?**
|
||||
- **RESOLVED**: Yes, updated `idle_loop()` to call `/sleep?level=2` instead of `dcompose down`
|
||||
|
||||
2. **Should we optimize model switching?**
|
||||
- Current: Restart container on model switch (clean, reliable)
|
||||
- Could optimize later: Use `/sleep` + `/wake_up` + `reload_weights` for faster switches
|
||||
- Decision: Start with restart, optimize later if needed
|
||||
|
||||
3. ⚠️ **What about `/metrics` endpoint?**
|
||||
- Currently accessible through nginx (no auth)
|
||||
- **Decision**: Leave open for now, add auth later if deployment becomes public
|
||||
|
||||
---
|
||||
|
||||
## Estimated Effort (Updated)
|
||||
|
||||
- Phase 1 (vLLM Sleep Mode): ~45 minutes (increased due to more comprehensive changes)
|
||||
- Phase 2 (nginx): ~30 minutes
|
||||
- Phase 3 (Testing): ~45 minutes (more thorough testing)
|
||||
- **Total**: ~2 hours (updated from 1.5 hours)
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
Once you approve this revised plan, I'll:
|
||||
1. Create the modified files (`compose.yml`, `nginx.conf`, `vllmctl`)
|
||||
2. Test in a worktree or provide step-by-step instructions
|
||||
3. Document rollback procedure
|
||||
|
||||
**All critical issues from the first review have been addressed.**
|
||||
|
||||
Ready to proceed?
|
||||
Reference in New Issue
Block a user