Initial commit: router front-door vLLM stack
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2, sleep mode) behind a FastAPI router that auto-wakes models on request. Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010. Routine control via vllmctl is pure HTTP — no docker on the request path. Verified: 91 router unit tests + 15-test E2E on real hardware (measurements in CALIBRATION.md; design record in .claude/memory/router-front-door-plan.md). Old nginx stack files removed before git init; design survives in .claude/memory/sleep-mode-implementation-plan.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
164
.claude/memory/front-door-proxy-design.md
Normal file
164
.claude/memory/front-door-proxy-design.md
Normal file
@@ -0,0 +1,164 @@
|
||||
# Front Door Proxy Design — Handling Requests to Unloaded Models
|
||||
|
||||
**Date:** 2026-08-13
|
||||
**Status:** Design exploration — vLLM Sleep Mode available as better alternative
|
||||
|
||||
> **TL;DR:** vLLM 0.11.0+ includes **Sleep Mode** — a built-in feature that enables fast model switching (18-200x faster than full reload) by hibernating weights while preserving process state. **This is likely the solution we need instead of building a custom front door.**
|
||||
|
||||
## Current Behavior: Request to Unloaded Model
|
||||
|
||||
When a client makes a request to `http://<host>:8000/v1/...` and no model is loaded:
|
||||
|
||||
1. **Container is down** (auto-unload via `idle-watch` or manual `down`)
|
||||
2. **Immediate connection failure** — client gets `Connection refused` or similar
|
||||
3. **No retry guidance** — client doesn't know if the service is down, moving, or loading
|
||||
4. **Manual recovery required** — someone must run `./vllmctl up <model>`
|
||||
|
||||
### What the client sees
|
||||
|
||||
```bash
|
||||
$ curl http://host:8000/v1/models
|
||||
curl: (7) Failed to connect to host port 8000: Connection refused
|
||||
```
|
||||
|
||||
No indication of:
|
||||
- Whether the service is permanently down
|
||||
- Which model should be loaded
|
||||
- How long loading would take (~2-10 min for large models from NFS)
|
||||
|
||||
## The Shared Front Door Concept
|
||||
|
||||
A reverse proxy that sits in front of vLLM and handles the "unloaded model" case gracefully.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
Client → Front Door (port 8000) → vLLM Container (port 8001 or similar)
|
||||
↓
|
||||
[Model unloaded?]
|
||||
↓
|
||||
Trigger load / Queue / Inform
|
||||
```
|
||||
|
||||
### Design Options
|
||||
|
||||
#### Option 1: Wake-on-Request (Auto-Load)
|
||||
|
||||
On receiving a request when no model is loaded:
|
||||
|
||||
1. **Spawn the container** with the requested model (from URL path or header)
|
||||
2. **Queue the request** (or return 503 with `Retry-After`)
|
||||
3. **Client retries** after model is ready (or proxy forwards queued request)
|
||||
|
||||
**Pros:** Transparent to clients, no manual intervention
|
||||
**Cons:** Long wait times (2-10 min), cold start, need to decide which model to load
|
||||
**Complexity:** Need request queue, loading state tracking, model selection logic
|
||||
|
||||
#### Option 2: Friendly Error + Status Endpoint
|
||||
|
||||
On receiving a request when no model is loaded:
|
||||
|
||||
1. **Return 503 Service Unavailable** with clear JSON body
|
||||
2. **Include status info:** "No model loaded. Load with `vllmctl up <model>`"
|
||||
3. **Provide a status endpoint** clients can poll: `GET /front-door/status`
|
||||
- Shows: models available, currently loaded, loading state (if any)
|
||||
|
||||
**Pros:** Simple to implement, clear client guidance
|
||||
**Cons:** Still requires manual load or client-side logic
|
||||
**Complexity:** Low — just a small nginx/envoy/python proxy
|
||||
|
||||
#### Option 3: Request Queue + Load Notification
|
||||
|
||||
On receiving a request when no model is loaded:
|
||||
|
||||
1. **Return 202 Accepted** with a `Location` header pointing to a job status URL
|
||||
2. **Trigger model load** in background
|
||||
3. **Client polls status** or gets notified (webhook) when ready
|
||||
4. **When ready, client retries** original request
|
||||
|
||||
**Pros:** Clear async flow, client knows what's happening
|
||||
**Cons:** More complex, needs job tracking, notification or polling
|
||||
**Complexity:** Medium — need job store, status API
|
||||
|
||||
## Implementation Sketch (Option 2: Friendly Error)
|
||||
|
||||
### Simple Python Front Door
|
||||
|
||||
```python
|
||||
# front-door.py
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
import subprocess
|
||||
import requests
|
||||
import json
|
||||
|
||||
VLLM_PORT = 8001
|
||||
FRONT_PORT = 8000
|
||||
MODELS_ROOT = "/data/home/renbaibing/huggingface"
|
||||
|
||||
class FrontDoorHandler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
# Check if vLLM is healthy
|
||||
try:
|
||||
resp = requests.get(f"http://127.0.0.1:{VLLM_PORT}/health", timeout=1)
|
||||
# Proxy the request
|
||||
proxy_resp = requests.get(f"http://127.0.0.1:{VLLM_PORT}{self.path}")
|
||||
self.wfile.write(proxy_resp.content)
|
||||
except requests.exceptions.RequestException:
|
||||
# vLLM is not ready
|
||||
self.send_response(503)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.end_headers()
|
||||
response = {
|
||||
"error": "No model currently loaded",
|
||||
"message": "Load a model with: ./vllmctl up <model>",
|
||||
"available_models": self.list_models(),
|
||||
"status_url": f"http://{self.server.server_port}/status"
|
||||
}
|
||||
self.wfile.write(json.dumps(response).encode())
|
||||
|
||||
def list_models(self):
|
||||
# Scan MODEL_ROOT for model directories
|
||||
# Return list of available model names
|
||||
pass
|
||||
|
||||
HTTPServer(("", FRONT_PORT), FrontDoorHandler).serve_forever()
|
||||
```
|
||||
|
||||
### Integration with existing setup
|
||||
|
||||
1. Move vLLM to port 8001 (`VLLM_PORT=8001` in .env)
|
||||
2. Run front door on port 8000 (or via compose as a separate service)
|
||||
3. Front door checks vLLM health before proxying
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Model selection:** If auto-loading, how does the proxy know which model to load?
|
||||
- From URL path? (`/v1/models/NAME/completions`?)
|
||||
- From header? (`X-Model-Name: Qwen3.6-35B-A3B`)
|
||||
- Configured default?
|
||||
|
||||
2. **Cold start cost:** Large models take 2-10 minutes to load from NFS
|
||||
- Is this acceptable for clients?
|
||||
- Should we cache recently-used models?
|
||||
|
||||
3. **Multi-model concurrent serving:** Does vLLM support serving multiple models simultaneously?
|
||||
- If yes, the proxy could load models on-demand and keep them warm
|
||||
- If no, we need model switching logic (potentially disruptive to existing requests)
|
||||
|
||||
4. **Authentication:** Should the front door handle API keys?
|
||||
- Current setup has no auth
|
||||
- Adding auth at the front door would be natural
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Confirm requirements:** Decide between Option 1 (auto-load), 2 (friendly error), or 3 (async queue)
|
||||
2. **Prototype:** Build a minimal front door for the chosen option
|
||||
3. **Integration:** Wire into existing `compose.yml` and `vllmctl` workflow
|
||||
4. **Testing:** Simulate various failure modes and client behaviors
|
||||
|
||||
## References
|
||||
|
||||
- Current setup: `README.md`, `NOTES-2026-08-13.md`
|
||||
- vLLM health endpoint: `http://127.0.0.1:8000/health`
|
||||
- vLLM metrics: `http://127.0.0.1:8000/metrics` (used by idle-watch)
|
||||
- OpenAI API compatibility: `http://127.0.0.1:8000/v1/...`
|
||||
Reference in New Issue
Block a user