vllm-frontdoor/.claude/memory/front-door-proxy-design.md

# Front Door Proxy Design — Handling Requests to Unloaded Models

**Date:** 2026-08-13
**Status:** Design exploration — vLLM Sleep Mode available as better alternative

> **TL;DR:** vLLM 0.11.0+ includes **Sleep Mode** — a built-in feature that enables fast model switching (18-200x faster than full reload) by hibernating weights while preserving process state. **This is likely the solution we need instead of building a custom front door.**

## Current Behavior: Request to Unloaded Model

When a client makes a request to `http://<host>:8000/v1/...` and no model is loaded:

1. **Container is down** (auto-unload via `idle-watch` or manual `down`)
2. **Immediate connection failure** — client gets `Connection refused` or similar
3. **No retry guidance** — client doesn't know if the service is down, moving, or loading
4. **Manual recovery required** — someone must run `./vllmctl up <model>` 

### What the client sees

```bash
$ curl http://host:8000/v1/models
curl: (7) Failed to connect to host port 8000: Connection refused
```

No indication of:
- Whether the service is permanently down
- Which model should be loaded
- How long loading would take (~2-10 min for large models from NFS)

## The Shared Front Door Concept

A reverse proxy that sits in front of vLLM and handles the "unloaded model" case gracefully.

### Architecture

```
Client → Front Door (port 8000) → vLLM Container (port 8001 or similar)
         ↓
   [Model unloaded?]
         ↓
   Trigger load / Queue / Inform
```

### Design Options

#### Option 1: Wake-on-Request (Auto-Load)

On receiving a request when no model is loaded:

1. **Spawn the container** with the requested model (from URL path or header)
2. **Queue the request** (or return 503 with `Retry-After`)
3. **Client retries** after model is ready (or proxy forwards queued request)

**Pros:** Transparent to clients, no manual intervention  
**Cons:** Long wait times (2-10 min), cold start, need to decide which model to load  
**Complexity:** Need request queue, loading state tracking, model selection logic

#### Option 2: Friendly Error + Status Endpoint

On receiving a request when no model is loaded:

1. **Return 503 Service Unavailable** with clear JSON body
2. **Include status info:** "No model loaded. Load with `vllmctl up <model>`"
3. **Provide a status endpoint** clients can poll: `GET /front-door/status`
   - Shows: models available, currently loaded, loading state (if any)

**Pros:** Simple to implement, clear client guidance  
**Cons:** Still requires manual load or client-side logic  
**Complexity:** Low — just a small nginx/envoy/python proxy

#### Option 3: Request Queue + Load Notification

On receiving a request when no model is loaded:

1. **Return 202 Accepted** with a `Location` header pointing to a job status URL
2. **Trigger model load** in background
3. **Client polls status** or gets notified (webhook) when ready
4. **When ready, client retries** original request

**Pros:** Clear async flow, client knows what's happening  
**Cons:** More complex, needs job tracking, notification or polling  
**Complexity:** Medium — need job store, status API

## Implementation Sketch (Option 2: Friendly Error)

### Simple Python Front Door

```python
# front-door.py
from http.server import HTTPServer, BaseHTTPRequestHandler
import subprocess
import requests
import json

VLLM_PORT = 8001
FRONT_PORT = 8000
MODELS_ROOT = "/data/home/renbaibing/huggingface"

class FrontDoorHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        # Check if vLLM is healthy
        try:
            resp = requests.get(f"http://127.0.0.1:{VLLM_PORT}/health", timeout=1)
            # Proxy the request
            proxy_resp = requests.get(f"http://127.0.0.1:{VLLM_PORT}{self.path}")
            self.wfile.write(proxy_resp.content)
        except requests.exceptions.RequestException:
            # vLLM is not ready
            self.send_response(503)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            response = {
                "error": "No model currently loaded",
                "message": "Load a model with: ./vllmctl up <model>",
                "available_models": self.list_models(),
                "status_url": f"http://{self.server.server_port}/status"
            }
            self.wfile.write(json.dumps(response).encode())
    
    def list_models(self):
        # Scan MODEL_ROOT for model directories
        # Return list of available model names
        pass

HTTPServer(("", FRONT_PORT), FrontDoorHandler).serve_forever()
```

### Integration with existing setup

1. Move vLLM to port 8001 (`VLLM_PORT=8001` in .env)
2. Run front door on port 8000 (or via compose as a separate service)
3. Front door checks vLLM health before proxying

## Open Questions

1. **Model selection:** If auto-loading, how does the proxy know which model to load?
   - From URL path? (`/v1/models/NAME/completions`?)
   - From header? (`X-Model-Name: Qwen3.6-35B-A3B`)
   - Configured default?

2. **Cold start cost:** Large models take 2-10 minutes to load from NFS
   - Is this acceptable for clients?
   - Should we cache recently-used models?

3. **Multi-model concurrent serving:** Does vLLM support serving multiple models simultaneously?
   - If yes, the proxy could load models on-demand and keep them warm
   - If no, we need model switching logic (potentially disruptive to existing requests)

4. **Authentication:** Should the front door handle API keys?
   - Current setup has no auth
   - Adding auth at the front door would be natural

## Next Steps

1. **Confirm requirements:** Decide between Option 1 (auto-load), 2 (friendly error), or 3 (async queue)
2. **Prototype:** Build a minimal front door for the chosen option
3. **Integration:** Wire into existing `compose.yml` and `vllmctl` workflow
4. **Testing:** Simulate various failure modes and client behaviors

## References

- Current setup: `README.md`, `NOTES-2026-08-13.md`
- vLLM health endpoint: `http://127.0.0.1:8000/health`
- vLLM metrics: `http://127.0.0.1:8000/metrics` (used by idle-watch)
- OpenAI API compatibility: `http://127.0.0.1:8000/v1/...`