vllm-frontdoor/.claude/memory/front-door-proxy-design.md

Front Door Proxy Design — Handling Requests to Unloaded Models

Date: 2026-08-13 Status: Design exploration — vLLM Sleep Mode available as better alternative

TL;DR: vLLM 0.11.0+ includes Sleep Mode — a built-in feature that enables fast model switching (18-200x faster than full reload) by hibernating weights while preserving process state. This is likely the solution we need instead of building a custom front door.

Current Behavior: Request to Unloaded Model

When a client makes a request to http://<host>:8000/v1/... and no model is loaded:

1. Container is down (auto-unload via idle-watch or manual down)
2. Immediate connection failure — client gets Connection refused or similar
3. No retry guidance — client doesn't know if the service is down, moving, or loading
4. Manual recovery required — someone must run ./vllmctl up <model>

What the client sees

$ curl http://host:8000/v1/models
curl: (7) Failed to connect to host port 8000: Connection refused

No indication of:

• Whether the service is permanently down
• Which model should be loaded
• How long loading would take (~2-10 min for large models from NFS)

The Shared Front Door Concept

A reverse proxy that sits in front of vLLM and handles the "unloaded model" case gracefully.

Architecture

Client → Front Door (port 8000) → vLLM Container (port 8001 or similar)
         ↓
   [Model unloaded?]
         ↓
   Trigger load / Queue / Inform

Design Options

Option 1: Wake-on-Request (Auto-Load)

On receiving a request when no model is loaded:

1. Spawn the container with the requested model (from URL path or header)
2. Queue the request (or return 503 with Retry-After)
3. Client retries after model is ready (or proxy forwards queued request)

Pros: Transparent to clients, no manual intervention
Cons: Long wait times (2-10 min), cold start, need to decide which model to load
Complexity: Need request queue, loading state tracking, model selection logic

Option 2: Friendly Error + Status Endpoint

On receiving a request when no model is loaded:

1. Return 503 Service Unavailable with clear JSON body
2. Include status info: "No model loaded. Load with vllmctl up <model>"
3. Provide a status endpoint clients can poll: GET /front-door/status
   • Shows: models available, currently loaded, loading state (if any)

Pros: Simple to implement, clear client guidance
Cons: Still requires manual load or client-side logic
Complexity: Low — just a small nginx/envoy/python proxy

Option 3: Request Queue + Load Notification

On receiving a request when no model is loaded:

1. Return 202 Accepted with a Location header pointing to a job status URL
2. Trigger model load in background
3. Client polls status or gets notified (webhook) when ready
4. When ready, client retries original request

Pros: Clear async flow, client knows what's happening
Cons: More complex, needs job tracking, notification or polling
Complexity: Medium — need job store, status API

Implementation Sketch (Option 2: Friendly Error)

Simple Python Front Door

# front-door.py
from http.server import HTTPServer, BaseHTTPRequestHandler
import subprocess
import requests
import json

VLLM_PORT = 8001
FRONT_PORT = 8000
MODELS_ROOT = "/data/home/renbaibing/huggingface"

class FrontDoorHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        # Check if vLLM is healthy
        try:
            resp = requests.get(f"http://127.0.0.1:{VLLM_PORT}/health", timeout=1)
            # Proxy the request
            proxy_resp = requests.get(f"http://127.0.0.1:{VLLM_PORT}{self.path}")
            self.wfile.write(proxy_resp.content)
        except requests.exceptions.RequestException:
            # vLLM is not ready
            self.send_response(503)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            response = {
                "error": "No model currently loaded",
                "message": "Load a model with: ./vllmctl up <model>",
                "available_models": self.list_models(),
                "status_url": f"http://{self.server.server_port}/status"
            }
            self.wfile.write(json.dumps(response).encode())
    
    def list_models(self):
        # Scan MODEL_ROOT for model directories
        # Return list of available model names
        pass

HTTPServer(("", FRONT_PORT), FrontDoorHandler).serve_forever()

Integration with existing setup

1. Move vLLM to port 8001 (VLLM_PORT=8001 in .env)
2. Run front door on port 8000 (or via compose as a separate service)
3. Front door checks vLLM health before proxying

Open Questions

1. Model selection: If auto-loading, how does the proxy know which model to load?

   • From URL path? (/v1/models/NAME/completions?)
   • From header? (X-Model-Name: Qwen3.6-35B-A3B)
   • Configured default?

2. Cold start cost: Large models take 2-10 minutes to load from NFS

   • Is this acceptable for clients?
   • Should we cache recently-used models?

3. Multi-model concurrent serving: Does vLLM support serving multiple models simultaneously?

   • If yes, the proxy could load models on-demand and keep them warm
   • If no, we need model switching logic (potentially disruptive to existing requests)

4. Authentication: Should the front door handle API keys?

   • Current setup has no auth
   • Adding auth at the front door would be natural

Next Steps

1. Confirm requirements: Decide between Option 1 (auto-load), 2 (friendly error), or 3 (async queue)
2. Prototype: Build a minimal front door for the chosen option
3. Integration: Wire into existing compose.yml and vllmctl workflow
4. Testing: Simulate various failure modes and client behaviors

References

• Current setup: README.md, NOTES-2026-08-13.md
• vLLM health endpoint: http://127.0.0.1:8000/health
• vLLM metrics: http://127.0.0.1:8000/metrics (used by idle-watch)
• OpenAI API compatibility: http://127.0.0.1:8000/v1/...