bing/vllm-frontdoor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: router front-door vLLM stack

Browse Source 
Three always-running vLLM services (text TP=2 GPU0+1, ocr + embed on GPU2,
sleep mode) behind a FastAPI router that auto-wakes models on request.
Tiered idle (sleep 15 min / offload 3 h), depth-aware 503s with
Retry-After, persisted wake-intent recovery, admin API on 127.0.0.1:8010.
Routine control via vllmctl is pure HTTP — no docker on the request path.

Verified: 91 router unit tests + 15-test E2E on real hardware
(measurements in CALIBRATION.md; design record in
.claude/memory/router-front-door-plan.md).

Old nginx stack files removed before git init; design survives in
.claude/memory/sleep-mode-implementation-plan.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
bing
2026-08-17 10:17:42 +00:00
commit 80eef4ce6a
35 changed files with 6506 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

783
.claude/memory/sleep-mode-implementation-plan.md Normal file
View File

@@ -0,0 +1,783 @@
# Implementation Plan: vLLM Sleep Mode + nginx Front Door
**Date:** 2026-08-14
**Status:** Revised (v2) — Critical issues from review addressed
## Changelog (v2)
**Fixed Critical Issues:**
1. ✅ Healthcheck dependency — changed to `service_started` (doesn't require healthy)
2. ✅ nginx upstream host — changed from `127.0.0.1:8001` to `vllm:8000`
3. ✅ Port binding clarified — vLLM: `127.0.0.1:8001:8000` (localhost only), nginx: external 8000
4. ✅ Added `--enable-sleep-mode` flag to EXTRA_ARGS
5. ✅ Implemented idle watcher update with `/sleep` calls
**Important Concerns Addressed:**
- Added note about `/metrics` exposure
- Clarified in-flight request handling during `/sleep`
- Updated wake time estimate to be more realistic
- Removed redundant `return 403` after `deny all`
---
## Overview
Transform the current vLLM setup from full container teardown (`docker compose down`) to vLLM Sleep Mode, with nginx as a security front door.
### Current Architecture
```
Client → [vLLM Container :8000]
↓
down = docker compose down (full teardown)
up = docker compose up (cold start, 2-10 min)
```
### Target Architecture
```
Internet → [nginx :8000] → [vLLM :8000 (Docker network) ]
↓
[vLLM :8001 (localhost only) ]
↓
sleep = POST /sleep (model hibernates, GPU freed)
wake = POST /wake_up (model resumes, ~5-15s for 35B)
```
### Port Binding Clarification
| Port | Access | Purpose |
|------|--------|---------|
| `8000` (nginx) | External (0.0.0.0:8000) | Public API, clients connect here |
| `8001` (vLLM) | Localhost only (127.0.0.1:8001) | Admin access for `vllmctl`, sleep/wake endpoints |
| `8000` (Docker network) | Internal | nginx → vLLM communication within Docker network |
---
## Phase 1: Enable vLLM Sleep Mode
### 1.1 Update `compose.yml`
**Critical changes:**
1. Add `VLLM_SERVER_DEV_MODE=1` to environment (required for Sleep Mode endpoints)
2. Add `--enable-sleep-mode` to EXTRA_ARGS (this was missing in v1!)
3. Expose vLLM on localhost only: `127.0.0.1:8001:8000`
4. Keep vLLM accessible on Docker network at port 8000
**Before:**
```yaml
services:
vllm:
image: vllm/vllm-openai:${VLLM_VERSION:-v0.27.1}
container_name: vllm
restart: unless-stopped
ports:
- "${VLLM_PORT:-8000}:8000"
volumes:
- ${MODEL_ROOT:-/data/home/renbaibing/huggingface}:/models:ro
ipc: host
gpus: all
environment:
- HF_TOKEN=${HF_TOKEN:-}
- MODEL_NAME=${MODEL_NAME}
- TP_SIZE=${TP_SIZE:-2}
- MAX_MODEL_LEN=${MAX_MODEL_LEN:-32768}
- GPU_MEM_UTIL=${GPU_MEM_UTIL:-}
- EXTRA_ARGS=${EXTRA_ARGS:-}
- NCCL_P2P_DISABLE=${NCCL_P2P_DISABLE:-1}
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)\" || exit 1"]
interval: 15s
timeout: 10s
retries: 8
start_period: 1200s
```
**After:**
```yaml
services:
vllm:
image: vllm/vllm-openai:${VLLM_VERSION:-v0.27.1}
container_name: vllm
restart: unless-stopped
# Two port bindings:
# 1. Docker network port 8000 (internal, for nginx)
# 2. Localhost port 8001 (admin access for vllmctl)
ports:
- "127.0.0.1:8001:8000" # Admin access - localhost only
expose:
- "8000" # Docker network - for nginx
volumes:
- ${MODEL_ROOT:-/data/home/renbaibing/huggingface}:/models:ro
ipc: host
gpus: all
environment:
- VLLM_SERVER_DEV_MODE=1 # Required for Sleep Mode endpoints
- HF_TOKEN=${HF_TOKEN:-}
- MODEL_NAME=${MODEL_NAME}
- TP_SIZE=${TP_SIZE:-2}
- MAX_MODEL_LEN=${MAX_MODEL_LEN:-32768}
- GPU_MEM_UTIL=${GPU_MEM_UTIL:-}
# EXTRA_ARGS MUST include --enable-sleep-mode
- EXTRA_ARGS=${EXTRA_ARGS:- --enable-sleep-mode}
- NCCL_P2P_DISABLE=${NCCL_P2P_DISABLE:-1}
# Healthcheck stays the same - container will be "unhealthy" during sleep
# This is OK - nginx uses service_started, not service_healthy
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)\" || exit 1"]
interval: 15s
timeout: 10s
retries: 8
start_period: 1200s
```
**Rationale:**
- `VLLM_SERVER_DEV_MODE=1` enables dev endpoints including `/sleep`, `/wake_up`, `/is_sleeping`
- `--enable-sleep-mode` in EXTRA_ARGS is REQUIRED for sleep functionality (this was the critical missing piece in v1)
- Port `127.0.0.1:8001` gives `vllmctl` localhost-only access to admin endpoints
- `expose: - "8000"` makes port 8000 available on Docker network for nginx
- Healthcheck will fail during sleep, but that's OK — nginx uses `service_started` condition
### 1.2 Update `vllmctl`
**First, update the API URL:**
```bash
# Change at the top of vllmctl:
VLLM_PORT="$(env_get VLLM_PORT 8000)"
API="http://127.0.0.1:8001" # Direct to vLLM localhost port, bypasses nginx
```
**Modify `cmd_down()`:**
```bash
cmd_down() {
say 'Putting vLLM to sleep (frees GPU memory, keeps server alive)...'
# Try to sleep the model (level 2 = discard weights, minimal RAM)
if curl -fs -X POST "$API/sleep?level=2" >/dev/null 2>&1; then
ok "Model is sleeping. Server still running."
# Wait a moment for sleep to complete
sleep 2
# Verify sleep state
local sleeping
sleeping="$(curl -fs "$API/is_sleeping" 2>/dev/null)"
if [ "$sleeping" = "true" ]; then
ok "Confirmed: Model is in sleep state."
else
warn "Sleep state unclear - check with: curl $API/is_sleeping"
fi
else
warn "Sleep request failed — server may not be ready. Falling back to full stop."
dcompose down -t 20 || return 1
fi
# Show GPU memory after sleep
local used
used="$(nvidia-smi --query-gpu=index,memory.used --format=csv,noheader,nounits 2>/dev/null | awk -F', *' '{printf "GPU%s:%sMiB ", $1, $2}')"
say "GPU memory now: $used"
}
```
**Rationale:**
- `level=2` discards weights entirely (minimal RAM usage)
- Wake time for 35B model: ~5-15s (more realistic than the 0.8-2.6s estimate)
- Adds sleep verification to confirm the operation worked
- Falls back to full stop if sleep fails
**Modify `cmd_up()`:**
```bash
cmd_up() {
local model="${1:-}"
if [ -n "$model" ]; then
if [ ! -d "$MODEL_ROOT/$model" ]; then
err "Model '$model' not found in $MODEL_ROOT"
err "Available:"; cmd_list
return 1
fi
env_set MODEL_NAME "$model"
fi
model="$(env_get MODEL_NAME)"
[ -n "$model" ] || { err 'No MODEL_NAME set in .env'; return 1; }
local st
st="$(container_state)"
# Case 1: Container not running at all
case "$st" in
absent*|exited*|dead*)
say "Starting vLLM with model '${C_CYAN}$model${C_OFF}' (TP=$(env_get TP_SIZE 2), Sleep Mode enabled)…"
dcompose up -d || return 1
wait_for_ready "$model"
return 0
;;
esac
# Case 2: Container running - check if sleeping or different model
local sleeping loaded
sleeping="$(curl -fs "$API/is_sleeping" 2>/dev/null)"
loaded="$(loaded_model)"
# Model already loaded and serving
if [ "$sleeping" != "true" ] && [ "$loaded" = "$model" ]; then
ok "Already serving '$model'."
return 0
fi
# Model is sleeping - wake it up
if [ "$sleeping" = "true" ]; then
say "Model is sleeping. Waking up '${C_CYAN}$model${C_OFF}'…"
# Level 2 sleep requires reload_weights after wake_up
curl -fs -X POST "$API/wake_up" >/dev/null 2>&1 || {
err "Wake-up request failed."
return 1
}
# Reload weights (required for level 2 sleep)
curl -fs -X POST "$API/collective_rpc" \
-H 'Content-Type: application/json' \
-d '{"method":"reload_weights"}' >/dev/null 2>&1 || {
warn "reload_weights request failed - model may still wake up"
}
# Reset prefix cache (required for level 2 sleep)
curl -fs -X POST "$API/reset_prefix_cache" >/dev/null 2>&1 || {
warn "reset_prefix_cache failed - non-critical"
}
# Wait for model to be ready
wait_for_ready "$model"
ok "Model '$model' is awake and ready."
return 0
fi
# Different model loaded - need to switch
if [ -n "$loaded" ] && [ "$loaded" != "$model" ]; then
say "Switching from '${C_YELLOW}$loaded${C_OFF}' to '${C_CYAN}$model${C_OFF}'…"
say "Restarting container with new model (this will take a few minutes)…"
dcompose down -t 20
sleep 2
dcompose up -d || return 1
wait_for_ready "$model"
ok "Now serving '$model'."
return 0
fi
# Shouldn't reach here, but handle gracefully
err "Unexpected state - container running but no model loaded. Try: ./vllmctl restart"
return 1
}
```
**Rationale:**
- Now handles the sleeping state properly
- For level 2 sleep, calls the required sequence: `/wake_up` → `reload_weights` → `reset_prefix_cache`
- Keeps container restart for model switching (cleaner than in-place reload)
- Better error messages and state handling
**Modify `cmd_status()`:**
```bash
cmd_status() {
local st loaded cfg sleeping
st="$(container_state)"
cfg="$(env_get MODEL_NAME)"
say "${C_CYAN}Container:${C_OFF} $st"
# Check if sleeping first (loaded_model won't work when sleeping)
sleeping="$(curl -fs "$API/is_sleeping" 2>/dev/null)"
if [ "$sleeping" = "true" ]; then
say "${C_CYAN}Serving:${C_OFF} ${C_DIM}model sleeping${C_OFF} (configured: $cfg)"
else
loaded="$(loaded_model)"
if [ -n "$loaded" ]; then
say "${C_CYAN}Serving:${C_OFF} ${C_GREEN}$loaded${C_OFF} at http://127.0.0.1:${VLLM_PORT}/v1"
else
say "${C_CYAN}Serving:${C_OFF} ${C_DIM}nothing loaded${C_OFF} (configured: $cfg)"
fi
fi
say "${C_CYAN}GPU:${C_OFF}"
nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu \
--format=csv,noheader 2>/dev/null | sed 's/^/ /'
# External access (nginx)
say "${C_CYAN}Public API:${C_OFF} http://$(hostname -f | head -1):${VLLM_PORT}/v1"
say "${C_CYAN}Admin API:${C_OFF} $API (localhost only)"
# Idle watcher status
local pidfile="$ROOT/.idle.pid"
if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile" 2>/dev/null)" 2>/dev/null; then
say "${C_CYAN}Idle-watch:${C_OFF} active (pid $(cat "$pidfile), $(cat "$ROOT/.idle.minutes" 2>/dev/null) min timeout)"
else
say "${C_CYAN}Idle-watch:${C_OFF} ${C_DIM}off${C_OFF} (enable: ./vllmctl idle-watch on [minutes])"
fi
}
```
**Modify `idle_loop()` — CRITICAL FIX:**
```bash
idle_loop() {
local timeout_min="$1"
local timeout_s=$(( timeout_min * 60 ))
local last_active=0
say "[idle-watch] started: will auto-sleep after ${timeout_min} min without requests (poll 30s)"
# Use internal API for sleep calls
local SLEEP_API="http://127.0.0.1:8001"
while true; do
sleep 30
# Check if container is running at all
local st
st="$(container_state)"
case "$st" in
absent*|exited*|dead*)
# Container not running - nothing to do
last_active=0
continue
;;
esac
# Check if already sleeping
local sleeping
sleeping="$(curl -fs "$SLEEP_API/is_sleeping" 2>/dev/null)"
if [ "$sleeping" = "true" ]; then
# Already sleeping, nothing to do
continue
fi
# Check for activity using metrics
local idle_s='' run_wait=0
idle_s="$(curl -fs -m 5 "$SLEEP_API/metrics" 2>/dev/null \
| awk '/^vllm:time_since_last_request_seconds/ {print $2; exit}')"
if [ -z "$idle_s" ] || [ "$idle_s" = "+Inf" ] || [ "$idle_s" = "NaN" ]; then
# Fall back: count running/waiting requests
run_wait="$(curl -fs -m 5 "$SLEEP_API/metrics" 2>/dev/null \
| awk '/^vllm:num_requests_(running|waiting)/ {s+=$2} END {print s+0}')"
if [ "${run_wait:-0}" != "0" ]; then
last_active="$(date +%s)"
continue
fi
[ "$last_active" = 0 ] && last_active="$(date +%s)"
idle_s=$(( $(date +%s) - last_active ))
fi
# Integer compare (strip possible decimals)
local idle_i=${idle_s%%.*}
if [ -n "$idle_i" ] && [ "$idle_i" -ge "$timeout_s" ] 2>/dev/null; then
say "[idle-watch] idle for ${idle_i}s ≥ ${timeout_s}s → putting model to sleep"
if curl -fs -X POST "$SLEEP_API/sleep?level=2" >/dev/null 2>&1; then
say "[idle-watch] model is now sleeping at $(date '+%F %T')"
else
warn '[idle-watch] sleep request failed - check if server is responsive'
fi
last_active=0
fi
done
}
```
**Rationale:**
- Uses internal API (`:8001`) for all calls
- Checks if container is running before attempting API calls
- Checks if already sleeping (avoid redundant sleep calls)
- Calls `/sleep?level=2` instead of `dcompose down`
- Graceful error handling if server is unresponsive
### 1.3 Update `.env`
No changes needed — everything is in `compose.yml` and `vllmctl`.
---
## Phase 2: Add nginx Front Door
### 2.1 Create `nginx.conf`
```nginx
events {
worker_connections 1024;
}
http {
# Rate limiting (optional, can be commented out)
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
# Upstream using Docker service name - CRITICAL FIX
upstream vllm {
server vllm:8000; # Docker service name, not 127.0.0.1
keepalive 32;
}
server {
listen 8000;
server_name _;
# Block dev endpoints — deny all is sufficient
location /sleep {
deny all;
}
location /wake_up {
deny all;
}
location /is_sleeping {
deny all;
}
location /collective_rpc {
deny all;
}
location /reset_prefix_cache {
deny all;
}
# Main API proxy
location / {
limit_req zone=api_limit burst=20 nodelay;
proxy_pass http://vllm;
proxy_http_version 1.1;
# Headers for WebSocket/streaming support
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts for LLM inference (increased from v1)
proxy_read_timeout 900s;
proxy_send_timeout 900s;
proxy_connect_timeout 10s;
# Disable buffering for streaming
proxy_buffering off;
}
# Health endpoint (for health checks, allows unhealthy during sleep)
location /health {
proxy_pass http://vllm/health;
access_log off;
# Don't fail if unhealthy - vLLM can be sleeping
proxy_next_upstream error timeout http_502 http_503 http_504;
}
# Metrics endpoint (WARNING: exposed without auth)
# Consider adding authentication if this becomes publicly accessible
location /metrics {
proxy_pass http://vllm/metrics;
}
}
}
```
**Rationale:**
- Fixed: `server vllm:8000` uses Docker service name (was `127.0.0.1:8001` in v1)
- Removed redundant `return 403` after `deny all`
- Increased timeouts to 900s for long-running inference
- Added `proxy_next_upstream` for health endpoint tolerance
- Added warning comment about `/metrics` exposure
### 2.2 Update `compose.yml`
Add nginx service with CRITICAL FIX to dependency:
```yaml
services:
nginx:
image: nginx:alpine
container_name: vllm-nginx
restart: unless-stopped
ports:
- "${VLLM_PORT:-8000}:8000"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
# CRITICAL FIX: Use service_started, not service_healthy
# vLLM healthcheck fails during sleep, but that's OK
depends_on:
vllm:
condition: service_started # Changed from service_healthy
networks:
- default
vllm:
# ... (as shown in Phase 1.1)
```
**Rationale:**
- **CRITICAL FIX**: Changed from `service_healthy` to `service_started`
- This allows nginx to start even when vLLM healthcheck fails (during sleep)
- nginx will still return 502 if vLLM is completely down, which is correct behavior
### 2.3 Port Binding Summary
After all changes, the port layout is:
| From | To | Who Can Access | Purpose |
|-----|-----|----------------|---------|
| `0.0.0.0:8000` | nginx:8000 | Everyone | Public API |
| nginx | vllm:8000 | Docker only | nginx→vLLM |
| `127.0.0.1:8001` | vllm:8000 | Localhost only | Admin (`vllmctl`) |
**Security Model:**
- External clients hit nginx on port 8000
- nginx blocks `/sleep`, `/wake_up`, `/is_sleeping`, etc.
- `vllmctl` uses localhost:8001 to bypass nginx and access admin endpoints
- vLLM containers can talk to each other on Docker network at vllm:8000
---
## Phase 3: Testing & Validation
### 3.1 Test Sleep Mode
```bash
# 1. Start model
./vllmctl up Qwen3.6-35B-A3B
# 2. Verify serving through nginx
curl http://localhost:8000/v1/models
# Should show: {"object":"list","data":[{"id":"Qwen3.6-35B-A3B",...}]}
# 3. Put to sleep
./vllmctl down
# Should show: "Model is sleeping. Server still running."
# 4. Check GPU memory freed
nvidia-smi
# GPU memory should be significantly lower
# 5. Verify external API blocked - dev endpoints
curl http://localhost:8000/sleep
# Should return: 403 Forbidden
# 6. Verify internal API works
curl http://localhost:8001/is_sleeping
# Should return: true
# 7. Test public API during sleep
curl http://localhost:8000/v1/models
# Should return error or timeout (model is sleeping)
# 8. Wake up
./vllmctl up
# Should wake up the existing model
# 9. Verify serving again
curl http://localhost:8000/v1/models
# Should work again
```
### 3.2 Test Idle Watcher
```bash
# Enable short idle timeout for testing
./vllmctl idle-watch on 1
# Wait 1 minute, then check status
./vllmctl status
# Should show: "model sleeping"
# Test that wake-up works
./vllmctl up
./vllmctl status
# Should show: serving the model
# Turn off when done
./vllmctl idle-watch off
```
### 3.3 Test Error Handling
```bash
# 1. Stop vLLM container
docker compose stop vllm
# 2. Try request through nginx
curl http://localhost:8000/v1/models
# Should get: 502 Bad Gateway
# 3. Start vLLM again
docker compose start vllm
# 4. Verify recovery
curl http://localhost:8000/v1/models
# Should work again
```
### 3.4 Test In-Flight Request Handling
**Note:** vLLM Sleep Mode will allow in-flight requests to complete before sleeping. Test:
```bash
# Start a long-running request in background
curl -N http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{"model":"Qwen3.6-35B-A3B","prompt":"Tell me a long story","max_tokens":500}' &
CURL_PID=$!
# Immediately put to sleep
./vllmctl down
# The request should complete (may take a moment)
wait $CURL_PID
echo "Request completed with exit code: $?"
```
---
## Phase 4: Optional Enhancements
### 4.1 Protect /metrics Endpoint
If `/metrics` exposure is a concern:
```nginx
location /metrics {
# Optional: basic auth
# auth_basic "Metrics";
# auth_basic_user_file /etc/nginx/.htpasswd;
# Or restrict to localhost
# allow 127.0.0.1;
# deny all;
proxy_pass http://vllm/metrics;
}
```
### 4.2 Friendly Error When Model Is Sleeping
Add Lua-based checking (requires nginx with Lua module):
```nginx
# This requires nginx-lua or OpenResty - can add later if needed
```
### 4.3 Request Logging
Enable access logging for debugging:
```nginx
http {
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
# ...
}
```
And mount log volume in compose.yml:
```yaml
nginx:
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./logs/nginx:/var/log/nginx
```
---
## Migration Steps
1. **Backup current setup:**
```bash
cp compose.yml compose.yml.bak
cp vllmctl vllmctl.bak
```
2. **Update `compose.yml`:**
- Add nginx service
- Update vLLM service with new ports and environment
- Add `--enable-sleep-mode` to EXTRA_ARGS
3. **Create `nginx.conf`:**
- Copy the config above
- Adjust as needed
4. **Update `vllmctl`:**
- Change API URL to `:8001`
- Modify `cmd_down()`, `cmd_up()`, `cmd_status()`, `idle_loop()`
- Use the code blocks from Phase 1.2
5. **Test:**
```bash
docker compose down
docker compose up -d
./vllmctl status
```
6. **Rollback if needed:**
```bash
docker compose down
cp compose.yml.bak compose.yml
cp vllmctl.bak vllmctl
docker compose up -d
```
---
## Security Considerations
1. **Dev endpoints are protected** — nginx blocks `/sleep`, `/wake_up`, `/is_sleeping`, `/collective_rpc`, `/reset_prefix_cache`
2. **Internal access only** — `vllmctl` uses port 8001 which is bound to localhost only
3. **Container isolation** — vLLM and nginx are in the same Docker network
4. **WARNING: /metrics is exposed** — Consider adding authentication if deployment becomes public
5. **VLLM_SERVER_DEV_MODE=1** — This enables dev endpoints; ensure nginx properly blocks them
---
## Performance Impact
- **nginx overhead**: Minimal (~1-2ms latency, sub-1% CPU)
- **Sleep Mode wake**: ~5-15s for level 2 with 35B model (realistic estimate)
- **Memory**: Level 2 sleep uses minimal CPU RAM, frees ~90% GPU memory
- **In-flight requests**: vLLM allows completion before sleep
---
## Open Questions (Resolved)
1. ✅ **Should idle watcher call `/sleep` directly?**
- **RESOLVED**: Yes, updated `idle_loop()` to call `/sleep?level=2` instead of `dcompose down`
2. **Should we optimize model switching?**
- Current: Restart container on model switch (clean, reliable)
- Could optimize later: Use `/sleep` + `/wake_up` + `reload_weights` for faster switches
- Decision: Start with restart, optimize later if needed
3. ⚠️ **What about `/metrics` endpoint?**
- Currently accessible through nginx (no auth)
- **Decision**: Leave open for now, add auth later if deployment becomes public
---
## Estimated Effort (Updated)
- Phase 1 (vLLM Sleep Mode): ~45 minutes (increased due to more comprehensive changes)
- Phase 2 (nginx): ~30 minutes
- Phase 3 (Testing): ~45 minutes (more thorough testing)
- **Total**: ~2 hours (updated from 1.5 hours)
---
## Next Steps
Once you approve this revised plan, I'll:
1. Create the modified files (`compose.yml`, `nginx.conf`, `vllmctl`)
2. Test in a worktree or provide step-by-step instructions
3. Document rollback procedure
**All critical issues from the first review have been addressed.**
Ready to proceed?