vllm-frontdoor/.claude/memory/sleep-mode-implementation-plan.md

Implementation Plan: vLLM Sleep Mode + nginx Front Door

Date: 2026-08-14
Status: Revised (v2) — Critical issues from review addressed

Changelog (v2)

Fixed Critical Issues:

1. ✅ Healthcheck dependency — changed to service_started (doesn't require healthy)
2. ✅ nginx upstream host — changed from 127.0.0.1:8001 to vllm:8000
3. ✅ Port binding clarified — vLLM: 127.0.0.1:8001:8000 (localhost only), nginx: external 8000
4. ✅ Added --enable-sleep-mode flag to EXTRA_ARGS
5. ✅ Implemented idle watcher update with /sleep calls

Important Concerns Addressed:

• Added note about /metrics exposure
• Clarified in-flight request handling during /sleep
• Updated wake time estimate to be more realistic
• Removed redundant return 403 after deny all

---

Overview

Transform the current vLLM setup from full container teardown (docker compose down) to vLLM Sleep Mode, with nginx as a security front door.

Current Architecture

Client → [vLLM Container :8000]
           ↓
    down = docker compose down (full teardown)
    up   = docker compose up (cold start, 2-10 min)

Target Architecture

Internet → [nginx :8000] → [vLLM :8000 (Docker network) ]
                           ↓
                           [vLLM :8001 (localhost only) ]
                           ↓
                      sleep = POST /sleep (model hibernates, GPU freed)
                      wake  = POST /wake_up (model resumes, ~5-15s for 35B)

Port Binding Clarification

Port	Access	Purpose
8000 (nginx)	External (0.0.0.0:8000)	Public API, clients connect here
8001 (vLLM)	Localhost only (127.0.0.1:8001)	Admin access for vllmctl, sleep/wake endpoints
8000 (Docker network)	Internal	nginx → vLLM communication within Docker network

---

Phase 1: Enable vLLM Sleep Mode

1.1 Update compose.yml

Critical changes:

1. Add VLLM_SERVER_DEV_MODE=1 to environment (required for Sleep Mode endpoints)
2. Add --enable-sleep-mode to EXTRA_ARGS (this was missing in v1!)
3. Expose vLLM on localhost only: 127.0.0.1:8001:8000
4. Keep vLLM accessible on Docker network at port 8000

Before:

services:
  vllm:
    image: vllm/vllm-openai:${VLLM_VERSION:-v0.27.1}
    container_name: vllm
    restart: unless-stopped
    ports:
      - "${VLLM_PORT:-8000}:8000"
    volumes:
      - ${MODEL_ROOT:-/data/home/renbaibing/huggingface}:/models:ro
    ipc: host
    gpus: all
    environment:
      - HF_TOKEN=${HF_TOKEN:-}
      - MODEL_NAME=${MODEL_NAME}
      - TP_SIZE=${TP_SIZE:-2}
      - MAX_MODEL_LEN=${MAX_MODEL_LEN:-32768}
      - GPU_MEM_UTIL=${GPU_MEM_UTIL:-}
      - EXTRA_ARGS=${EXTRA_ARGS:-}
      - NCCL_P2P_DISABLE=${NCCL_P2P_DISABLE:-1}
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)\" || exit 1"]
      interval: 15s
      timeout: 10s
      retries: 8
      start_period: 1200s

After:

services:
  vllm:
    image: vllm/vllm-openai:${VLLM_VERSION:-v0.27.1}
    container_name: vllm
    restart: unless-stopped
    
    # Two port bindings:
    # 1. Docker network port 8000 (internal, for nginx)
    # 2. Localhost port 8001 (admin access for vllmctl)
    ports:
      - "127.0.0.1:8001:8000"  # Admin access - localhost only
      expose:
        - "8000"  # Docker network - for nginx
    
    volumes:
      - ${MODEL_ROOT:-/data/home/renbaibing/huggingface}:/models:ro
    ipc: host
    gpus: all
    
    environment:
      - VLLM_SERVER_DEV_MODE=1  # Required for Sleep Mode endpoints
      - HF_TOKEN=${HF_TOKEN:-}
      - MODEL_NAME=${MODEL_NAME}
      - TP_SIZE=${TP_SIZE:-2}
      - MAX_MODEL_LEN=${MAX_MODEL_LEN:-32768}
      - GPU_MEM_UTIL=${GPU_MEM_UTIL:-}
      # EXTRA_ARGS MUST include --enable-sleep-mode
      - EXTRA_ARGS=${EXTRA_ARGS:- --enable-sleep-mode}
      - NCCL_P2P_DISABLE=${NCCL_P2P_DISABLE:-1}
    
    # Healthcheck stays the same - container will be "unhealthy" during sleep
    # This is OK - nginx uses service_started, not service_healthy
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)\" || exit 1"]
      interval: 15s
      timeout: 10s
      retries: 8
      start_period: 1200s

Rationale:

• VLLM_SERVER_DEV_MODE=1 enables dev endpoints including /sleep, /wake_up, /is_sleeping
• --enable-sleep-mode in EXTRA_ARGS is REQUIRED for sleep functionality (this was the critical missing piece in v1)
• Port 127.0.0.1:8001 gives vllmctl localhost-only access to admin endpoints
• expose: - "8000" makes port 8000 available on Docker network for nginx
• Healthcheck will fail during sleep, but that's OK — nginx uses service_started condition

1.2 Update vllmctl

First, update the API URL:

# Change at the top of vllmctl:
VLLM_PORT="$(env_get VLLM_PORT 8000)"
API="http://127.0.0.1:8001"  # Direct to vLLM localhost port, bypasses nginx

Modify cmd_down():

cmd_down() {
    say 'Putting vLLM to sleep (frees GPU memory, keeps server alive)...'
    
    # Try to sleep the model (level 2 = discard weights, minimal RAM)
    if curl -fs -X POST "$API/sleep?level=2" >/dev/null 2>&1; then
        ok "Model is sleeping. Server still running."
        
        # Wait a moment for sleep to complete
        sleep 2
        
        # Verify sleep state
        local sleeping
        sleeping="$(curl -fs "$API/is_sleeping" 2>/dev/null)"
        if [ "$sleeping" = "true" ]; then
            ok "Confirmed: Model is in sleep state."
        else
            warn "Sleep state unclear - check with: curl $API/is_sleeping"
        fi
    else
        warn "Sleep request failed — server may not be ready. Falling back to full stop."
        dcompose down -t 20 || return 1
    fi
    
    # Show GPU memory after sleep
    local used
    used="$(nvidia-smi --query-gpu=index,memory.used --format=csv,noheader,nounits 2>/dev/null | awk -F', *' '{printf "GPU%s:%sMiB ", $1, $2}')"
    say "GPU memory now: $used"
}

Rationale:

• level=2 discards weights entirely (minimal RAM usage)
• Wake time for 35B model: ~5-15s (more realistic than the 0.8-2.6s estimate)
• Adds sleep verification to confirm the operation worked
• Falls back to full stop if sleep fails

Modify cmd_up():

cmd_up() {
    local model="${1:-}"
    if [ -n "$model" ]; then
        if [ ! -d "$MODEL_ROOT/$model" ]; then
            err "Model '$model' not found in $MODEL_ROOT"
            err "Available:"; cmd_list
            return 1
        fi
        env_set MODEL_NAME "$model"
    fi
    model="$(env_get MODEL_NAME)"
    [ -n "$model" ] || { err 'No MODEL_NAME set in .env'; return 1; }

    local st
    st="$(container_state)"
    
    # Case 1: Container not running at all
    case "$st" in
        absent*|exited*|dead*)
            say "Starting vLLM with model '${C_CYAN}$model${C_OFF}' (TP=$(env_get TP_SIZE 2), Sleep Mode enabled)…"
            dcompose up -d || return 1
            wait_for_ready "$model"
            return 0
            ;;
    esac

    # Case 2: Container running - check if sleeping or different model
    local sleeping loaded
    sleeping="$(curl -fs "$API/is_sleeping" 2>/dev/null)"
    loaded="$(loaded_model)"
    
    # Model already loaded and serving
    if [ "$sleeping" != "true" ] && [ "$loaded" = "$model" ]; then
        ok "Already serving '$model'."
        return 0
    fi

    # Model is sleeping - wake it up
    if [ "$sleeping" = "true" ]; then
        say "Model is sleeping. Waking up '${C_CYAN}$model${C_OFF}'…"
        # Level 2 sleep requires reload_weights after wake_up
        curl -fs -X POST "$API/wake_up" >/dev/null 2>&1 || {
            err "Wake-up request failed."
            return 1
        }
        # Reload weights (required for level 2 sleep)
        curl -fs -X POST "$API/collective_rpc" \
            -H 'Content-Type: application/json' \
            -d '{"method":"reload_weights"}' >/dev/null 2>&1 || {
            warn "reload_weights request failed - model may still wake up"
        }
        # Reset prefix cache (required for level 2 sleep)
        curl -fs -X POST "$API/reset_prefix_cache" >/dev/null 2>&1 || {
            warn "reset_prefix_cache failed - non-critical"
        }
        # Wait for model to be ready
        wait_for_ready "$model"
        ok "Model '$model' is awake and ready."
        return 0
    fi

    # Different model loaded - need to switch
    if [ -n "$loaded" ] && [ "$loaded" != "$model" ]; then
        say "Switching from '${C_YELLOW}$loaded${C_OFF}' to '${C_CYAN}$model${C_OFF}'…"
        say "Restarting container with new model (this will take a few minutes)…"
        dcompose down -t 20
        sleep 2
        dcompose up -d || return 1
        wait_for_ready "$model"
        ok "Now serving '$model'."
        return 0
    fi

    # Shouldn't reach here, but handle gracefully
    err "Unexpected state - container running but no model loaded. Try: ./vllmctl restart"
    return 1
}

Rationale:

• Now handles the sleeping state properly
• For level 2 sleep, calls the required sequence: /wake_up → reload_weights → reset_prefix_cache
• Keeps container restart for model switching (cleaner than in-place reload)
• Better error messages and state handling

Modify cmd_status():

cmd_status() {
    local st loaded cfg sleeping
    st="$(container_state)"
    cfg="$(env_get MODEL_NAME)"

    say "${C_CYAN}Container:${C_OFF} $st"
    
    # Check if sleeping first (loaded_model won't work when sleeping)
    sleeping="$(curl -fs "$API/is_sleeping" 2>/dev/null)"
    
    if [ "$sleeping" = "true" ]; then
        say "${C_CYAN}Serving:${C_OFF}   ${C_DIM}model sleeping${C_OFF}  (configured: $cfg)"
    else
        loaded="$(loaded_model)"
        if [ -n "$loaded" ]; then
            say "${C_CYAN}Serving:${C_OFF}   ${C_GREEN}$loaded${C_OFF} at http://127.0.0.1:${VLLM_PORT}/v1"
        else
            say "${C_CYAN}Serving:${C_OFF}   ${C_DIM}nothing loaded${C_OFF}  (configured: $cfg)"
        fi
    fi

    say "${C_CYAN}GPU:${C_OFF}"
    nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu \
        --format=csv,noheader 2>/dev/null | sed 's/^/    /'

    # External access (nginx)
    say "${C_CYAN}Public API:${C_OFF} http://$(hostname -f | head -1):${VLLM_PORT}/v1"
    say "${C_CYAN}Admin API:${C_OFF} $API (localhost only)"

    # Idle watcher status
    local pidfile="$ROOT/.idle.pid"
    if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile" 2>/dev/null)" 2>/dev/null; then
        say "${C_CYAN}Idle-watch:${C_OFF} active (pid $(cat "$pidfile), $(cat "$ROOT/.idle.minutes" 2>/dev/null) min timeout)"
    else
        say "${C_CYAN}Idle-watch:${C_OFF} ${C_DIM}off${C_OFF}  (enable: ./vllmctl idle-watch on [minutes])"
    fi
}

Modify idle_loop() — CRITICAL FIX:

idle_loop() {
    local timeout_min="$1"
    local timeout_s=$(( timeout_min * 60 ))
    local last_active=0
    say "[idle-watch] started: will auto-sleep after ${timeout_min} min without requests (poll 30s)"
    
    # Use internal API for sleep calls
    local SLEEP_API="http://127.0.0.1:8001"
    
    while true; do
        sleep 30
        
        # Check if container is running at all
        local st
        st="$(container_state)"
        case "$st" in
            absent*|exited*|dead*)
                # Container not running - nothing to do
                last_active=0
                continue
                ;;
        esac
        
        # Check if already sleeping
        local sleeping
        sleeping="$(curl -fs "$SLEEP_API/is_sleeping" 2>/dev/null)"
        if [ "$sleeping" = "true" ]; then
            # Already sleeping, nothing to do
            continue
        fi
        
        # Check for activity using metrics
        local idle_s='' run_wait=0
        idle_s="$(curl -fs -m 5 "$SLEEP_API/metrics" 2>/dev/null \
            | awk '/^vllm:time_since_last_request_seconds/ {print $2; exit}')"
        
        if [ -z "$idle_s" ] || [ "$idle_s" = "+Inf" ] || [ "$idle_s" = "NaN" ]; then
            # Fall back: count running/waiting requests
            run_wait="$(curl -fs -m 5 "$SLEEP_API/metrics" 2>/dev/null \
                | awk '/^vllm:num_requests_(running|waiting)/ {s+=$2} END {print s+0}')"
            if [ "${run_wait:-0}" != "0" ]; then
                last_active="$(date +%s)"
                continue
            fi
            [ "$last_active" = 0 ] && last_active="$(date +%s)"
            idle_s=$(( $(date +%s) - last_active ))
        fi
        
        # Integer compare (strip possible decimals)
        local idle_i=${idle_s%%.*}
        if [ -n "$idle_i" ] && [ "$idle_i" -ge "$timeout_s" ] 2>/dev/null; then
            say "[idle-watch] idle for ${idle_i}s ≥ ${timeout_s}s → putting model to sleep"
            if curl -fs -X POST "$SLEEP_API/sleep?level=2" >/dev/null 2>&1; then
                say "[idle-watch] model is now sleeping at $(date '+%F %T')"
            else
                warn '[idle-watch] sleep request failed - check if server is responsive'
            fi
            last_active=0
        fi
    done
}

Rationale:

• Uses internal API (:8001) for all calls
• Checks if container is running before attempting API calls
• Checks if already sleeping (avoid redundant sleep calls)
• Calls /sleep?level=2 instead of dcompose down
• Graceful error handling if server is unresponsive

1.3 Update .env

No changes needed — everything is in compose.yml and vllmctl.

---

Phase 2: Add nginx Front Door

2.1 Create nginx.conf

events {
    worker_connections 1024;
}

http {
    # Rate limiting (optional, can be commented out)
    limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;

    # Upstream using Docker service name - CRITICAL FIX
    upstream vllm {
        server vllm:8000;  # Docker service name, not 127.0.0.1
        keepalive 32;
    }

    server {
        listen 8000;
        server_name _;

        # Block dev endpoints — deny all is sufficient
        location /sleep {
            deny all;
        }
        location /wake_up {
            deny all;
        }
        location /is_sleeping {
            deny all;
        }
        location /collective_rpc {
            deny all;
        }
        location /reset_prefix_cache {
            deny all;
        }

        # Main API proxy
        location / {
            limit_req zone=api_limit burst=20 nodelay;
            
            proxy_pass http://vllm;
            proxy_http_version 1.1;
            
            # Headers for WebSocket/streaming support
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            
            # Timeouts for LLM inference (increased from v1)
            proxy_read_timeout 900s;
            proxy_send_timeout 900s;
            proxy_connect_timeout 10s;
            
            # Disable buffering for streaming
            proxy_buffering off;
        }

        # Health endpoint (for health checks, allows unhealthy during sleep)
        location /health {
            proxy_pass http://vllm/health;
            access_log off;
            # Don't fail if unhealthy - vLLM can be sleeping
            proxy_next_upstream error timeout http_502 http_503 http_504;
        }

        # Metrics endpoint (WARNING: exposed without auth)
        # Consider adding authentication if this becomes publicly accessible
        location /metrics {
            proxy_pass http://vllm/metrics;
        }
    }
}

Rationale:

• Fixed: server vllm:8000 uses Docker service name (was 127.0.0.1:8001 in v1)
• Removed redundant return 403 after deny all
• Increased timeouts to 900s for long-running inference
• Added proxy_next_upstream for health endpoint tolerance
• Added warning comment about /metrics exposure

2.2 Update compose.yml

Add nginx service with CRITICAL FIX to dependency:

services:
  nginx:
    image: nginx:alpine
    container_name: vllm-nginx
    restart: unless-stopped
    ports:
      - "${VLLM_PORT:-8000}:8000"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    # CRITICAL FIX: Use service_started, not service_healthy
    # vLLM healthcheck fails during sleep, but that's OK
    depends_on:
      vllm:
        condition: service_started  # Changed from service_healthy
    networks:
      - default

  vllm:
    # ... (as shown in Phase 1.1)

Rationale:

• CRITICAL FIX: Changed from service_healthy to service_started
• This allows nginx to start even when vLLM healthcheck fails (during sleep)
• nginx will still return 502 if vLLM is completely down, which is correct behavior

2.3 Port Binding Summary

After all changes, the port layout is:

From	To	Who Can Access	Purpose
0.0.0.0:8000	nginx:8000	Everyone	Public API
nginx	vllm:8000	Docker only	nginx→vLLM
127.0.0.1:8001	vllm:8000	Localhost only	Admin (vllmctl)

Security Model:

• External clients hit nginx on port 8000
• nginx blocks /sleep, /wake_up, /is_sleeping, etc.
• vllmctl uses localhost:8001 to bypass nginx and access admin endpoints
• vLLM containers can talk to each other on Docker network at vllm:8000

---

Phase 3: Testing & Validation

3.1 Test Sleep Mode

# 1. Start model
./vllmctl up Qwen3.6-35B-A3B

# 2. Verify serving through nginx
curl http://localhost:8000/v1/models
# Should show: {"object":"list","data":[{"id":"Qwen3.6-35B-A3B",...}]}

# 3. Put to sleep
./vllmctl down
# Should show: "Model is sleeping. Server still running."

# 4. Check GPU memory freed
nvidia-smi
# GPU memory should be significantly lower

# 5. Verify external API blocked - dev endpoints
curl http://localhost:8000/sleep
# Should return: 403 Forbidden

# 6. Verify internal API works
curl http://localhost:8001/is_sleeping
# Should return: true

# 7. Test public API during sleep
curl http://localhost:8000/v1/models
# Should return error or timeout (model is sleeping)

# 8. Wake up
./vllmctl up
# Should wake up the existing model

# 9. Verify serving again
curl http://localhost:8000/v1/models
# Should work again

3.2 Test Idle Watcher

# Enable short idle timeout for testing
./vllmctl idle-watch on 1

# Wait 1 minute, then check status
./vllmctl status
# Should show: "model sleeping"

# Test that wake-up works
./vllmctl up
./vllmctl status
# Should show: serving the model

# Turn off when done
./vllmctl idle-watch off

3.3 Test Error Handling

# 1. Stop vLLM container
docker compose stop vllm

# 2. Try request through nginx
curl http://localhost:8000/v1/models
# Should get: 502 Bad Gateway

# 3. Start vLLM again
docker compose start vllm

# 4. Verify recovery
curl http://localhost:8000/v1/models
# Should work again

3.4 Test In-Flight Request Handling

Note: vLLM Sleep Mode will allow in-flight requests to complete before sleeping. Test:

# Start a long-running request in background
curl -N http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-35B-A3B","prompt":"Tell me a long story","max_tokens":500}' &
CURL_PID=$!

# Immediately put to sleep
./vllmctl down

# The request should complete (may take a moment)
wait $CURL_PID
echo "Request completed with exit code: $?"

---

Phase 4: Optional Enhancements

4.1 Protect /metrics Endpoint

If /metrics exposure is a concern:

location /metrics {
    # Optional: basic auth
    # auth_basic "Metrics";
    # auth_basic_user_file /etc/nginx/.htpasswd;
    
    # Or restrict to localhost
    # allow 127.0.0.1;
    # deny all;
    
    proxy_pass http://vllm/metrics;
}

4.2 Friendly Error When Model Is Sleeping

Add Lua-based checking (requires nginx with Lua module):

# This requires nginx-lua or OpenResty - can add later if needed

4.3 Request Logging

Enable access logging for debugging:

http {
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
    # ...
}

And mount log volume in compose.yml:

nginx:
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./logs/nginx:/var/log/nginx

---

Migration Steps

1. Backup current setup:

   cp compose.yml compose.yml.bak
   cp vllmctl vllmctl.bak
   

2. Update compose.yml:

   • Add nginx service
   • Update vLLM service with new ports and environment
   • Add --enable-sleep-mode to EXTRA_ARGS

3. Create nginx.conf:

   • Copy the config above
   • Adjust as needed

4. Update vllmctl:

   • Change API URL to :8001
   • Modify cmd_down(), cmd_up(), cmd_status(), idle_loop()
   • Use the code blocks from Phase 1.2

5. Test:

   docker compose down
   docker compose up -d
   ./vllmctl status
   

6. Rollback if needed:

   docker compose down
   cp compose.yml.bak compose.yml
   cp vllmctl.bak vllmctl
   docker compose up -d
   

---

Security Considerations

1. Dev endpoints are protected — nginx blocks /sleep, /wake_up, /is_sleeping, /collective_rpc, /reset_prefix_cache
2. Internal access only — vllmctl uses port 8001 which is bound to localhost only
3. Container isolation — vLLM and nginx are in the same Docker network
4. WARNING: /metrics is exposed — Consider adding authentication if deployment becomes public
5. VLLM_SERVER_DEV_MODE=1 — This enables dev endpoints; ensure nginx properly blocks them

---

Performance Impact

• nginx overhead: Minimal (~1-2ms latency, sub-1% CPU)
• Sleep Mode wake: ~5-15s for level 2 with 35B model (realistic estimate)
• Memory: Level 2 sleep uses minimal CPU RAM, frees ~90% GPU memory
• In-flight requests: vLLM allows completion before sleep

---

Open Questions (Resolved)

1. ✅ Should idle watcher call /sleep directly?

   • RESOLVED: Yes, updated idle_loop() to call /sleep?level=2 instead of dcompose down

2. Should we optimize model switching?

   • Current: Restart container on model switch (clean, reliable)
   • Could optimize later: Use /sleep + /wake_up + reload_weights for faster switches
   • Decision: Start with restart, optimize later if needed

3. ⚠️ What about /metrics endpoint?

   • Currently accessible through nginx (no auth)
   • Decision: Leave open for now, add auth later if deployment becomes public

---

Estimated Effort (Updated)

• Phase 1 (vLLM Sleep Mode): ~45 minutes (increased due to more comprehensive changes)
• Phase 2 (nginx): ~30 minutes
• Phase 3 (Testing): ~45 minutes (more thorough testing)
• Total: ~2 hours (updated from 1.5 hours)

---

Next Steps

Once you approve this revised plan, I'll:

1. Create the modified files (compose.yml, nginx.conf, vllmctl)
2. Test in a worktree or provide step-by-step instructions
3. Document rollback procedure

All critical issues from the first review have been addressed.

Ready to proceed?